Missing joblib dependency

[benowen-bom]

Recently on new builds of our local env, certain improver cli calls fail with the following error: ModuleNotFoundError: No module named 'joblib'

Looking within IMPROVER, joblib is explicitly used (imported) in a number of files, but is not included in any of the environment files.

The recency of these failures suggest that joblib is coming in via another dependency, but that this has recently changed, or that the relevant dependency is not present in our local environment. While it seems that the environments this repository are fine, it seems sensible that joblib should be an explicit dependency to ensure that this issue isn't possible.

An example of a CLI call which results in this error is below:

improver standardise --attributes-config stage_attributes_config.json cycle/20240801T1200Z/ecmwf_ens/windspdpresslev/fetchlevel1/20240802T1200Z-PT0024H00M-wind_speed_on_pressure_levels.nc --output cycle/20240801T1200Z/ecmwf_ens/windspdpresslev/standardise/20240802T1200Z-PT0024H00M-wind_speed_on_pressure_levels.nc
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/548/bo1606/improver_suite/improver/improver/cli/__main__.py", line 10, in <module>
    cli.run_main()
  File "/home/548/bo1606/improver_suite/improver/improver/cli/__init__.py", line 611, in run_main
    run(main, args=argv)
  File "/g/data/zd19/bo1606/miniforge3/envs/improver/lib/python3.12/site-packages/sigtools/modifiers.py", line 161, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/zd19/bo1606/miniforge3/envs/improver/lib/python3.12/site-packages/clize/runner.py", line 404, in run
    ret = cli(*args)
          ^^^^^^^^^^
  File "/g/data/zd19/bo1606/miniforge3/envs/improver/lib/python3.12/site-packages/clize/runner.py", line 271, in __call__
    return func(*posargs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/548/bo1606/improver_suite/improver/improver/cli/__init__.py", line 582, in main
    result = exec_cmd(
             ^^^^^^^^^
  File "/home/548/bo1606/improver_suite/improver/improver/cli/__init__.py", line 520, in execute_command
    result = dispatcher(prog_name, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/zd19/bo1606/miniforge3/envs/improver/lib/python3.12/site-packages/clize/runner.py", line 271, in __call__
    return func(*posargs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/zd19/bo1606/miniforge3/envs/improver/lib/python3.12/site-packages/clize/runner.py", line 313, in _cli
    return func('{0} {1}'.format(name, command), *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/zd19/bo1606/miniforge3/envs/improver/lib/python3.12/site-packages/clize/runner.py", line 270, in __call__
    func, name, posargs, kwargs = self.read_commandline(args)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/zd19/bo1606/miniforge3/envs/improver/lib/python3.12/site-packages/clize/runner.py", line 280, in read_commandline
    ba = self.signature.read_arguments(args[1:], args[0])
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/zd19/bo1606/miniforge3/envs/improver/lib/python3.12/site-packages/clize/parser.py", line 1201, in read_arguments
    ba.process_arguments()
  File "/g/data/zd19/bo1606/miniforge3/envs/improver/lib/python3.12/site-packages/clize/parser.py", line 1398, in process_arguments
    param.read_argument(self, i)
  File "/g/data/zd19/bo1606/miniforge3/envs/improver/lib/python3.12/site-packages/clize/parser.py", line 529, in read_argument
    super(OptionParameter, self).read_argument(ba, i)
  File "/g/data/zd19/bo1606/miniforge3/envs/improver/lib/python3.12/site-packages/clize/parser.py", line 372, in read_argument
    self.set_value(ba, self.coerce_value(self.get_value(ba, i), ba))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/zd19/bo1606/miniforge3/envs/improver/lib/python3.12/site-packages/clize/parser.py", line 388, in coerce_value
    ret = self.conv(arg)
          ^^^^^^^^^^^^^^
  File "/home/548/bo1606/improver_suite/improver/improver/cli/__init__.py", line 184, in inputjson
    from improver.utilities.cli_utilities import load_json_or_none
  File "/home/548/bo1606/improver_suite/improver/improver/utilities/cli_utilities.py", line 10, in <module>
    import joblib
ModuleNotFoundError: No module named 'joblib'

[benowen-bom]

I've managed to create a simple reproducible test case on this:

mamba create --name improver_test python=3.12 improver
mamba activate improver_test
python
>>> from improver.cli import standardise
>>> standardise.process()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/g/data/zd19/bo1606/miniforge3/envs/improver_test/lib/python3.12/site-packages/sigtools/wrappers.py", line 122, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/zd19/bo1606/miniforge3/envs/improver_test/lib/python3.12/site-packages/improver/cli/__init__.py", line 350, in with_output
    import joblib
ModuleNotFoundError: No module named 'joblib'

[gavinevans]

Thanks @benowen-bom. We've started to use the joblib package in the last ~6 months, but you're right that we haven't included it as an explicit dependency. I've created this PR: #2344 to address this. Would you be able to review it?