Increase minimal copernicusmarine version

[veenstrajelmer]

Was added in #630

Remaining issues with copernicus-marine-client >> copernicusmarine:

• [CMCS-160] Subset command is taking more time than expected to yied results. Note from cmc developers: "we currently recommend you to use .open_dataset() instead of .subset() whenever you can. It is currently faster, will be fixed in the future." >> fixed in 0.10.3 or earlier version
• [CMCS-160] Query hanging for ever (from EDITO-INFRA chat) >> fixed in 0.10.3 or earlier version
• [CMCS-172] Degraded performance for describe command since version 0.9.2 (from EDITO-INFRA chat) >> fixed in 0.10.3 or earlier version
• [CMCD-350] Package dependencies matrix are tested with tox. Currently (version 0.10.0) has caret requirement for dask, resulting in installation conflicts with dfm_tools (requires dask>2023.4.0): copernicus-marine-client 0.9.11 requires dask<2023.0.0,>=2022.1.1, but you have dask 2023.11.0 which is incompatible. >> reported on 27-11-2023. This also goes for netcdf4 (netCDF4<2.0.0 and >=1.6.3), maybe also others. This can easily be tested with the pytest-py39-mindeps, which currently fails. Current dependencies are listed below. dfm_tools installation now complains with "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. distributed 2023.11.0 requires dask==2023.11.0, but you have dask 2022.12.1 which is incompatible." >> fixed in 0.10.3? >> dask requirement was loosened in cmc 0.10.6
• copernicus-marine-client 0.10.6 depends on dask >=2022,<2024 >> this now corresponds to the xarray range (good). What is the reason for excluding 2024?
• The user requested xarray==2023.4.0. copernicusmarine 1.0.0 depends on xarray >=2023.9 >> what is the reason for requiring 2023.9? (temporarily testing with higher version in 692-increase-minimal-copernicus-marine-client-version to make pytest-py39-mindeps testcase succeed) >> should be fixed in copernicusmarine>1.0.0 >> dfm_tools now requires the same minimal xarray version since 539 deprecate dfmtpreprocess hirlam after additional xarray fixes for curvilinear grids #770
• The user requested netcdf4==1.5.4. copernicusmarine 1.0.0 depends on netCDF4 >=1.6.3 >> what is the reason for requiring 1.6.3? (temporarily testing with higher version in 692-increase-minimal-copernicus-marine-client-version to make pytest-py39-mindeps testcase succeed) >> fixed in copernicusmarine>1.0.1
• [CMCS-207] no support for py312: Fix python 3.12 testbank #551 >> copernicusmarine still does not allow this even though the issue is fixed (copernicus-marine-client did allow this). This issue currently blocks the py3.12 testbank and with that a new dfm_tools release. >> fixed in copernicusmarine 1.0.1

Requirements from METADATA at 8-2-2024 (copernicusmarine 1.0.1):

Requires-Dist: aiohttp (>=3.8.5)
Requires-Dist: boto3 (>=1.25)
Requires-Dist: cachier (>=2.2.1)
Requires-Dist: click (>=8.0.4)
Requires-Dist: dask (>=2022)
Requires-Dist: lxml (>=4.9.0)
Requires-Dist: motuclient (>=1.8.4)
Requires-Dist: nest-asyncio (>=1.5.8)
Requires-Dist: netCDF4 (>=1.5.4)
Requires-Dist: pydap (>=3.2.2)
Requires-Dist: pystac (>=1.8.3)
Requires-Dist: requests (>=2.27.1)
Requires-Dist: semver (>=3.0.2)
Requires-Dist: setuptools (>=68.2.2)
Requires-Dist: tqdm (>=4.65.0)
Requires-Dist: xarray (>=2023.4.0)
Requires-Dist: zarr (>=2.13.3)

update 25-1-2024: a new toolbox was launched called copernicusmarine instead of copernicus-marine-client. Update to this.

Follow-up issue: #773 and #811

[dbazin-eo]

@veenstrajelmer please note a status about the following:

• [CMCS-160] Subset command is taking more time than expected to yied results
• [CMCS-160] Query hanging for ever
• [CMCS-172] Degraded performance for describe command since version 0.9.2
• [CMCS-207] no support for py312 -> This is independent of the toolbox itself: several package dependencies are not yet available for py312, they will be one day and therefore automatically the toolbox.
• [CMCD-350] Package dependencies matrix are tested with tox -> Ongoing
• [CMCS-176] Series of version upgrade in the same environment might alter execution on WindowsOS -> has been evaluated and will take long to be released
• [CMCS-174] Suggestion - Provide more context when prompting for user's credentials -> not been assessed as a blocking point but use case is understood and been agreed to be implemented eventually

We keep in touch 🙏

[veenstrajelmer]

Thanks @dbazin-eo for sharing this progress. I have one additional remark about the py312 support. I was indeed aware that there were several dependencies not yet supporting py312 (keeping track for dfm_tools in #551 and I think the last one will be finished very soon). However, as far as I understood the copernicus-marine-client pyproject.toml also contains python = ">=3.9, <3.12" which might prevent installation in a py312 environment even after the last dependency supports py312. Could this be updated to python = ">=3.9"? I read here that adding an upper cap on the python version is not recommended also. However, this might be outdated information, I do not have access to the current pyproject.toml.

[dbazin-eo]

thanks @veenstrajelmer 🙏 please do not worry, the pyproject.toml will be updated in due time. There were too many users reporting issues with python = ">=3.9" so we had to modify it to python = ">=3.9, <3.12" as a mitigating (and temporary) action, while dependencies such as multidict, are still NOK.
For the "^" issue, it is already flagged and work is in progress thanks to [CMCD-350].
thanks a lot for your question, we keep in touch

[veenstrajelmer]

Great @dbazin-eo, thanks for the clarifications. Sounds good to me. Thanks also for the useful pyreadiness website.

[veenstrajelmer]

@dbazin-eo I see there are new releases for cmc. Could you let me know if any of the open questions in this issue were tackled and could you please share the [tool.poetry.dependencies] for cmc 0.10.6? Thanks!