Update xcube-cds to work with new CDS backend API

[pont-us]

The current CDS API servers are scheduled to be shut down on 2024-09-03. As of 2024-07-25, ECMWF has not yet deployed any production-grade replacement, but there is a beta version of the planned successor at https://cds-beta.climate.copernicus.eu/. xcube-cds will need to support this new version before the current version is shut down.

More information:

• Informative page in readiness for new CDS-beta!
• Please read: CDS and ADS migrating to new infrastructure: Common Data Store (CDS) Engine

[pont-us]

The main difficulty is the change in ERA5. Previously any ERA5 request to the cdsapi backend would produce a single NetCDF file. Now, some requests produce a Zip containing multiple NetCDFs. ECMWF tech support clarified this behaviour as follows:

…whenever there a GRIB to NetCDF incompatibility is detected, the output NetCDF file will be split and delivered as a compressed zipped file (with the .zip extension). An example of incompatibility is requesting atmospheric and oceanic variables at the same time.

Here's a fairly minimal example demonstrating this behaviour with the new API:

import cdsapi

dataset = "reanalysis-era5-single-levels-monthly-means"
request = {
    'product_type': ['monthly_averaged_reanalysis'],
    'variable': ['2m_temperature', 'mean_wave_direction'],
    'year': ['2015'],
    'month': ['10'],
    'time': ['00:00'],
    'data_format': 'netcdf',
    'area': [1, -1, -1, 1]
}

client = cdsapi.Client()
client.retrieve(dataset, request).download()

This produces a Zip file containing two NetCDFs, each containing one of the requested variables. The NetCDFs have different resolutions, but this isn't always the case: sometimes a Zip is produced containing multiple NetCDFs with identical dimensions. The equivalent code for the old API is:

import cdsapi

dataset = "reanalysis-era5-single-levels-monthly-means"
request = {
    'product_type': ['monthly_averaged_reanalysis'],
    'variable': ['2m_temperature', 'mean_wave_direction'],
    'year': ['2015'],
    'month': ['10'],
    'time': ['00:00'],
    'data_format': 'netcdf',
    'area': [1, -1, -1, 1]
}

client = cdsapi.Client()
client.retrieve(dataset, request).download()

This produces a single NetCDF on a common grid.

[pont-us]

Sketch of a solution for ERA5:

• Detect whether the downloaded data is NetCDF or Zip. Currently NetCDF is simply assumed.
• If NetCDF, follow existing code path
• If Zip,
  • Unpack Zip into a temporary directory
  • Check if all NetCDFs have identical dimensions
    • If so, merge them into one dataset with xarray.open_mfdataset
    • If not, raise an exception