Update backend calls to match changes to the CDS API #69
Conversation
The schema used by the CDS backend has changed, so our schema has to change accordingly.
The mocked CDS client is now obtained via a factory method which can alternatively return a wrapped genuine CDS API client which saves requests and results for use in future mocks. This allows fast regeneration of all the mocked CDS data whenever the CDS backend changes.
There are now separate options to test with mocked data, test against the live API, and test against the live API while also saving data for future mocks.
URL and key parameters were not being correctly handled in all cases.
The CDS sea ice API now uses the value "cryosat_2" rather than "cryosat-2" to denote CryoSat-2. SeaIceThicknessHandler.transform_params has been updated accordingly.
Update the end of soil moisture time coverage expected by a unit test to match the new actual length of coverage.
The cdsapi backed has changed the request parameter name for the geopotential variable from "orography" to "geopotential". This commit updates the xcube-cds parameter mapping tables accordingly.
- Automatically create mock data directories when needed - Specify test data directory names for sea ice tests and test_client_url_and_key_environment_variables
This brings the mock data up to date with the current behaviour of the actual CDS API.
Remove _update_all_mock_results and _update_mock_result from store.py. These weren't needed in the end, and would be trivial to reimplement if they ever are needed.
Strange -- for me they all run. Which tests fail? And do you have credentials set in environment variables? |
In ClientUrlTest, it is test_client_url_and_key_environment_variables, test_client_url_and_key_parameters and test_client_url_and_key_rc_file . I set the behaviour to _Behaviour.SAVE_RESULTS. What is weird to me is that it is obvious they need to fail. They specify a wrong url and api key and then these are used to retrieve data from a client. |
Client parameterization tests now always use a mock, to avoid feeding invalid dummy parameters to the real client.
|
Thanks @TonioF , I was able to reproduce and I think I've fixed it -- flagging for re-review. |
Related issues
Closes #68 .
Motivation
The parameter schemas for the CDS API backend (a.k.a.
cdsapi) have slightly (and silently) changed -- for instance, the soil moisture product version numberv201912.0.0now has to be given asv201912. These changes break many xcube-cdsopen_datacalls. This PR updates the relevant backend calls and parameter schemas inxcube-cdsto make them work with the current live API again.Description
test/mock_results
These are just the updates of the canned API requests and responses used by the mock
cdsapiclient. They don't need much (or any?) reviewing.test/mocks.py
This file defines the mock
cdsapiclient. Significant changes, aimed at making it much easier to regenerate mock data from the live CDS API when the API changes. Previously this was a semi-manual process; now it just involves changing the_BEHAVIOURvariable and running pytest.test_era5.py, test_sea_ice_thickness.py, test_soil_moisture.py, test_store.py
Mainly just boilerplate changes to use the updated mock client code, which now supplies the client via a factory method.
xcube_cds/datasets/*.{json,py}
These are the actual (fairly minor) changes in the backend calls to make them work with the current CDS API.