Improve slev obsdata subset and retrieve functions

[veenstrajelmer]

TODO:

• Add slev obsdata download function #245
• retrieve cmems insitu data on FTP via cmc instead of ftplib #715
• update cmems insitu from FTP to copernicusmarine files service #817
• enable cmems authentication on github and re-enable cmems source in test_ssh_catalog_subset() and test_ssh_retrieve_data()
• conversion to generic netcdf format, maybe based on DCSM slev files: "p:\archivedprojects\11205259-004-dcsm-fm\waterlevel_data\all\ncFiles\A12.nc". All datasets already have a waterlevel variable. Also enrich with global attrs from metadata and variable attributes like units/standardname if not present yet. Prevent adding uneccesary dims, in that case improve tidal_plot_tools. >> Conversion to hydrotools-compliant ds was done in add slev retrieval from ddl #794
• cmems my dataset has LATITUDE/LONGITUDE/POSITION dimensions, can be flattened but might not be an issue anymore after moving from ftp to api >> remove these vars (also for nrt) since lat/lon attrs are already present and station_x_coordinate (and y) are added anyway.
• add slev retrieval from ddl #794
• Create netcdf overview function #806
• GESLA obs netcdf file has multiple lon/lat values #862
• remove escape for unexpected cmems observation dataset formats #998
• remove cmems registration message #997
• cmems disable_progress_bar=True sometimes hangs for insitu data #893
• promote via internal email (after release)
• solve code smells (including convert TODO to issues)
• cleanup ds.attrs, for instance latitude vs geospatial-lat-min and max, probably more duplicate data present.
• are station id/name from hydrotools-compliant dataset aligned with attrs in read_catalog? Probably not. Consider adding all catalog values as attrs to ds. Make clear what attrs are available in ds always and check presence in retrieve testcase.
• cannot do multiple cmems runs at the same time, since the temporary raw filename is the same always and we can get hdf5 errors. Hopefully solved once we can access the data via copernicusmarine
• maybe merge with https://github.com/Deltares-research/cht_observations (NOAA and NDBC). Or alternative source for ndbc retrieval. Also discuss with hydromt-team. However, when moving to external package, we need to duplicate the credentials function(s) for cmems. Also regular releases would be great
• also link with https://github.com/oceanmodeling/searvey
• maybe merge with NWDM, this will be a Python package to retrieve insitu water quality data from several sources: https://github.com/Deltares-research/nwdm
• convenient but not complete SSC translation list: http://www.ioc-sealevelmonitoring.org/ssc/
• merge based on distance from ssc list? (including standardized station names)
• request gesla direct download of meta and zip?
• consider dropping all preselection of ds (like gesla coastal and ioc no dart), make it simpler. Simplify ioc no_uhslc selection by checking len for these two columns and check if both are >0.
• metadata collaboration initiative SONEL https://www.sonel.org/tgcat. E.g. uses by psmsl
• add velicities and water quality measurements? Or more
• add observation points (SSC or other sources?) to modelbuilder notebook and example script (including xyn file)
• consider clip arg ipv 4x lat/lon, dan ook poly support
• add to notebook: note that cmems is public but requires credentials, be mindful of licenses. Add information about quality and url for each dataset. Add dfmt.references() with per source a url, license, nrt/historic, quality and more information.
• maybe remove disabled ssc code, or see if still useful
• ask UHSLC to include some gesla/ioc stations. Gesla has high coverage in Canada, Chili(?), Japan and Australia. IOC has high coverage in Chili(?) and India. Probably more countries. Overview of gesla data providers is available. Also check this for cmems in europe, although coverage has already increased drastically.
• NOAA, NHS (Norway) and Marine Institute (Ireland) have data via API. CMEMS probably covers most of it.
• extra obspoints via OSM (no obsdata): add beaches, places, ports as observation points #748
• overwrite argument for ssh_retrieve_data
• align country/country_code column/attribute? Sometimes it is a name, sometimes a 3-digit/2-digit letter code and sometimes a numeric code
• add option to subset "all" sources at once. It might already be possible to pass a catalog dataframe of mixed sources so that would simplify the user code a bit. We would need time-subsetting for rwsddl in this case, and also suppress the progresbar for rwsddl.
• https://intake.readthedocs.io/en/latest/

Sources and connections:

• CMEMS: script from dec2023 subsets and downloads directly from FTP (no temporal subsetting possible).
• CMEMS multiyear: cmems_obs-ins_glo_phy-ssh_my_na_PT1H is probably the best source. It contains validated hourly data with 6-18 months delay. Not yet subsettable via API
• CMEMS multiyear irregular: hourly is nice, but maybe cmems_obs-ins_glo_phy-ssh_my_na_irr is better for model surge validation (if also validated)? However, this dataset contains multiple time frequencies for french stations
• CMEMS nrt: add insitu retrieval for cmems nrt #790
• Move CMEMS insitu from copernicusmarine files service to subset once available #835
• UHSLC: script from dec2023 subsets from geojson and downloads via erddap. GLOSS fast mode redirects to UHSLC fast and there is also UHSLC rqds
• GESLA3: gesla metadata and data have no direct download link, so cannot automate retrieval. Otherwise, we could consider downloading to cachedir and unzipping. Dataset is also static and a collection, so consider retrieving data at original data providers instead. Now local p-drive link.
• RWS DDL: add slev retrieval from ddl #794
• RWS DDL: Will be updated to WADAR and maybe that also slightly changes the DDL API. Consider to add support for time_min/time_max >> add function for latest measurements ddlpy#34 and add function for number of measurements per year/day ddlpy#36 would make it possible but latter would be slow and inaccurate. >> also add in dfm_tools observations.py and insitu notebook
• NODC: seems to have RWS data, added value?
• EMODNET: script from feb2023 can download from ERDDAP, how to quickly subset stations? Data is collected, so not a unique dataset: "For the Sea Level we are getting the data from CMEMS, IOC, TAD, UHSLC and PSMSL". Not necessary to add.
• TAD is the only source in EMODNET that was not inlcluded already. Table seems not easy to read with python, json format available? The data is available via json. There is an overview of providers. Could add coverage in India and potentially elsewhere. All stations in india were red/offline in march 2024, so first check if added value. Also historic available? Or would this be collected in EMODNET?
• IOC: scripts on gtsm repos, status unknown. GLOSS realtime mode redirects to IOC
• PSMSL: scripts on gtsm repos, status unknown. GLOSS delayed mode redirects to BODC and PSMSL high freq but much of that is UHSLC data. Potentially add monthly/yearly means and vertical reference data.
• PSMSL GNSS-IR: https://psmsl.org/data/gnssir and https://psmsl.org/data/gnssir/table.php
• GTSM3-reanalysis: maybe via CDS API, but spatial subsetting is not yet possible.
• BODC has only monthly zip files: https://www.bodc.ac.uk/data/hosted_data_systems/sea_level/uk_tide_gauge_network/processed, hourly data is in cmems_obs-ins_glo_phy-ssh_my_na_PT1H (this cmems my dataset might be historically extended and the original 15min res data might be taken in in the cmems nrt dataset)
• meetnetvlaamsebanken: p:\11210366-003-getijverandering\insitu_data\MVB\retrieve_mvb.py

Also tidal water level from components:

• IHO although data is not maintained (compatibility fix voor iho.nc mail)
• TICON-3 (components from GESLA-3)
• Arctic Tidal Constituent Atlas (ArcTiCA): https://arcticdata.io/catalog/view/doi%3A10.18739%2FA2D795C4N
• what to use to predict tide?
• hatyan predict tide from ticon-3 in bes project: p:\11209231-003-bes-modellering\hydrodynamica\preprocessing\modelbuilder\modelbuilder_parts_for_waterleveldata_and_xynFile_v2.py