Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve slev obsdata subset and retrieve functions #712

Open
19 of 60 tasks
veenstrajelmer opened this issue Dec 10, 2023 · 0 comments
Open
19 of 60 tasks

Improve slev obsdata subset and retrieve functions #712

veenstrajelmer opened this issue Dec 10, 2023 · 0 comments

Comments

@veenstrajelmer
Copy link
Collaborator

veenstrajelmer commented Dec 10, 2023

TODO:

  • Add slev obsdata download function #245
  • retrieve cmems insitu data on FTP via cmc instead of ftplib #715
  • update cmems insitu from FTP to copernicusmarine files service #817
  • enable cmems authentication on github and re-enable cmems source in test_ssh_catalog_subset() and test_ssh_retrieve_data()
  • conversion to generic netcdf format, maybe based on DCSM slev files: "p:\archivedprojects\11205259-004-dcsm-fm\waterlevel_data\all\ncFiles\A12.nc". All datasets already have a waterlevel variable. Also enrich with global attrs from metadata and variable attributes like units/standardname if not present yet. Prevent adding uneccesary dims, in that case improve tidal_plot_tools. >> Conversion to hydrotools-compliant ds was done in add slev retrieval from ddl #794
  • cmems my dataset has LATITUDE/LONGITUDE/POSITION dimensions, can be flattened but might not be an issue anymore after moving from ftp to api >> remove these vars (also for nrt) since lat/lon attrs are already present and station_x_coordinate (and y) are added anyway.
  • add slev retrieval from ddl #794
  • Create netcdf overview function #806
  • GESLA obs netcdf file has multiple lon/lat values #862
  • remove escape for unexpected cmems observation dataset formats #998
  • remove cmems registration message #997
  • cmems disable_progress_bar=True sometimes hangs for insitu data #893
  • promote via internal email (after release)
  • solve code smells (including convert TODO to issues)
  • cleanup ds.attrs, for instance latitude vs geospatial-lat-min and max, probably more duplicate data present.
  • are station id/name from hydrotools-compliant dataset aligned with attrs in read_catalog? Probably not. Consider adding all catalog values as attrs to ds. Make clear what attrs are available in ds always and check presence in retrieve testcase.
  • cannot do multiple cmems runs at the same time, since the temporary raw filename is the same always and we can get hdf5 errors. Hopefully solved once we can access the data via copernicusmarine
  • maybe merge with https://github.com/Deltares-research/cht_observations (NOAA and NDBC). Or alternative source for ndbc retrieval. Also discuss with hydromt-team. However, when moving to external package, we need to duplicate the credentials function(s) for cmems. Also regular releases would be great
  • also link with https://github.com/oceanmodeling/searvey
  • maybe merge with NWDM, this will be a Python package to retrieve insitu water quality data from several sources: https://github.com/Deltares-research/nwdm
  • convenient but not complete SSC translation list: http://www.ioc-sealevelmonitoring.org/ssc/
  • merge based on distance from ssc list? (including standardized station names)
  • request gesla direct download of meta and zip?
  • consider dropping all preselection of ds (like gesla coastal and ioc no dart), make it simpler. Simplify ioc no_uhslc selection by checking len for these two columns and check if both are >0.
  • metadata collaboration initiative SONEL https://www.sonel.org/tgcat. E.g. uses by psmsl
  • add velicities and water quality measurements? Or more
  • add observation points (SSC or other sources?) to modelbuilder notebook and example script (including xyn file)
  • consider clip arg ipv 4x lat/lon, dan ook poly support
  • add to notebook: note that cmems is public but requires credentials, be mindful of licenses. Add information about quality and url for each dataset. Add dfmt.references() with per source a url, license, nrt/historic, quality and more information.
  • maybe remove disabled ssc code, or see if still useful
  • ask UHSLC to include some gesla/ioc stations. Gesla has high coverage in Canada, Chili(?), Japan and Australia. IOC has high coverage in Chili(?) and India. Probably more countries. Overview of gesla data providers is available. Also check this for cmems in europe, although coverage has already increased drastically.
  • NOAA, NHS (Norway) and Marine Institute (Ireland) have data via API. CMEMS probably covers most of it.
  • extra obspoints via OSM (no obsdata): add beaches, places, ports as observation points #748
  • overwrite argument for ssh_retrieve_data
  • align country/country_code column/attribute? Sometimes it is a name, sometimes a 3-digit/2-digit letter code and sometimes a numeric code
  • add option to subset "all" sources at once. It might already be possible to pass a catalog dataframe of mixed sources so that would simplify the user code a bit. We would need time-subsetting for rwsddl in this case, and also suppress the progresbar for rwsddl.
  • https://intake.readthedocs.io/en/latest/

Sources and connections:

Also tidal water level from components:

  • IHO although data is not maintained (compatibility fix voor iho.nc mail)
  • TICON-3 (components from GESLA-3)
  • Arctic Tidal Constituent Atlas (ArcTiCA): https://arcticdata.io/catalog/view/doi%3A10.18739%2FA2D795C4N
  • what to use to predict tide?
  • hatyan predict tide from ticon-3 in bes project: p:\11209231-003-bes-modellering\hydrodynamica\preprocessing\modelbuilder\modelbuilder_parts_for_waterleveldata_and_xynFile_v2.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant