Modify nb on ESGF access to remove authentication process

[huard]

• Remove authentication process
• Add basic search functionality using pyesgf client.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[huard]

Looks like pyesgf is not installed on pavics in the latest current env.

[tlvu]

Looks like pyesgf is not installed on pavics in the latest current env.

Hehe, because it has never been previously installed :D

Are we talking about this one https://anaconda.org/conda-forge/esgf-pyclient ?

[tlvu]

Looks like pyesgf is not installed on pavics in the latest current env.

Hehe, because it has never been previously installed :D

Are we talking about this one https://anaconda.org/conda-forge/esgf-pyclient ?

https://esgf-pyclient.readthedocs.io/en/latest/quickstart.html, looks like it's the same code in this PR, I'll add this module to the Jupyter env.

[tlvu]

The "Alpha" version of the Jupyter env has esgf-pyclient installed if you want to reproduce the error or refresh the notebook.

Jenkins test on this PR found the following error:

http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build/125/consoleFull

22:44:05  _____ pavics-sdi-fix-esgf-dap/docs/source/notebooks/esgf-dap.ipynb::Cell 0 _____
22:44:05  Notebook cell execution failed
22:44:05  Cell 0: Cell outputs differ
22:44:05  
22:44:05  Input:
22:44:05  from pyesgf.search import SearchConnection
22:44:05  
22:44:05  # Create a connection for distributed search on ESGF nodes.
22:44:05  conn = SearchConnection("https://esgf.ceda.ac.uk/esg-search", distrib=True)
22:44:05  
22:44:05  # Launch a search query.
22:44:05  # Here we're looking for any variable related to humidity within the CMIP6 SSP2-4.5 experiment.
22:44:05  # Results will be stored in a dictionary with keys defined by the `facets` argument.
22:44:05  ctx = conn.new_context(
22:44:05      project="CMIP6",
22:44:05      experiment_id="ssp245",
22:44:05      query="humidity",
22:44:05      facets="variable_id,source_id",
22:44:05  )
22:44:05  
22:44:05  print("Number of results: ", ctx.hit_count)
22:44:05  print("Variables related to humidity: ")
22:44:05  ctx.facet_counts["variable_id"]
22:44:05  
22:44:05  Traceback:
22:44:05  Missing output fields from running code: {'evalue', 'ename'}
22:44:05  
22:44:05  _____ pavics-sdi-fix-esgf-dap/docs/source/notebooks/esgf-dap.ipynb::Cell 5 _____
22:44:05  Notebook cell execution failed
22:44:05  Cell 5: Cell execution caused an exception
22:44:05  
22:44:05  Input:
22:44:05  # Open the opendap link with xarray
22:44:05  import xarray as xr
22:44:05  
22:44:05  ds = xr.open_dataset(agg.opendap_url)
22:44:05  ds
22:44:05  
22:44:05  Traceback:
22:44:05  
22:44:05  ---------------------------------------------------------------------------
22:44:05  KeyError                                  Traceback (most recent call last)
22:44:05  File /opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/file_manager.py:201, in CachingFileManager._acquire_with_cache_info(self, needs_lock)
22:44:05      200 try:
22:44:05  --> 201     file = self._cache[self._key]
22:44:05      202 except KeyError:
22:44:05  
22:44:05  File /opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/lru_cache.py:55, in LRUCache.__getitem__(self, key)
22:44:05       54 with self._lock:
22:44:05  ---> 55     value = self._cache[key]
22:44:05       56     self._cache.move_to_end(key)
22:44:05  
22:44:05  KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('http://vesg.ipsl.upmc.fr/thredds/dodsC/CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r1i1p1f1.Amon.huss.gr.huss.20190119.aggregation.1',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]
22:44:05  
22:44:05  During handling of the above exception, another exception occurred:
22:44:05  
22:44:05  OSError                                   Traceback (most recent call last)
22:44:05  Cell In[6], line 4
22:44:05        1 # Open the opendap link with xarray
22:44:05        2 import xarray as xr
22:44:05  ----> 4 ds = xr.open_dataset(agg.opendap_url)
22:44:05        5 ds
22:44:05  
22:44:05  File /opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/api.py:539, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, backend_kwargs, **kwargs)
22:44:05      527 decoders = _resolve_decoders_kwargs(
22:44:05      528     decode_cf,
22:44:05      529     open_backend_dataset_parameters=backend.open_dataset_parameters,
22:44:05     (...)
22:44:05      535     decode_coords=decode_coords,
22:44:05      536 )
22:44:05      538 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
22:44:05  --> 539 backend_ds = backend.open_dataset(
22:44:05      540     filename_or_obj,
22:44:05      541     drop_variables=drop_variables,
22:44:05      542     **decoders,
22:44:05      543     **kwargs,
22:44:05      544 )
22:44:05      545 ds = _dataset_from_backend_dataset(
22:44:05      546     backend_ds,
22:44:05      547     filename_or_obj,
22:44:05     (...)
22:44:05      555     **kwargs,
22:44:05      556 )
22:44:05      557 return ds
22:44:05  
22:44:05  File /opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/netCDF4_.py:555, in NetCDF4BackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, format, clobber, diskless, persist, lock, autoclose)
22:44:05      534 def open_dataset(
22:44:05      535     self,
22:44:05      536     filename_or_obj,
22:44:05     (...)
22:44:05      551     autoclose=False,
22:44:05      552 ):
22:44:05      554     filename_or_obj = _normalize_path(filename_or_obj)
22:44:05  --> 555     store = NetCDF4DataStore.open(
22:44:05      556         filename_or_obj,
22:44:05      557         mode=mode,
22:44:05      558         format=format,
22:44:05      559         group=group,
22:44:05      560         clobber=clobber,
22:44:05      561         diskless=diskless,
22:44:05      562         persist=persist,
22:44:05      563         lock=lock,
22:44:05      564         autoclose=autoclose,
22:44:05      565     )
22:44:05      567     store_entrypoint = StoreBackendEntrypoint()
22:44:05      568     with close_on_error(store):
22:44:05  
22:44:05  File /opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/netCDF4_.py:384, in NetCDF4DataStore.open(cls, filename, mode, format, group, clobber, diskless, persist, lock, lock_maker, autoclose)
22:44:05      378 kwargs = dict(
22:44:05      379     clobber=clobber, diskless=diskless, persist=persist, format=format
22:44:05      380 )
22:44:05      381 manager = CachingFileManager(
22:44:05      382     netCDF4.Dataset, filename, mode=mode, kwargs=kwargs
22:44:05      383 )
22:44:05  --> 384 return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
22:44:05  
22:44:05  File /opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/netCDF4_.py:332, in NetCDF4DataStore.__init__(self, manager, group, mode, lock, autoclose)
22:44:05      330 self._group = group
22:44:05      331 self._mode = mode
22:44:05  --> 332 self.format = self.ds.data_model
22:44:05      333 self._filename = self.ds.filepath()
22:44:05      334 self.is_remote = is_remote_uri(self._filename)
22:44:05  
22:44:05  File /opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/netCDF4_.py:393, in NetCDF4DataStore.ds(self)
22:44:05      391 @property
22:44:05      392 def ds(self):
22:44:05  --> 393     return self._acquire()
22:44:05  
22:44:05  File /opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/netCDF4_.py:387, in NetCDF4DataStore._acquire(self, needs_lock)
22:44:05      386 def _acquire(self, needs_lock=True):
22:44:05  --> 387     with self._manager.acquire_context(needs_lock) as root:
22:44:05      388         ds = _nc4_require_group(root, self._group, self._mode)
22:44:05      389     return ds
22:44:05  
22:44:05  File /opt/conda/envs/birdy/lib/python3.8/contextlib.py:113, in _GeneratorContextManager.__enter__(self)
22:44:05      111 del self.args, self.kwds, self.func
22:44:05      112 try:
22:44:05  --> 113     return next(self.gen)
22:44:05      114 except StopIteration:
22:44:05      115     raise RuntimeError("generator didn't yield") from None
22:44:05  
22:44:05  File /opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/file_manager.py:189, in CachingFileManager.acquire_context(self, needs_lock)
22:44:05      186 @contextlib.contextmanager
22:44:05      187 def acquire_context(self, needs_lock=True):
22:44:05      188     """Context manager for acquiring a file."""
22:44:05  --> 189     file, cached = self._acquire_with_cache_info(needs_lock)
22:44:05      190     try:
22:44:05      191         yield file
22:44:05  
22:44:05  File /opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/file_manager.py:207, in CachingFileManager._acquire_with_cache_info(self, needs_lock)
22:44:05      205     kwargs = kwargs.copy()
22:44:05      206     kwargs["mode"] = self._mode
22:44:05  --> 207 file = self._opener(*self._args, **kwargs)
22:44:05      208 if self._mode == "w":
22:44:05      209     # ensure file doesn't get overridden when opened again
22:44:05      210     self._mode = "a"
22:44:05  
22:44:05  File src/netCDF4/_netCDF4.pyx:2463, in netCDF4._netCDF4.Dataset.__init__()
22:44:05  
22:44:05  File src/netCDF4/_netCDF4.pyx:2026, in netCDF4._netCDF4._ensure_nc_success()
22:44:05  
22:44:05  OSError: [Errno -77] NetCDF: Access failure: b'http://vesg.ipsl.upmc.fr/thredds/dodsC/CMIP6.ScenarioMIP.IPSL.IPSL-CM6A-LR.ssp245.r1i1p1f1.Amon.huss.gr.huss.20190119.aggregation.1'

[huard]

Thanks ! Will look into it.

[huard]

Runs on alpha

[tlogan2000]

LGTM