Filter catalog entries by version number

[sgyeager]

Data in some catalogs (e.g., CMIP6-GLADE) has multiple 'version' entries (e.g., experiment_id=='omip1'). This complicates data aggregation and/or necessitates extra effort to identify and extract only particular versions. Would it be possible to add an argument to the search method to identify and select only the latest version of a dataset?

[andersy005]

thank you for your patience, @sgyeager!

Would it be possible to add an argument to the search method to identify and select only the latest version of a dataset?

To identify the latest version of a dataset, we would have to standardize what data is contained in the catalog's version attribute. CMIP6 specifies the version as a date string. It is unlikely that the other datasets use date strings to denote version. Consequently, it is complicated to develop a generic intake-esm solution that would work for any catalog.

Perhaps that this issue can be addressed in one of two ways:

• Option 1) Ask the person building the catalog to filter out the outdated versions before publishing the catalog
• Option 2) Filter out outdated versions yourself by modifying the pandas dataframe directly and re-attaching it to the catalog object. we've put together an example notebook showing how to do this: https://intake-esm.readthedocs.io/en/stable/how-to/modify-catalog.html#use-case-1-complex-search-queries

Would any of these options work for you?