Proposed APIs

[khider]

class LiPD:

The object should behave like a dictionary so we can still access the various keys as needed.

• read
  • Read from a LiPD file or directory stored locally. Read from a URL (following same protocol as pandas.read_csv or json.load)
  • priority: very high - basic functionality
  • replaces lipd.readLipd()
• append
  • Append a LiPD file or a directory of LiPD file to the existing LiPD object
  • priority - high
  • This function wasn't supported in the previous utilities explicitly but it was easy to do so with the dictionary structure
• readExcel
  • Create a LiPD file from an excel template
  • priority - high
  • Replaces lipd.readLipd()/lipd.excel(). We were planning to no longer support this functionality and have researchers use the playground instead. But it seems to be pretty popular.
  • Will utilize the existing doi functionalities to get updated pub info
• readNoaa
  • Create a LiPD file from a NOAA template.
  • priority - medium [In scope of PaleoCube]
  • would replace lipd.readNoaa(). The functionality in the LiPD utils tries to read the metadata from the file, resulting in low accuracy. However, NOAA has a new API we can use for better accuracy and we only need to lift the values. Student worked on this last summer. Study ID13000 and above are templated.
  • Will utilize the existing doi functionalities to get updated pub info
• to_xarrayDataset
  • Create an xarray dataset object from the LiPD file.
  • priority - medium [in scope of PaleoCube]
  • New functionality
• to_pandasDataFrame
  • Create a pandas dataframe
  • priority-medium [in scope of paleocube]
  • Requires time extension to pandas.
• to_NOAA
  • convert to a NOAA templated file
  • Priority low - Not in scope
  • Need to replace this functionality for the playground eventually.
• validate
  • validate a lipd file
  • priority - very high
  • replaces lipd.validate(). Needed to make sure that LiPD object can be used by Pyleoclim and other tools
• addEnsemble
  • Add an ensemble table to a new or existing model (two modes: paleo and chron)
  • priority - medium
  • Replaces lipd.addensemble() - This function is written but no longer available from the main API. Should be done from a numpy array
• write
  • write out to file
  • priority - very high
  • replaces lipd.write()
• show
  • Show some metadata for the LiPD files
  • priority - high
  • replaces lipd.showMetadata(). Some recent feedback suggested that it wasn't as useful
  • Should include level of show (things that most people want to look at: coordinates, archive, variables) vs all.
• showEnsemble
  • Show whether the files have ensembles attached to them and granularity (how many models/tables)
  • priority - medium
  • New functionality. Would make coding around a little bit easier. Pyleoclim has something similar when more than one table is available and need to show the user so they can choose which ensemble to play with at a time.
• getCsv
  • Get the name of the csv files associated with LiPD object
  • Priority: very high. This supports all the ensemble functionalities in Pyleoclim under the hood.
  • replaces lipd.getCSV()

class LipdTimeSeries

The object should still behave like a dictionary for various key access

• extract
  • Extract all timeseries from the LiPD file
  • priority - very high -- all functionalities are built around this
  • replaces extractTS()
  • To do: if multiple "age" or "year" are present; ask a user which one should default to key "age"; "year". Right now, the keys are not necessarily standard, which makes it hard to code around.
• collapse
  • Collapse timeseries back into the LiPD object
  • priority - low -- I've never used this functionality at all.
  • replaces lipd.collapse()
• filter
  • Filter the timeseries with some specific criteria. Returns a new Timeseries Object. The filtering capability should also include unique TS IDs.
  • priority - high
  • replaces lipd.filterTs()
• query
  • Find the indices corresponding to the filtering
  • priority - medium/high or None. Should we collapse filter and query into one function? Does indexing really matter if a new object is created?
  • replaces lipd.queryTs() .
• append
  • Append two timeseries objects
  • priority - high
  • New functionality. Needed since we won't have dictionaries anymore. Will make it easier to create MS object in Pyleoclim.
• show
  • Show the metadata for the timeseries
  • priority - high. Would really help with coding.
  • I would include levels of "show". Pyleoclim shows very basic info (dataset name + archive + proxy) which is sufficient for filtering (especially if combined with unique TSID). show(level='all') would print out everything.
• assignEnsemble
  • Extract and assign ensemble tables to timeseries. If more than one ensemble is available, then duplicate entry and assign the second table. (warn the user you are doing so).
  • Priority - medium
  • We do it under the hood in Pyleoclim but this should be a LiPD utils functionality. Once this is done, would remove the need for getCsv()
• to_pandasSeries
  • Extract individual timeseries to a pandas Series object
  • Priority - medium [in scope of PaleoCube]

[CommonClimate]

Can we use this opportunity to rethink the terminology of the horribly confusing "timeseries object" that is nothing like a pandas or pyleoclim Series?
In terms of export functions, I vote for keeping names short:

• to_pandasDataframe --> to_pandas (is to_pandasSeries really needed? I am not sure what the use case is)
• to_xarrayDataset --> to_xarray

[khider]

1. I'm all for it but that's a bigger decision since it affects LiPD. We might want to keep the terminology consistent with the R utils too
2. Yes, we can simplify to pandas/xarray as long as we understand that under the hood, a lipd file will go to a data frame and a timeseries will go to Series. Series are one column data frame, which is what we have when we expand into TS object. Same with the xarray distinction between dataset and data array.

[CommonClimate]

Points taken!

1. Perhaps an intermediate solution is to keep the TSO terminology as it is within the LiPD utilities (R and Python), but phrase things differently at the Pyleoclim API level so users don't get endlessly confused between LiPDSeries and LiPD "timeseries" objects. You have to admit that's asking for trouble.... Perhaps Pyleoclim LiPDSeries should be AugmentedSeries to emphasize that it is more metadata-rich? Or PaleoSeries, perhaps? Something along those lines.
2. I understand the distinction between Series/Dataframe and DataArray/Dataset and it should be clear from the context, as long as the documentation is explicit on this point.

[khider]

1. Decision was to use LipdTimeSeries. The confusing part of Pyleoclim will be solved with the new approach since to_tso would become obsolete. I'm not keen on renaming fundamental objects in Pyleoclim. But once we have a definition of LipdTimeSeries, pyleoclim.LipsSeries will become a subset of lipd.LipdTimeSeries and pyleoclim.Series.

[CommonClimate]

That sounds great. The to_tso() method was always deeply confusing to me, so I won't be sad to see it go.

I'm sorry I had to leave the meeting to go teach. Did you get through all the topics you wanted to discuss? It seems like in a future meeting we should discuss the alignment of pyLipd to the LinkedEarth ontology.

[CommonClimate]

Example of use case from a collection of LiPD files (here subsumed under a pkl file) to pandas dataframe: https://fzhu2e.github.io/cfr/notebooks/proxy-ops.html#Create-a-ProxyDatabase-and-load-from-the-pandas-DataFrame