Programmatic access to the IOOS Core Variables is not easy

[kwilcox]

Should be abstracted to a library or well documented helper function

[jkupiec]

Great issue title![?]

On Tue, Jul 29, 2014 at 9:25 AM, Kyle Wilcox [email protected]
wrote:

Should be abstracted to a library or well documented helper function

—
Reply to this email directly or view it on GitHub
#128.

[hdean83]

@kwilcox can you elaborate on that? I wrote a notebook for ioos core variable vocabularies - it might be a start - it's in Theme 1

[kwilcox]

The average user doesn't want to deal with RDF or SPARQL. We need to abstract that out so they can just do: get_ioos_core_variables()

[emiliom]

@kwilcox, can you spell out what you're trying to accomplish?

Should get_ioos_core_variables() just return a list of all core variables, plus their MMI vocabulary attributes and the MMI vocabulary where the term is found? (though it could be found in more than one valid and accepted vocabulary). Or are you looking for more?

@dpsnowden and @robragsdale , is this still an up-to-date representation of the core variables?
http://mmisw.org/ont/ioos/core_variable
This vocabulary has always needed one or more additional attributes; the "definition" is currently barely more than a better formatted version of the terms, and not of much extra help. Besides, some of these variables are really broad concepts (eg, contaminants) rather than narrow, directly measured, unambiguous variables. Some MMI mappings do exist between this vocabulary and more useful vocabularies (IOOS parameters, CF standard names, etc).

I think @hdean83 has SPARQL examples for such "narrow match" queries against core variables; if not, I have an example; see below the section heading "Narrow Match of IOOS Parameter Vocabulary terms from IOOS Core Variable dissolved_nutrients term"

If you just want to pull in the attributes from one specific vocabulary, my code here may be of some use; see the section at the end, below the heading "Test the validity of several classifiers (vocabulary terms) against MMI vocabularies". Though what it currently does is only test and pull attributes for the vocabulary term that is passed.

[kwilcox]

I'm only looking to make it easier for people to gain access to the IOOS Core Variables list.

I recently needed a list of them to use to use in a free text search against CSW servers. I came up with this (see cells 3, 4 and 5):
http://nbviewer.ipython.org/github/kwilcox/system-test/blob/t1sbb/Theme_1_Baseline/Scenario_1B_CoreVariable_Strings/Scenario_1B_CoreVariable_Strings.ipynb

That is much too complicated for someone new to programming, python, RDF, SPARQL, MMI, etc. to come up with on their own. Even a CSV download link on the IOOS homepage could be parsed. My only other option was to type them all in manually from a Powerpoint presentation I found from a few years ago.

Is there an easier way to gain access to the list of Core Variables (name, definition, uri)? My method seems verbose. Looking for suggestions here...

[dpsnowden]

@emiliom This is the current list of core variables. http://mmisw.org/ont/ioos/core_variable. It hasn't changed for several years.

I'm a little embarrassed that the definitions of the core variables are just the phrases themselves without the underscores. I don't know if actual precise definitions exist or not. @robragsdale have you ever searched through the IOOS development plan 2002 for actual definitions?

[robragsdale]

@dpsnowden I've searched the IOOS Developement Plan (2006) and Airlie House Report and neither defined the core variables.

[robragsdale]

For some of the core variables, they are more like buckets, which was probably the intent. So I think that we as a community jumped to the specific terms that filled those buckets. .

[emiliom]

@kwilcox , I totally agree that a simple, csv-like version would be very useful, specially for such a small vocabulary. I don't know of a way to get a csv table listing of the vocabulary directly from MMI. @carueda, is there one? The only machine-friendly (and I'll lump well-formed csv as machine friendly) formats I've seen for an entire vocabulary are RDF and N3, at these URL's:
http://mmisw.org/ont?form=rdf&uri=http://mmisw.org/ont/ioos/core_variable
http://mmisw.org/ont?form=n3&uri=http://mmisw.org/ont/ioos/core_variable

In the short term, It'd be easy to create a csv version and upload it to github/ioos. core_variable is not going to change frequently, so that should be fine. But in the long run it'd be awesome if MMI supported that directly, based on, say, a new "form" argument like "form=csvtable" (ie, vocabulary table only, no vocabulary metadata). That would have widespread benefits. @dpsnowden , here are my 2.5 votes for sending funds in Carlos' and John's way to support this enhancement :)

[graybeal]

The cvs table enhancement should be do-able, especially if the 2.5 votes is enough for funding!

Regarding the definitions and weaknesses of core variables, I can attest that these issues were noted when the core variables were created. (They were not created by developers, let alone information architects, so, stuff happens. :-) )

An interim suggestion for making it easy to obtain the list to core variables is to add that SPARQL query (that produces a something-separated list) to one of the examples pages. It isn't as slick as CSV but at least it can produce CSV, or something like it.