ncSOS vs 52N on IOOS Catalog Map

[ebridger]

I have a question about the IOOS Catalog display of regional SOS services on the Catalog map, which has been brought to my attention.
http://catalog.ioos.us/map/
Choose NERACOOS and SOS Service Type.
All results show “UNKNOWN” or “Attribute not present in source data”.

This effects all the regions which are using the ncSOS plugin for THREDDS, 6 out of 11.

NERACOOS, PacIOOS, CARICOOS, SCCOOS, MARACOOS.
Regions using 52 North have no such problem.

NERACOOS is using TDS 1.3 and ncSOS 1.1

Is there an explanation for this? Is there a solution?

[kknee]

@ebridger we are looking into the issue.

[ebridger]

@kknee Any progress on this issue?

[cheryldmorse]

ncSOS is not finding the variables that it is expecting in order to set the attributes in the response. Below is a mapping of what variables/ attributes map to the response:

If the NetCDF file contains more than one station then the following variables are needed:

Variable Name	Response Element
platform_short_name	<sml:identifier name="shortName">
platform_long_name	<sml:identifier name="longName">
platform_wmo_code	<sml:identifier name="wmoId">

If the NetCDF file only contains one station then the following attributes to the variable that has the attribute 'cf_role':

Variable Attributes	Response Element
short_name	<sml:identifier name="shortName">
long_name	<sml:identifier name="longName">
wmo_code	<sml:identifier name="wmoId">

All files should also contain the following global attributes:

Global Attributes	Response Element
platform_type	<sml:classifier name = "platformType"> <sml:value> Attribute value </sml:value></sml:classifier>
operator_sector	<sml:classifier name = "operatorSector"><sml:value> Attribute value </sml:value></sml:classifier>
publisher	<sml:classifier name = "publisher"> <sml:value> Attribute value </sml:value></sml:classifier>
sponsor	<sml:classifier name = "sponsor"> <sml:value> Attribute value </sml:value></sml:classifier>

[lukecampbell]

So data providers need to fill in those attributes for each variable, and each dataset's global attributes for it to pick these up correct?

[dpsnowden]

I believe these are documented on the wiki. I haven't checked that the list above is 100% consistent with the wiki page but I hope it is.

@ebridger let us know if this answered your questions. Once the data shows up on the catalog we should close this ticket.

[ebridger]

Yes. Confirmed that the global attribute changes can be made via ncml, which is nice. Will be a few days before I get these changes onto our production THREDDS.

[ebridger]

@cheryldmorse Your suggestions, implemented with ncml have not fixed the problem in the IOOS catalog. This has been very hard to test, since the current IOOS Catalog SOS map display for those using ncSOS does not allow easy identification of any particular buoy. All our ncSOS's are for single buoys. ncSOS always, also adds a network-all entry. Adding global shortName, longName does work for the network-all DescribeSensor response but does not work for the Station procedures. I've tried multiple ncml permutations.

While all the SOS's have added attributes I picked one particular file for more extensive testing Buoy B01 Accelerometer (wave mesaurements) at this THREDDS SOS.

Here are some example DescribeSensor requests:
network-all
B01 station

The network-all response finds shortName, etc. showing that the ncml is working but the station, while picking up some of the added attributes, e.g. platform_type, etc. still fails for shortName, longName:

 <sml:identifier name="shortName">
<sml:Term definition="http://mmisw.org/ont/ioos/definition/shortName">
<sml:value>Attribute not present in source data</sml:value>
</sml:Term>
</sml:identifier>

I suspect that this is why the IOOS Catalog map is still displaying garbage in the drop down lists.

[cheryldmorse]

@ebridger You're right that the "Attribute Not present in source data" is the reason that the IOOS Catalog map isn't displaying the correct data. It looks like I made a mistake when I said that you could add a global attribute 'shortName' to solve the issue. You should actually be adding that attribute 'short_name' to the variable that has the 'cf_role' attribute. Can you try updating this dataset: http://www.neracoos.org/thredds/dodsC/UMO/DSG/SOS/B01/Accelerometer/HistoricRealtime/Agg.ncml.html and changing the 'shortName' attribute for variable 'platform' to 'short_name'. Please let me know if that works.

[abirger]

@cheryldmorse , it looks like the DescribeSensor section of the wiki's NetCDF Mapping page should be corrected in reference to a station metadata.

[ebridger]

@cheryldmorse Sorry but adding 'short_name', 'long_name' attributes to the station variable, which has cf_role=timeseries_id did not work. Even tried renaming the station variable to platform and other tests did not work. I run these tests on my laptop THREDDS not on the production server. I think I am going to try to add various attributes to the underlying NetCDF file to see if that works.

[cheryldmorse]

@ebridger I'm looking into it. I downloaded the file and will let you know what variable/attribute is needed in order to get it working.

[dpsnowden]

Can I just ask that whatever the resolution, it's documented on the wiki? Also, I like the table presentation above more than the bulleted list presentation on the wiki.

Not to throw too much of a monkey wrench into this....Some of the mappings rely on non-standard netCDF attributes. There are no "short_name" attributes in ACDD for example or in the NODC (NCEI) templates which are supposed to serve as our guidelines for creating netCDF files.

[cheryldmorse]

@ebridger A bug in ncSOS was causing it to parse your netCDF file as a file that contained more than one station. Release v1.1.1 should fix the issue.

[ebridger]

@cheryldmorse Thanks Cheryl. This has indeed fixed the issue. Confirmed on my local TDS. Hope to get this to our production server soon and then we can see if it fixes the IOOS Catalog Map.

[abirger]

@dpsnowden , there also are no "operator_sector" or "sponsor" global attributes in the templates; however, we need them to populate the corresponding SOS element values. The variables/attributes that got extended are RECOMMENDED in the original templates anyway; I guess that it gives us some right to stretch them a bit making a specific IOOS Template Profile.

@cheryldmorse , in the end, where does ncSOS get the station ID for the offering URN? I am kind of confused now.

[ebridger]

I may have spoken too soon. While the NERACOOS display in the IOOS Catalog Map is much improved the shortName, longName, short_name, long_name global or station attributes, via ncml, still are not showing up in the DescribeSensor Station response. Only in the DescribeSensor network-all response. I will have more details soon. Need to test if the actual station variable attributes in the underlying NetCDF make a difference. Currently the NetCDF file station variable, with cf_role=timeseries has both short_name and long_name set to 'B01' . But the global shortName is "B01 Western Maine Shelf Waves" does not show up.

[cheryldmorse]

@ebridger Do you have an example Describe sensor request that shows the issue?

[ebridger]

@cheryldmorse The B01 accelerometer (waves file), same links as above, shows the issue.

B01 network-all
B01 station

The underlying ncSOS is at http://www.neracoos.org/thredds/UMO_SOS_historical_realtime_agg.html?dataset=SOS_DSG_B01_ACCELEROMETER_Historic_Realtime_Agg

But I did find a non NCML fix.
This ncSOS is actually an aggregation of 2 files. An historic and realtime file. By editing both of the station variable long_name attributes in the NetCDF files DescribeSensor for the station works as expected.

I'll probably go ahead and implement this at some point then watch the IOOS Catalog Map to see if things improve. Will be a few days before I can do this.

[ebridger]

@cheryldmorse I did finally update all my underlying NetCDF files, adding long_name and short_name attributes to the station variable. Just wanted to confirm that this did indeed work B01 Station

Also confirming that I could not get this to work using NCML, not sure why.
http://catalog.ioos.us/map/

[cheryldmorse]

@ebridger Is everything working for you now with the updated datasets? Can this issue be closed?

[ebridger]

@cheryldmorse I'd say yes.
The catalog map is improved but there is still quite a bit of junk "UNKNOWN" entries in the drop down list but my guess is this is more of a catalog issue and I probably just need it to be cleared and a fresh WAF harvest done.
There is also @abirger email re: NetCDF attributes to IOOS SOS Templates crosswalk So there may well be a need to redo many of our NetCDF attributes.