Is "variable the same name as dimension is a coordinate" part of CF?

[ChrisBarker-NOAA]

Question

In the xarray package, if a variable has the same name as a dimension, it is assumed to be a coordinate -- and xarray does a lot of special stuff with coordinates.

This works great for typical files: "time", "lat", "lon" and the like.

However, I'm finding a fair number of files in the wild where a variable with the same name as a dimension is not, in fact a coordinate. Or maybe it is, but a more complex one like a 2-D depth, or .....

(this comes up a lot in unstructured grids)

So: is xarray making an unreasonable assumption here? Or is that in fact a a part of CF, and I should complain to the providers about it?

[sethmcg]

It's recommended against, but not forbidden, so xarray is indeed making a bad assumption.

Section 5, paragraph 4:

We recommend that the name of a multidimensional coordinate variable should not match the name of any of its dimensions because that precludes supplying a coordinate variable for the dimension. This practice also avoids potential bugs in applications that determine coordinate variables by only checking for a name match between a dimension and a variable and not checking that the variable is one dimensional.

CF takes its definition of coordinate variables from the NetCDF Users Guide:

A coordinate variable is a one-dimensional variable with the same name as a dimension, which names the coordinate values of the dimension.

CF also says:

The use of coordinate variables is required for all dimensions that correspond to one dimensional space or time coordinates.

So under CF, if it corresponds to X, Y, Z, or T, a 1-D variable with the same name as a dimension must be a coordinate variable, and vice-versa.

But that doesn't hold for other kinds of dimensions. If you have a string-valued dimension (such as "model") of type char rather than string (the only option under netCDF-3), the coordinate variable will be 2-D, since you need a second dimension for the string length. In projected grids, lat & lon are 2-D, so they become auxiliary coordinate variables instead of regular ones. And coordinate variables aren't required at all for things like a length-2 bounds dimension.

[larsbarring]

Thanks Seth for an excellent summary! I think that this would be a perfect addition to the FAQ (after minor edits to remove the explicit reference to xarray).

[davidhassell]

I agree, great summary, Seth.

Not impacting on the original question, I wondered about:

The use of coordinate variables is required for all dimensions that correspond to one dimensional space or time coordinates.

This comes from section 1.4, but I don't think that it's always true - one-dimensional spatial coordinate information for a discrete axis is represented by an auxiliary coordinate variable (e.g. as used by DSGs, UGRID, etc.). This text is as old as the hills, having been in CF since version 1.0, a long time before DSGs and the like. I'd be inclined just to delete that sentence, changing the short paragraph to the even shorter:

There are two methods used to identify variables that contain coordinate data, they are either NUG-defined "coordinate variables", or else are identified by the coordinates attribute.

There's probably a better way than this first thought, though!

[JonathanGregory]

@davidhassell, I think rephrasing along the lines of your new sentence "There are two methods ..." would be helpful to do anyway, because the existing text in 1.4 does not explicitly say what the second method is, like you have. It is unclear what the quoted sentence "The use of ..." in 1.4 is supposed to mean, but I think it means that you shouldn't use an auxiliary coordinate variable (with dimension name and coordinate variable name not equal) when you could use a coordinate variable. In 1.4 it only applies to XYZT, but in the preamble of section 4 there is a possibly related statement, which is also somewhat unclear in meaning, "We strongly recommend that coordinate variables be used for all coordinate types whenever they are applicable," - but instead of what? I think it means, instead of auxiliary coordinate variables, again. If I am right, we should replace both of these sentences with the same, clearer, sentence.

One implication is that you shouldn't use a discrete axis when you could use a coordinate variable. According to sect 4.5, you can use a discrete axis when you have either an ordered list or an unordered collection of elements which does not correspond to any continuous coordinate variable. It is recommended to use a coordinate variable when the data values along the axis could be regarded as a monotonic function of a single continuous geophysical variable, I would say.

[JonathanGregory]

Thanks for the question and answer, @ChrisBarker-NOAA and @sethmcg. I agree that they would be useful to add to the FAQ.

[taylor13]

Regarding @ChrisBarker-NOAA 's original question, should we bring it to the attention of whoever might be interested in the xarray community that they are making an incorrect assumption? Seems like lots of scientists are moving to use of xarray in their analysis packages, so they should understand that they may be misinterpreting some coordinate information.

[ChrisBarker-NOAA]

Is xarray mamking an incorrect assumption?

I think the assumption is that:

"a variable with the same name as a dimension is a coordinate variable",

and from this conversation, that is correct, yes?

Or is there a more refined definition:

"a one dimensional variable with the same name as a dimension is a coordinate variable"

If so, then I'll need to confirm what xarray is doing now -- a year ago, it would try to make a coordinate out of a n-d variable with the same name, then fail to load the Dataset. Now it doesn't fail, but I'm not sure what it does do.

[sethmcg]

The latter. It's only 1-D variables with the same name as a dimension that are necessarily coordinate variables.

[ChrisBarker-NOAA]

Thanks -- I'll see what xarray is doing, and post an issue if it's not doing the right thing :-)

[DocOtak]

Check the xarray docs about Coordinates: https://docs.xarray.dev/en/stable/user-guide/data-structures.html#coordinates and their note about differing terminology from CF. I don't think I've seen xarray deviate from what I would expect for most inputs, also note that xarray has its own internal data model that does not support everything a netCDF4 file can do.

[DocOtak]

xarray will respect the "coordinates" attribute by default when reading a netCDF file, the decode_cf method has some details on how to control this. In the above linked docs, the example dataset has the scalar coordinate of reference_time

When xarray finds a multidimensional variable with the same name as a dimension, it will set it as a "non-dimension coordinate" (xarray terminology). These coordinates aren't as special as "dimension coordinates" in xarray, you cannot use them for indexing for example and only some functions (e.g. groupby) know what to do with these. They do persist as "coordinates" when you select other variables, but that's about it as far a special treatment.

[ChrisBarker-NOAA]

Thanks @DocOtak -- it sounds like xarray is, in fact, doing the right thing, though I'm going to make sure -- I thought I was seeing some odd behavior, I'll check on that.

I think there is a fundamental struggle because:

A) xarray and CF terminology is a bit different
B) xarray's data model doesn't cover the full complexity of coordinates types that CF does.

"When xarray finds a multidimensional variable with the same name as a dimension, it will set it as a "non-dimension coordinate" (xarray terminology)."

That does seem to be the right thing to do -- though I'm still a bit confused as to the implications of a non-dimension coordinate -- but that's a question for the xarray folks.

My original question has been answered -- should I close the discussion?
I'm not sure of the protocol for discussions, if this were an issue, I would close it ...

[DocOtak]

The way I personally like to think about this is, CF is defines a serialization format in netCDF (or the netCDF data model), and xarray has its own internal data model that is closely related, but really has its own history. I consider the separation to be xarray for doing compute or in memory, and CF for at rest on disk.

The actual implications for non-dimension coordinates in xarray is that the variable in question will be added to the .coords attribute of the xarray data structures (Dataset or DataArray). These persist when you select subsets of data or do other operations if xarray can figure out how to handle that.

As an example, I work with data that uses community preferred natural composite keys for keeping track of data, it's not really important what they are, but they are not dimension coordinates and are not 1-D. Using the CF conventions, I set these key variables to the coordinates attribute of the data variables they apply to. When xarray reads these data, it will put these variables into the .coords attribute. When I do more data manipulation or exploration, xarray just keeps these coordinates with anything I select.

[DocOtak]

Re what to do about this open discussion, I thought there was a way to mark an answer and "resolve" the discussion, but I don't see that option for me. perhaps you @ChrisBarker-NOAA as the question asker can do that? https://github.blog/changelog/2023-04-17-marking-a-threaded-comment-as-the-answer/

Marking something as the answer will pin it under the question in "green" so folks who find this question again can jump straight to the answer.

[JonathanGregory]

@DocOtak - please could you repeat that comment in https://github.com/orgs/cf-convention/discussions/313?

[JonathanGregory]

@ChrisBarker-NOAA asked "My original question has been answered -- should I close the discussion? I'm not sure of the protocol for discussions, if this were an issue, I would close it ..." I asked the same in discussion 313! Any thoughts? I tend to feel we don't need to close discussions.

[ChrisBarker-NOAA]

OK -- turns out I'm not done with the question :-)

It turns out there's some complexity with coordinates and UGRID.

For example, I'm looking right now at an output file from FVCOM, and it has (in part):

dimensions:
         ...
	node = 54120 ;
...
variables:
...
	float lon(node) ;
		lon:standard_name = "longitude" ;
		lon:long_name = "nodal longitude" ;
		lon:units = "degree_east" ;
	float lat(node) ;
		lat:standard_name = "latitude" ;
		lat:long_name = "nodal latitude" ;
		lat:units = "degree_north" ;

When I load this into xarray, I get these coordinates:

Coordinates:
    lon              (node) float32 216kB 237.0 237.0 237.0 ... 238.3 238.3
    lat              (node) float32 216kB 38.0 37.99 37.99 ... 38.16 38.16 38.16
...
  * time             (time) datetime64[ns] 200B 2024-05-31 ... 2024-06-01

So time is correct -- it's a proper time coordinate.

but lon and lat are the locations of the nodes of the grid -- not a coordinate. I presume xarray has done that because. lon and lat have standard_name of longitude and latitude. But I don't think those are coordinates. (Though I'm still confused about exactly what an xarray coordinate is ...)

So the question is -- is xarray making an incorrect assumption, or is this file non-compliant?
(they should use a different standard_name, or no standard_name ?

[DocOtak]

In this netCDF file, do any of the other variables have a coordinates attribute and if so, do they have the string "lat" and "lon" in them?

[ChrisBarker-NOAA]

Good catch -- yes, they do:

	float h(node) ;
		h:standard_name = "sea_floor_depth_below_geoid" ;
		h:long_name = "Bathymetry" ;
		h:units = "m" ;
		h:positive = "down" ;
		h:grid = "Bathymetry_Mesh" ;
		h:coordinates = "lat lon" ;
		h:type = "data" ;

Which means I'm an idiot -- those are indeed coordinate variables - in the CF sense, though still not sure of xarray sense (I think they are xarray coordintes, but not "indexing" coordinates ...)

Thanks!

This confusion (in my head anyway) comes from the UGRID convention -- for a UGRID, strictly speaking, you only need to know that a variable is defined "on the nodes" or "on the faces" (the "location" attribute).

But to keep things more compatible with CF, we specified that a variable should also have the CF-standard coordinates attribute. This was to facilitate some use by non-UGRID aware software -- in the above example, you could know what the sea_floor_depth_below_geoid at all the node locations, without understanding the mesh at all.

So I think it's all good.

Now I'm done :-)

[DocOtak]

Yes, these are "auxiliary coordinate" variables in the CF terminology, and "non-dimension coordinates" in xarray terminology. In the xarray repr the lack of asterisk next to the variable name is how these are distinguished.

One thing that caught me a few years ago is the xarray data model for the Dataset object doesn't allow for variables (DataArray) to have different sets of coordinates in the in memory data model, depending on dimensions. So things that do not have a coordinates attribute in the netCDF file will appear to have these coordinates when loaded into xarray. In your above file, all the variables that have the node dimension, will have lat and lon in their coordinates in xarray. There is an encoding attribute on each DataArray that keeps track of how the original data was actually serialized on disk. Opening a netCDF file in xarray and immediately exporting it again should result in the same structured netCDF, they do a lot of round trip tests in their CI for this.

[ChrisBarker-NOAA]

There is an encoding attribute on each DataArray that keeps track of how the original data was actually serialized on disk.

Yeah, I've seen that. I may have to get more familiar with that, as I'm working on code that will take a non-UGRID compliant dataset, and make it UGRID compliant -- so I may need to mess around with the encoding, too. (I have done a bit of that for time variables, so I have a vague idea where to start ...

[davidhassell]

This sort of case is where having a data model really helps :)

As @DocOtak said, whether lat, lon and height in the above example are logically auxiliary coordinates or auxiliary coordinate bounds depends on the context of the parent data variable. If that has a mesh location of "face" or "edge" then they are auxiliary coordinate bounds, but only if the location is "node" are they auxiliary coordinates.

In the face/edge case, it is allowed for for the auxiliary coordinate values to be missing - i.e. it's OK for that aspect of the cells to defined by only cell bounds (provided by the mesh nodes).

The CF data model definition of the auxiliary coordinate construct contains coordinate values, coordinate bounds, and any amount of descriptive properties, where either (but not both) of the values or bounds may be omitted. Omitting the cells' representative coordinate values is a special case only afforded to one-dimensional auxiliary coordinate constructs for discrete axes, which in practice usually means we have cell geometries or UGRID in play. In cases where there are only bounds, it is not possible to include the corresponding node_coordinate variables in the coordinates attribute. Section 5.9. Mesh Topology Variables says on this:

"The variables containing the coordinate values for cells indexed by the mesh topology are defined by the mesh topology variable but are equivalent to one-dimensional auxiliary coordinate variables, and so may also be provided by the data or domain variable’s coordinates attribute. Note that the mesh topology variable allows cell bounds to be provided without any cell coordinate values, via its node_coordinates attribute."