Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cdscan KeyError: 'height' #421

Open
mzelinka opened this issue Sep 2, 2020 · 6 comments
Open

cdscan KeyError: 'height' #421

mzelinka opened this issue Sep 2, 2020 · 6 comments
Labels
kind/bug Bug in the code.
Milestone

Comments

@mzelinka
Copy link

mzelinka commented Sep 2, 2020

Describe the bug
I can read in a netcdf file without issue, but when I try to read in data from a cdscan-generated xml that points to that same file, I get KeyError: 'height'.

To Reproduce
Steps to reproduce the behavior (on feedback2.llnl.gov):

  1. I am able to open this netcdf file without any problem:
    f=cdms2.open('/p/css03/esgf_publish/CMIP6/CMIP/KIOST/KIOST-ESM/abrupt-4xCO2/r1i1p1f1/Amon/tas/gr1/v20191106/tas_Amon_KIOST-ESM_abrupt-4xCO2_r1i1p1f1_gr1_185001-200012.nc')

  2. Generate an xml pointing to this file via cdscan:
    In [69]: import os

In [70]: os.system('cdscan -x temp.xml /p/css03/esgf_publish/CMIP6/CMIP/KIOST/KIOST-ESM/abrupt-4xCO2/r1i1p1f1/Amon/tas/gr1/v20191106/tas_Amon_KIOST-ESM_abrupt-4xCO2_r1i1p1f1_gr1_185001-200012.nc')
Finding common directory ...
Common directory: /p/css03/esgf_publish/CMIP6/CMIP/KIOST/KIOST-ESM/abrupt-4xCO2/r1i1p1f1/Amon/tas/gr1/v20191106/
Scanning files ...
/p/css03/esgf_publish/CMIP6/CMIP/KIOST/KIOST-ESM/abrupt-4xCO2/r1i1p1f1/Amon/tas/gr1/v20191106/tas_Amon_KIOST-ESM_abrupt-4xCO2_r1i1p1f1_gr1_185001-200012.nc
Setting reference time units to days since 1850-01-01
temp.xml written
Out[70]: 0

  1. Now, try to open this xml file:
    In [71]: f=cdms2.open('temp.xml')

KeyError Traceback (most recent call last)
in
----> 1 f=cdms2.open('temp.xml')

~/workdir2/miniconda3/envs/cdat82/lib/python3.7/site-packages/cdms2/dataset.py in openDataset(uri, mode, template, dods, dpath, hostObj)
570 dpath = head
571
--> 572 dataset = Dataset(uri, mode, datanode, None, dpath)
573 return dataset
574

~/workdir2/miniconda3/envs/cdat82/lib/python3.7/site-packages/cdms2/dataset.py in init(self, uri, mode, datasetNode, parent, datapath)
762 # Collect named children (having attribute 'id') into dictionaries
763 if datasetNode is not None:
--> 764 coordsaux = self.convention.getDsetnodeAuxAxisIds(datasetNode)
765
766 for node in list(datasetNode.getIdDict().values()):

~/workdir2/miniconda3/envs/cdat82/lib/python3.7/site-packages/cdms2/convention.py in getDsetnodeAuxAxisIds(self, dsetnode)
147 # Check the rank, don't include if not 1D or 2D (e.g.,
148 # scalar coordinate)
--> 149 domnode = dsetdict[item].getDomain()
150 if domnode.getChildCount() not in [1, 2]:
151 continue

KeyError: 'height'

Expected behavior
I expected no problem opening this xml file, just as I experienced no problem opening the netcdf that it points to.

Environment Information

`conda info`

(cdat82) -bash-4.1$ conda info

     active environment : cdat82
    active env location : /work/zelinka1/workdir2/miniconda3/envs/cdat82
            shell level : 2
       user config file : /work/zelinka1/.condarc
 populated config files : /work/zelinka1/.condarc
          conda version : 4.8.3
    conda-build version : not installed
         python version : 3.7.4.final.0
       virtual packages : __glibc=2.12
       base environment : /work/zelinka1/workdir2/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/cdat/label/v82/linux-64
                          https://conda.anaconda.org/cdat/label/v82/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /work/zelinka1/workdir2/miniconda3/pkgs
                          /work/zelinka1/.conda/pkgs
       envs directories : /work/zelinka1/workdir2/miniconda3/envs
                          /work/zelinka1/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.3 requests/2.24.0 CPython/3.7.4 Linux/2.6.32-754.28.1.el6.x86_64 rhel/6.10 glibc/2.12
                UID:GID : 25073:1026
             netrc file : None
           offline mode : False

`conda config --show-sources`

(cdat82) -bash-4.1$ conda config --show-sources
==> /work/zelinka1/.condarc <==
ssl_verify: False
channel_priority: strict
channels:
  - cdat/label/v82
  - conda-forge
  - defaults

`conda list --show-channel-urls`

(cdat82) -bash-4.1$ conda list --show-channel-urls
# packages in environment at /work/zelinka1/workdir2/miniconda3/envs/cdat82:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       0_gnu    conda-forge
alabaster                 0.7.12                     py_0    conda-forge
astroid                   2.3.3                    py37_1    conda-forge
attrdict                  2.0.1                      py_0    conda-forge
attrs                     19.3.0                     py_0    conda-forge
autopep8                  1.5                        py_0    conda-forge
babel                     2.8.0                      py_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
basemap                   1.2.1            py37hd759880_1    conda-forge
beautifulsoup4            4.8.2                    py37_0    conda-forge
bleach                    3.1.0                      py_0    conda-forge
bokeh                     1.4.0                    py37_0    conda-forge
bzip2                     1.0.8                h516909a_2    conda-forge
ca-certificates           2020.6.20            hecda079_0    conda-forge
cartopy                   0.17.0          py37hd759880_1006    conda-forge
cdat                      8.2                           3    cdat/label/v82
cdat_info                 8.2                        py_7    conda-forge
cdms2                     3.1.4                    pypi_0    pypi
cdp                       1.6.0                      py_0    conda-forge
cdtime                    3.1.2            py37h7eb8c7e_6    conda-forge
cdutil                    8.2                        py_2    cdat/label/v82
certifi                   2020.6.20        py37hc8dfbb8_0    conda-forge
cffi                      1.13.2           py37h8022711_0    conda-forge
cftime                    1.2.1            py37h03ebfcd_0    conda-forge
chardet                   3.0.4                 py37_1003    conda-forge
cibots                    0.3                   py37_1001    conda-forge
click                     7.0                        py_0    conda-forge
climlab                   0.7.6            py37hc7d476f_1    conda-forge
cloudpickle               1.3.0                      py_0    conda-forge
cryptography              2.8              py37h72c5cf5_1    conda-forge
curl                      7.68.0               hf8cf82a_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
cytoolz                   0.10.1           py37h516909a_0    conda-forge
dask                      2.10.1                     py_0    conda-forge
dask-core                 2.10.1                     py_0    conda-forge
dbus                      1.13.6               he372182_0    conda-forge
decorator                 4.4.1                      py_0    conda-forge
defusedxml                0.6.0                      py_0    conda-forge
distarray                 2.12.2                     py_1    conda-forge
distributed               2.10.0                     py_0    conda-forge
docopt                    0.6.2                      py_1    conda-forge
docutils                  0.16                     py37_0    conda-forge
dv3d                      8.2                        py_0    cdat/label/v82
entrypoints               0.3                   py37_1000    conda-forge
eofs                      1.4.0                      py_0    conda-forge
esmf                      7.1.0             hcda5390_1008    conda-forge
esmpy                     7.1.0            py37h5ca1d4c_3    conda-forge
expat                     2.2.9                he1b5a44_2    conda-forge
ffmpeg                    4.2                  h167e202_0    conda-forge
flake8                    3.7.9                    py37_0    conda-forge
fontconfig                2.13.1            h86ecdb6_1001    conda-forge
freetype                  2.10.0               he983fc9_1    conda-forge
fsspec                    0.6.2                      py_0    conda-forge
future                    0.18.2                   py37_0    conda-forge
g2clib                    1.6.0                hf3f1b0b_9    conda-forge
genutil                   8.2              py37hc1659b7_3    conda-forge
geos                      3.7.2                he1b5a44_2    conda-forge
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
ghostscript               9.22              hf484d3e_1001    conda-forge
glib                      2.58.3          py37h6f030ca_1002    conda-forge
gmp                       6.2.0                he1b5a44_1    conda-forge
gnutls                    3.6.5             hd3a4fd2_1002    conda-forge
gst-plugins-base          1.14.5               h0935bb2_2    conda-forge
gstreamer                 1.14.5               h36ae1b5_2    conda-forge
hdf4                      4.2.13            hf30be14_1003    conda-forge
hdf5                      1.10.5          nompi_h3c11f04_1104    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
idna                      2.8                   py37_1000    conda-forge
imagesize                 1.2.0                      py_0    conda-forge
importlib_metadata        1.5.0                    py37_0    conda-forge
ipykernel                 5.1.4            py37h5ca1d4c_0    conda-forge
ipython                   7.12.0           py37h5ca1d4c_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.5.1                      py_0    conda-forge
isort                     4.3.21                   py37_0    conda-forge
jasper                    1.900.1           h07fcdf6_1006    conda-forge
jedi                      0.16.0                   py37_0    conda-forge
jeepney                   0.4.2                      py_0    conda-forge
jinja2                    2.11.1                     py_0    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
jsonschema                3.2.0                    py37_0    conda-forge
jupyter                   1.0.0                      py_2    conda-forge
jupyter_client            5.3.4                    py37_1    conda-forge
jupyter_console           6.0.0                      py_0    conda-forge
jupyter_core              4.6.2                    py37_0    conda-forge
keyring                   21.1.0                   py37_0    conda-forge
kiwisolver                1.1.0            py37hc9558a2_0    conda-forge
krb5                      1.16.4               h2fd8d38_0    conda-forge
lame                      3.100             h14c3975_1001    conda-forge
lazy-object-proxy         1.4.3            py37h516909a_1    conda-forge
ld_impl_linux-64          2.33.1               h53a641e_8    conda-forge
libblas                   3.8.0               11_openblas    conda-forge
libcblas                  3.8.0               11_openblas    conda-forge
libcdms                   3.1.2                h7dcd879_5    conda-forge
libcf                     1.0.3            py37hb508d2e_1    conda-forge
libclang                  9.0.1           default_hde54327_0    conda-forge
libcurl                   7.68.0               hda55be3_0    conda-forge
libdrs                    3.1.2                h6e990d7_7    conda-forge
libdrs_f                  3.1.2                h322d3c2_6    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
libgfortran-ng            7.3.0                hdf63c60_5    conda-forge
libgomp                   9.2.0                h24d8f2e_2    conda-forge
libiconv                  1.15              h516909a_1005    conda-forge
liblapack                 3.8.0               11_openblas    conda-forge
libllvm9                  9.0.1                hc9558a2_0    conda-forge
libnetcdf                 4.6.2             h303dfb8_1003    conda-forge
libopenblas               0.3.6                h6e990d7_6    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libsodium                 1.0.17               h516909a_0    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
libtiff                   4.1.0                hc3755c2_3    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxkbcommon              0.10.0               he1b5a44_0    conda-forge
libxml2                   2.9.10               hee79883_0    conda-forge
libxslt                   1.1.33               h31b3aaa_0    conda-forge
locket                    0.2.0                      py_2    conda-forge
lxml                      4.5.0            py37he3881c9_1    conda-forge
lz4-c                     1.8.3             he1b5a44_1001    conda-forge
markupsafe                1.1.1            py37h516909a_0    conda-forge
matplotlib                3.1.3                    py37_0    conda-forge
matplotlib-base           3.1.3            py37h250f245_0    conda-forge
mccabe                    0.6.1                      py_1    conda-forge
mechanicalsoup            0.12.0                     py_0    conda-forge
mistune                   0.8.4           py37h516909a_1000    conda-forge
mpi                       1.0                       mpich    conda-forge
mpich                     3.3.2                hc856adb_0    conda-forge
msgpack-python            0.6.2            py37hc9558a2_0    conda-forge
nb_conda                  2.2.1                    py37_2    conda-forge
nb_conda_kernels          2.2.2                    py37_0    conda-forge
nbconvert                 5.6.1                    py37_0    conda-forge
nbformat                  5.0.4                      py_0    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
netcdf-fortran            4.4.5             h0789656_1004    conda-forge
netcdf4                   1.5.1.2          py37h73a1b54_1    conda-forge
nettle                    3.4.1             h1bed415_1002    conda-forge
notebook                  6.0.3                    py37_0    conda-forge
nspr                      4.25                 he1b5a44_0    conda-forge
nss                       3.47                 he751ad9_0    conda-forge
numpy                     1.17.5           py37h95a1406_0    conda-forge
numpydoc                  0.9.2                      py_0    conda-forge
olefile                   0.46                       py_0    conda-forge
openblas                  0.3.6                h6e990d7_6    conda-forge
openh264                  1.8.0             hdbcaa40_1000    conda-forge
openssl                   1.1.1g               h516909a_0    conda-forge
output_viewer             1.3.1                      py_1    conda-forge
owslib                    0.19.1                     py_0    conda-forge
packaging                 20.1                       py_0    conda-forge
pandas                    1.0.1            py37hb3f55d8_0    conda-forge
pandoc                    2.9.1.1                       0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.6.1                      py_0    conda-forge
partd                     1.1.0                      py_0    conda-forge
pcre                      8.44                 he1b5a44_0    conda-forge
pexpect                   4.8.0                    py37_0    conda-forge
pickleshare               0.7.5                 py37_1000    conda-forge
pillow                    7.0.0            py37hefe7db6_0    conda-forge
pip                       20.0.2                     py_2    conda-forge
proj4                     5.2.0             he1b5a44_1006    conda-forge
prometheus_client         0.7.1                      py_0    conda-forge
prompt_toolkit            2.0.10                     py_0    conda-forge
psutil                    5.6.7            py37h516909a_0    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
pycodestyle               2.5.0                      py_0    conda-forge
pycparser                 2.19                     py37_1    conda-forge
pydap                     3.2.2                 py37_1000    conda-forge
pyepsg                    0.4.0                      py_0    conda-forge
pyflakes                  2.1.1                      py_0    conda-forge
pygments                  2.5.2                      py_0    conda-forge
pykdtree                  1.3.1           py37hc1659b7_1002    conda-forge
pylint                    2.4.4                    py37_0    conda-forge
pyopenssl                 19.1.0                   py37_0    conda-forge
pyparsing                 2.4.6                      py_0    conda-forge
pyproj                    1.9.6           py37h516909a_1002    conda-forge
pyqt                      5.12.3           py37hcca6a23_1    conda-forge
pyqt5-sip                 4.19.18                  pypi_0    pypi
pyqtwebengine             5.12.1                   pypi_0    pypi
pyrsistent                0.15.7           py37h516909a_0    conda-forge
pyshp                     2.1.0                      py_0    conda-forge
pysocks                   1.7.1                    py37_0    conda-forge
pyspharm                  1.0.9           py37h7eb8c7e_1004    conda-forge
python                    3.7.6                h357f687_2    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
pytz                      2019.3                     py_0    conda-forge
pyyaml                    5.3              py37h516909a_0    conda-forge
pyzmq                     18.1.1           py37h1768529_0    conda-forge
qt                        5.12.5               hd8c4c69_1    conda-forge
qtawesome                 0.7.0                      py_0    conda-forge
qtconsole                 4.6.0                      py_0    conda-forge
qtpy                      1.9.0                      py_0    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
regrid2                   3.1.4                    pypi_0    pypi
requests                  2.22.0                   py37_1    conda-forge
requests-cache            0.5.2                    pypi_0    pypi
rope                      0.16.0                     py_0    conda-forge
scipy                     1.4.1            py37h921218d_0    conda-forge
secretstorage             3.1.2                    py37_0    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                45.2.0                   py37_0    conda-forge
shapely                   1.6.4           py37hec07ddf_1006    conda-forge
six                       1.14.0                   py37_0    conda-forge
snowballstemmer           2.0.0                      py_0    conda-forge
sortedcontainers          2.1.0                      py_0    conda-forge
soupsieve                 1.9.4            py37hc8dfbb8_1    conda-forge
sphinx                    2.4.1                      py_0    conda-forge
sphinxcontrib-applehelp   1.0.1                      py_0    conda-forge
sphinxcontrib-devhelp     1.0.1                      py_0    conda-forge
sphinxcontrib-htmlhelp    1.0.2                      py_0    conda-forge
sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
sphinxcontrib-qthelp      1.0.2                      py_0    conda-forge
sphinxcontrib-serializinghtml 1.1.3                      py_0    conda-forge
spyder                    3.3.6                    py37_1    conda-forge
spyder-kernels            0.5.2                    py37_0    conda-forge
sqlite                    3.30.1               hcee41ef_0    conda-forge
tblib                     1.6.0                      py_0    conda-forge
terminado                 0.8.3                    py37_0    conda-forge
testpath                  0.4.4                      py_0    conda-forge
tk                        8.6.10               hed695b0_0    conda-forge
toolz                     0.10.0                     py_0    conda-forge
tornado                   6.0.3            py37h516909a_4    conda-forge
traitlets                 4.3.3                    py37_0    conda-forge
typed-ast                 1.4.1            py37h516909a_0    conda-forge
udunits2                  2.2.27.6          h4e0c4b3_1001    conda-forge
urllib3                   1.25.7                   py37_0    conda-forge
vcs                       8.2                        py_2    cdat/label/v82
vcsaddons                 8.2              py37h516909a_1    cdat/label/v82
vtk-cdat                  8.2.0.8.2        py37h3df8362_0    cdat/label/v82
wcwidth                   0.1.8                      py_0    conda-forge
webencodings              0.5.1                      py_1    conda-forge
webob                     1.8.6                      py_0    conda-forge
wheel                     0.34.2                     py_1    conda-forge
widgetsnbextension        3.5.1                    py37_0    conda-forge
windspharm                1.7.0                 py37_1001    conda-forge
wk                        8.2                        py_0    cdat/label/v82
wrapt                     1.11.2           py37h516909a_0    conda-forge
wurlitzer                 2.0.0                    py37_0    conda-forge
x264                      1!152.20180806       h14c3975_0    conda-forge
xarray                    0.15.0                     py_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
yaml                      0.2.2                h516909a_1    conda-forge
zeromq                    4.3.2                he1b5a44_2    conda-forge
zict                      1.0.0                      py_0    conda-forge
zipp                      2.2.0                      py_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge
zstd                      1.4.4                h3b9ef0a_1    conda-forge

@mzelinka
Copy link
Author

mzelinka commented Sep 2, 2020

@durack1 @pochedls You guys may want to follow this

@durack1
Copy link
Member

durack1 commented Sep 2, 2020

@mzelinka you might want to bump your env up to cdat/label/v8.2.1 just to make sure this issue remains, which it most likely does

@pochedls
Copy link

pochedls commented Sep 2, 2020

I just checked and I do not have this issue with CDAT8.2.1. Maybe this fix helped: #397.

@mzelinka
Copy link
Author

mzelinka commented Sep 2, 2020

Upgrading to the latest CDAT via these instructions did not resolve this for me. Same error.

@pochedls
Copy link

pochedls commented Sep 3, 2020

Sorry - I was incorrect. I was mistakenly reading the netcdf file not the xml file. This does not work in CDAT8.2 or CDAT8.2.1.

@jasonb5 jasonb5 added the kind/bug Bug in the code. label Sep 16, 2020
@jasonb5 jasonb5 added this to the 3.1.6 milestone Sep 16, 2020
@pochedls
Copy link

@mzelinka - I've hit this bug now, too. I have modified my trimmodellist functionality to skip these files (and print a warning for now)...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Bug in the code.
Projects
None yet
Development

No branches or pull requests

4 participants