New behaviour for esmfmkfile in v8.4.0

[dhohn]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

Many thanks to @xylar and @zklaus for getting v8.4.0 running. Im especially happy that its also working with osx-arm64 now :)

Unfortunately the new version 8.4.0 has changed the behaviour for finding the esmfmkfile. See this commit part of this larger PR.

When testing a basic import I get the following:

>>> import esmpy
Traceback (most recent call last):
  File "~/miniconda3/envs/test_esmpy/lib/python3.10/site-packages/esmpy/interface/loadESMF.py", line 26, in <module>
    esmfmk = os.environ["ESMFMKFILE"]
  File "~/miniconda3/envs/test_esmpy/lib/python3.10/os.py", line 680, in __getitem__
    raise KeyError(key) from None
KeyError: 'ESMFMKFILE'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/miniconda3/envs/test_esmpy/lib/python3.10/site-packages/esmpy/__init__.py", line 112, in <module>
    from esmpy.api.esmpymanager import *
  File "~/miniconda3/envs/test_esmpy/lib/python3.10/site-packages/esmpy/api/esmpymanager.py", line 9, in <module>
    from esmpy.interface.cbindings import *
  File "~/miniconda3/envs/test_esmpy/lib/python3.10/site-packages/esmpy/interface/cbindings.py", line 13, in <module>
    from esmpy.interface.loadESMF import _ESMF
  File "~/miniconda3/envs/test_esmpy/lib/python3.10/site-packages/esmpy/interface/loadESMF.py", line 28, in <module>
    raise ImportError('The ESMFMKFILE environment variable is not available.')
ImportError: The ESMFMKFILE environment variable is not available.

I tested on linux-64, osx-64 and osx-arm64: all have this issue. The problem can be solved by setting the env variable beforehand e.g. export ESMFMKFILE=~/miniconda3/envs/test_esmpy/lib/esmf.mk

Can this be handled by conda?

Installed packages

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
curl                      7.86.0               h2283fc2_1    conda-forge
esmf                      8.4.0           mpi_mpich_h5a1934d_101    conda-forge
esmpy                     8.4.0           mpi_mpich_py310h515c5ea_101    conda-forge
hdf4                      4.2.15               h9772cbc_5    conda-forge
hdf5                      1.12.2          mpi_mpich_h5d83325_0    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.19.3               h08a2579_0    conda-forge
ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libcurl                   7.86.0               h2283fc2_1    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libnetcdf                 4.8.1           mpi_mpich_hcd871d9_6    conda-forge
libnghttp2                1.47.0               hff17c54_1    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libssh2                   1.10.0               hf14f497_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libxml2                   2.10.3               h7463322_0    conda-forge
libzip                    1.9.2                hc929e4a_1    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
mpi                       1.0                       mpich    conda-forge
mpi4py                    3.1.4           py310h37cc914_0    conda-forge
mpich                     4.0.3              h846660c_100    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
netcdf-fortran            4.6.0           mpi_mpich_hd09bd1e_1    conda-forge
numpy                     1.23.5          py310h53a5b5f_0    conda-forge
openssl                   3.0.7                h0b41bf4_1    conda-forge
parallelio                2.5.9           mpi_mpich_h50e6f33_101    conda-forge
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
python                    3.10.8          h4a9ceb5_0_cpython    conda-forge
python_abi                3.10                    3_cp310    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
setuptools                65.5.1             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge

Environment info

active environment : test_esmpy
    active env location : ~/miniconda3/envs/test_esmpy
            shell level : 2
       user config file : ~/.condarc
 populated config files : ~/.condarc
          conda version : 4.12.0
    conda-build version : not installed
         python version : 3.9.7.final.0
       virtual packages : __linux=4.18.0=0
                          __glibc=2.28=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : ~/miniconda3  (writable)
      conda av data dir : ~/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : ~/miniconda3/pkgs
                          ~/.conda/pkgs
       envs directories : ~/miniconda3/envs
                          ~/.conda/envs
               platform : linux-64
             user-agent : conda/4.12.0 requests/2.27.1 CPython/3.9.7 Linux/4.18.0-193.el8.x86_64 rhel/8.2 glibc/2.28
                UID:GID : 725893:8108
             netrc file : None
           offline mode : False

[xylar]

@dhohn, how are you activating test_esmpy? There is supposed to be an activation script that sets that variable. I'll test locally as well.

[xylar]

This worked great for me on Linux:

$ mamba create -y -n test python=3.11 esmpy=8.4.0
$ mamba activate test
$ python -c "import esmpy"
$ echo $ESMFMKFILE
/home/xylar/mambaforge/envs/test/lib/esmf.mk

[xylar]

Same if I use conda instead of mamba (it just takes longer).

[xylar]

Another question, what shell are you using? This should work for any typical shell but it could be that one or another of the non-bash activation scripts has an issue and didn't get tested.

[dhohn]

Hi @xylar ,

Thanks for double checking this! Im using all the normal stuff (bash v>4.4). Creating and activating envs with conda, installing with mamba

So I think my problem was that I did the following:

conda create -n test python
conda activate test
mamba install esmpy

So no activation with esmpy already installed. A deactivate+activate works like a charm! I feel a bit silly that I somehow managed to do it that way 3 seperate times yesterday while trying it out.

Anyway the good old "try turning it off and on again" works again :D

[xylar]

@dhohn, okay, no worries. Unfortunately, there's not a good way around that. It's a good habit to create your environment with all the packages you expect to want in part because of the activation difficulties. But also in part because you'll typically get a better mutually compatible set of packages if the solver can figure them all out at once rather than with each separate install.

Anyway, you may already know all that.

[aulemahal]

This issue affects Read the docs since the normal mechanism to install conda/mamba packages used there does not activate the environment.

For other people getting the same bug as I did, one possible workaround is to set the environment variable explicitly in the conf.py file:

import os

if os.environ.get('READTHEDOCS') and 'ESMFMKFILE' not in os.environ:
    # RTD doesn't activate the env, and esmpy depends on a env var set there
    # We assume the `os` package is in {ENV}/lib/pythonX.X/os.py
    # See conda-forge/esmf-feedstock#91 and readthedocs/readthedocs.org#4067
    os.environ['ESMFMKFILE'] = str(Path(os.__file__).parent.parent / 'esmf.mk')
 
import esmpy

[aulemahal]

Hi @xylar, my previous comment was mostly a note to other users finding this issue for that particular edge case, but I'd like to submit another case:

This new environment variable behaviour has also broken the possibility to use xESMF in jupyter notebooks that use a kernel different from the server's one. For example, I have all the jupyter stuff installed in my "base" conda environment, but I also have a different environment for each project I am working on. To avoid jumping between different jupyter servers, I use this method : https://ipython.readthedocs.io/en/latest/install/kernel_install.html. It allows accessing multiple kernels from the same single jupyter notebook server (through the "Kernel -> Change kernel" menu item).

However, here again the environment variable is not set when starting the new kernel, and importing esmpy fails.

I have ideas for workarounds, but they are all hacks. Would it be acceptable to have a default search location if the os environment variable is not set upon import? Something in the like of the fix suggest above?

[xylar]

@aulemahal, that's an interesting situation. I don't know a way to set an environment variable when the base environment gets activated. So I don't know of a way to set $ESMFMKFILE to a default value in the base environment. but I'll hunt around a bit and see if there's a way.

If you are asking for ESMPy to have some default behavior when $ESMFMKFILE isn't set, conda-forge isn't really the place for that to happen -- it would need to happen in the ESMPy package itself. But I also don't think there is an obviou and safe place for ESMPy to guess that esmf.mk might be.

[aulemahal]

Indeed, I was wondering about a default mechanism rather than an absolute dependency on an environment variable. You're right that this may not be the best place, I'll open an issue on ESMF, thanks!

[xylar]

@aulemahal, after a little more googling around and thinking about this, I feel like you're asking to have $ESMFMKFILE get set somehow when conda activate isn't being called on your various environments within jupyter, and I don't think conda provides any other mechanism at all for setting environment variables at the package level. As a user, you can use conda env config vars set my_var=value, see https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#setting-environment-variables. But a package has no other way to do this for you.

[zklaus]

Interesting discussion! I personally consider using base for anything other than the conda/mamba installation itself a bit brittle. Perhaps you might want to experiment with nested activation and --stack, @aulemahal, to layer two envs where both are activated.

[xylar]

@zklaus, oh, interesting! I also wasn't aware of the --stack option.

[valeriupredoi]

@aulemahal is spot-n - we were a bit slow at starting to support Python=3.11, but now we're almost there, and lo and behold the issue he ponts out on Readthedocs build https://readthedocs.org/projects/esmvalcore/builds/19994764/ (hope you have permissions to see it). I'm thinking of a solution for this, if I manage to make it work, I'll post here

[valeriupredoi]

I have tried in many ways to set that var in the RTD environment - to no avail, the problem is that RTD creates an env per doc build (the env is named either latest, or stable, or the number of the PR that triggered it) and doesn't do conda activate [env] explicitly so that variable is not set; trying to set it via the .readthedocs.yaml or conf.py conf files is like trying to talk to Captain Ahab that is currently stuck inside Moby Dick ie no control over the RTD's conda gubbins which are all under the hood. If anyone's managed to do this (specifically for the RTD case) I'd be very grateful for a workaround/fix 🍺

[kthyng]

I have had to find this issue many times over the years so that I can use @aulemahal's solution for being able to use xESMF with readthedocs.