Access the mpas_analysis package with an "entry point"

[xylar]

This means that the run_mpas_analysis script has been removed and its functionality has been moved to mpas_analysis/__main__.py. When mpas_analysis is installed as a conda package, there will be an mpas_analysis script in the path (as well as a download_analysis_data script) in the path. When users want to run the analysis from the repo rather than the package, they will need to use:

python -m mpas_analysis ...

instead of ./run_mpas_analysis .... Nothing else about execution has changed (e.g. all flags work as before).

The documentation, comments, scripts and example config files have been updated to reflect the change.

Reason for this change

In order to move over to conda-forge instead of building the mpas_analysis package ourselves for each release, we need to have an entry point rather than a script. An entry point also allows us to call the main script the same thing as the package (mpas_analysis), which is less clunky than adding "run" to the front just to prevent conflicts.

closes #452

[xylar]

I have tested this with a simple QU240wLI test case using the repo (i.e. python -m mpas_analysis ...). I haven't yet tested installing this locally as a conda package but will do so shortly.

[xylar]

It should be possible to do a test install of the conda package built from this branch using:

conda create -n mpas_analysis_1.1_py3.6 -c conda-forge -c xylar python=3.6 mpas_analysis=1.1
conda activate mpas_analysis_1.1_py3.6
mpas_analysis config.<run>

[xylar]

After a few rebuilds (removing nco and adding some missing XML files to setup.py), the conda package seems to be working, too.

[xylar]

@jhkennedy, if you would be willing to take a look at this when you have time, I think it's ready for a review. If you want to actually test it on a run, there are example runs pointed to in the configs/<machine> directories. If you'd rather do a review at the level of looking over the code, that's perfectly fine.

@milenaveneziani, would you be willing to run a test, once with the git repo (as usual) using the python -m mpas_analysis and once by creating a new conda environment? If you need help with the latter, let me know and maybe we can work on it next week. Make sure you don't try to add your conda environment to the e3sm-unified base install of miniconda, please.

[jhkennedy]

@xylar with a cursory glance it looks great! I'll look at it in depth tomorrow -- I should be able to test it on a run.

[milenaveneziani]

yes, I'd really like to run a couple of tests.
For the second one, with the conda package, I do have my own conda I can use.

[jhkennedy]

@xylar Everything looks pretty good to me!

I just have some minor comments:

• You are currently testing MPAS-Analysis with python 2.7, 3.5 and 3.6, but you only list 2.7 in the setup file: https://github.com/xylar/MPAS-Analysis/blob/add_entry_point/setup.py#L44-L46 . Are you not officially supporting python 3?

• When you fork the conda-forge recipe for MPAS-Analysis, you'll want to make sure you list you entry-point in that meta.yml here: https://github.com/xylar/MPAS-Analysis/blob/add_entry_point/conda/recipe/meta.yaml#L10 . I'm not sure if that's needed for your e3sm_unified builds.

[xylar]

Thanks @jhkennedy, those are helpful comments and very easy to address.

• We've tested and are supporting python 2.7, 3.5, 3.6 and 3.7 (I'm about to remove 3.5 from CI and switch to 3.7 instead so I'll likely keep 3.5 out of the list, just in case)
• I'll add the entry point to the conda recipe and rebuild the package. Doesn't hurt to do this early.

[milenaveneziani]

@xylar: I have been trying to run MPAS-Analysis on cooley but I am having trouble (I will report on the changes I had to make in order for the batch job script to work once I am successful..). Please bear with me for another day or so. thanks.

[milenaveneziani]

@xylar: MPAS-Analysis is finally working again for me on cooley (although there are some changes that we need to apply to the batch job script: I will open a quick PR for this).
This is with using e3sm-unified and the python -m mpas_analysis command.

Now I want to use it as a conda package, but before doing that: is there an easy way to use/install the e3sm-unified packages in my conda environment? Or do I just list all packages under e3sm-unified and install them under my conda?

[milenaveneziani]

although there are some changes that we need to apply to the batch job script: I will open a quick PR for this

Or should I just push the new job_script.cooley.bash to this branch?

[xylar]

@milenaveneziani,

Or should I just push the new job_script.cooley.bash to this branch?

As long as you think this PR is close to being merged, let's just update the Cooley job script in this PR.

Now I want to use it as a conda package, but before doing that: is there an easy way to use/install the e3sm-unified packages in my conda environment? Or do I just list all packages under e3sm-unified and install them under my conda?

This might be easiest to discuss in person. I'll come by really quick.

[milenaveneziani]

@xylar: the conda package works just fine. I installed it with no problems in my own ocnda.
The only warning message I get when running mpas_analysis... is this:

/lus/theta-fs0/projects/ClimateEnergy_2/milena/miniconda2.7/envs/mpas_analysis_1.1_py3.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)

don't know if you care. From my point of view, you can merge this.

ps: there was already a config file available for the high-res on alcf, so we are good about it. I just updated it for the parallelTasks issue.

[xylar]

@milenaveneziani, thanks so much for your careful testing! The warning you saw is really annoying but nothing to worry about. I suspect that it has to do with building numpy with one library and running it with another but I see that warning on every machine I use with the latest version of numpy.