[feat] lxplus condor submission without CMSSW and with mamba #99
Conversation
| micromamba activate | ||
| micromamba install python=3.10 -c conda-forge xrootd --yes | ||
| micromamba activate base | ||
| micromamba install setuptools=70.1.1 |
There was a problem hiding this comment.
Is there any reason to restricted here?
we actually include this in the env setup :)
https://github.com/cms-btv-pog/BTVNanoCommissioning/blob/master/test_env.yml#L17
There was a problem hiding this comment.
It was inspired by the current version of this script based on Miniconda. There, a full install of the test_env.yml on the worker node was avoided as well, likely due to conda's slowness of resolving dependencies. If you want, I can switch to setting up the entire env on the worker node using mamba
There was a problem hiding this comment.
Indeed the test_env.yml only contains the restricted info
indeed the current setup also mamba install with env file, just out of curiosity why we don't need this explicity :p
There was a problem hiding this comment.
Not sure either 😅 maybe it gets most packages from doing the pip install -e . command, i.e. from the setup.cfg plus resulting dependencies?
There was a problem hiding this comment.
Checked the job logs, indeed the other dependencies get installed via the pip install -e . command
| @@ -36,6 +36,12 @@ for key in workflow output samplejson year campaign isSyst isArray noHist overwr | |||
| ARGS[$key]=$(jq -r ".$key" $WORKDIR/arguments.json) | |||
| done | |||
|
|
|||
| # Create base env with python=3.10 and setuptools<=70.1.1 | |||
| micromamba activate | |||
| micromamba install python=3.10 -c conda-forge xrootd --yes | |||
There was a problem hiding this comment.
Do we really need to specify xrootd here? I think we can get rid of this?
There was a problem hiding this comment.
Do we really need to specify xrootd here? I think we can get rid of this?
I would need to test it
There was a problem hiding this comment.
Removing the installation of xrootd in the script throws the following error:
File "/pool/condor/dir_1916719/micromamba/lib/python3.10/site-packages/uproot/extras.py", line 78, in XRootD_client
raise ModuleNotFoundError(
ModuleNotFoundError: Install XRootD python bindings with:
conda install -c conda-forge xrootdThere was a problem hiding this comment.
Thanks for checking that!
condor/execute.sh, i.e. no more dependent on centos7 architectureminicondatomicromambamicromambainstallation script has no batch modeminiconda, but much slower installation thanmicromamba!argparseof condor/submitter.pywithrunner.py`Tested successfully on
lxplus9using command