Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat] lxplus condor submission without CMSSW and with mamba #99

Merged
merged 1 commit into from
Aug 9, 2024
Merged

[feat] lxplus condor submission without CMSSW and with mamba #99

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 21 additions & 15 deletions condor/execute.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,26 +8,26 @@ if [ -d /afs/cern.ch/user/${USER:0:1}/$USER ]; then
fi
env


WORKDIR=`pwd`

# Load a base cmssw env
export SCRAM_ARCH=slc7_amd64_gcc820
source /cvmfs/cms.cern.ch/cmsset_default.sh
scram p CMSSW CMSSW_11_1_0_pre5_PY3
cd CMSSW_11_1_0_pre5_PY3/src
eval `scram runtime -sh`
cd ../..

# Set up conda environment
wget https://repo.anaconda.com/miniconda/Miniconda3-py310_23.5.0-3-Linux-x86_64.sh
bash Miniconda3-py310_23.5.0-3-Linux-x86_64.sh -b -p $WORKDIR/miniconda
# Set up mamba environment
## Interactive bash script with fallback pointing to $HOME, hence setting $PWD of worker node as $HOME
export HOME=`pwd`
wget -L micro.mamba.pm/install.sh
chmod +x install.sh
## FIXME parsing arguments does not work. will use defaults in install.sh instead, see https://github.com/mamba-org/micromamba-releases/blob/main/install.sh
## Tried solutions listed in https://stackoverflow.com/questions/14392525/passing-arguments-to-an-interactive-program-non-interactively
./install.sh <<< $'bin\nY\nY\nmicromamba\n'
source .bashrc

export PATH=$WORKDIR:$PATH

export PATH=$WORKDIR/miniconda/bin:$PATH
export PYTHONPATH=$WORKDIR/miniconda/lib/python3.10/site-packages

if [ ! -d /afs/cern.ch/user/${USER:0:1}/$USER ]; then
## install necessary packages if on cmsconnect
conda install -c conda-forge jq --yes
micromamba install -c conda-forge jq --yes
fi

# Get arguments
Expand All @@ -36,6 +36,12 @@ for key in workflow output samplejson year campaign isSyst isArray noHist overwr
ARGS[$key]=$(jq -r ".$key" $WORKDIR/arguments.json)
done

# Create base env with python=3.10 and setuptools<=70.1.1
micromamba activate
micromamba install python=3.10 -c conda-forge xrootd --yes
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need to specify xrootd here? I think we can get rid of this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need to specify xrootd here? I think we can get rid of this?

I would need to test it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the installation of xrootd in the script throws the following error:

  File "/pool/condor/dir_1916719/micromamba/lib/python3.10/site-packages/uproot/extras.py", line 78, in XRootD_client
    raise ModuleNotFoundError(
ModuleNotFoundError: Install XRootD python bindings with:

    conda install -c conda-forge xrootd

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking that!

micromamba activate base
micromamba install setuptools=70.1.1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason to restricted here?
we actually include this in the env setup :)
https://github.com/cms-btv-pog/BTVNanoCommissioning/blob/master/test_env.yml#L17

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was inspired by the current version of this script based on Miniconda. There, a full install of the test_env.yml on the worker node was avoided as well, likely due to conda's slowness of resolving dependencies. If you want, I can switch to setting up the entire env on the worker node using mamba

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed the test_env.yml only contains the restricted info
indeed the current setup also mamba install with env file, just out of curiosity why we don't need this explicity :p
image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure either 😅 maybe it gets most packages from doing the pip install -e . command, i.e. from the setup.cfg plus resulting dependencies?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the job logs, indeed the other dependencies get installed via the pip install -e . command


# Install BTVNanoCommissioning
mkdir BTVNanoCommissioning
cd BTVNanoCommissioning
Expand All @@ -51,7 +57,6 @@ ln -s /cvmfs/cms.cern.ch/rsync/cms-nanoAOD/jsonpog-integration src/BTVNanoCommis
pip install -e .

## other dependencies
conda install -c conda-forge xrootd --yes
pip install psutil

# Build the sample json given the job id
Expand All @@ -62,9 +67,10 @@ OPTS="--wf ${ARGS[workflow]} --year ${ARGS[year]} --campaign ${ARGS[campaign]} -
if [ "${ARGS[voms]}" != "null" ]; then
OPTS="$OPTS --voms ${ARGS[voms]}"
fi
if [ "${ARGS[isSyst]}" != "null" ]; then
if [ "${ARGS[isSyst]}" != "false" ]; then
OPTS="$OPTS --isSyst ${ARGS[isSyst]}"
fi

for key in isArray noHist overwrite skipbadfiles; do
if [ "${ARGS[$key]}" == true ]; then
OPTS="$OPTS --$key"
Expand Down
16 changes: 9 additions & 7 deletions condor/submitter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,27 +65,30 @@ def get_main_parser():
help="JSON file containing dataset and file locations (default: %(default)s)",
)
## Configuations
parser.add_argument("--year", default="2017", help="Year")
parser.add_argument("--year", default="2023", help="Year")
parser.add_argument(
"--campaign",
default="Rereco17_94X",
default="Summer23",
choices=[
"Rereco17_94X",
"Winter22Run3",
"Summer22Run3",
"Summer22EERun3",
"Summer22",
"Summer22EE",
"Summer23",
"Summer23BPix",
"2018_UL",
"2017_UL",
"2016preVFP_UL",
"2016postVFP_UL",
"CAMPAIGN_prompt_dataMC",
],
help="Dataset campaign, change the corresponding correction files",
)
parser.add_argument(
"--isSyst",
default=None,
default=False,
type=str,
choices=[None, "all", "weight_only", "JERC_split"],
choices=[False, "all", "weight_only", "JERC_split", "JP_MC"],
help="Run with systematics, all, weights_only(no JERC uncertainties included),JERC_split, None",
)
parser.add_argument("--isArray", action="store_true", help="Output root files")
Expand Down Expand Up @@ -217,7 +220,6 @@ def get_main_parser():

Arguments = $(JOBNUM)

requirements = (OpSysAndVer =?= "CentOS7")
request_cpus = 1
request_memory = 2000
use_x509userproxy = true
Expand Down
Loading