modifying environmental variable OPENBLAS_NUM_THREADS

[gfudenberg]

probably not ideal to modify environmental variables inside of scripts

akita_utils/bin/insert_virtual_spacing_experiment/virtual_symmetric_experiment_spacing.py

Line 92 in 1e7246d

os.environ["OPENBLAS_NUM_THREADS"] = "1"

we should trace back why this was introduced & probably remove across the library
cc @PSmaruj @Kamulegeya-Fahad

[PSmaruj]

I found this (my chat with Geoff from September 17th):

I’ve modified the multi-script, although there are some problems:
I got a weird error at the beginning:

(basenji) [smaruj@discovery2 insert_virtual_flanks_experiment]$ python multiGPU-virtual_symmetric_experiment.py /project/fudenber_735/tensorflow_models/akita/v2/models/f0c0/train/params.json /project/fudenber_735/tensorflow_models/akita/v2/models/f0c0/train/model1_best.h5 out.tsv -f /project/fudenber_735/genomes/mm10/mm10.fa --head-index 1 --model-index 1 --batch-size 4  --stats SCD,INS-16 -p 4
OpenBLAS blas_thread_init: pthread_create failed for thread 31 of 32: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 1031243 max
Traceback (most recent call last):
  File "/home1/smaruj/miniconda3/envs/basenji/lib/python3.8/site-packages/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/home1/smaruj/miniconda3/envs/basenji/lib/python3.8/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/home1/smaruj/miniconda3/envs/basenji/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: PyCapsule_Import could not import module "datetime"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "multiGPU-virtual_symmetric_experiment.py", line 36, in <module>
    import h5py
  File "/home1/smaruj/miniconda3/envs/basenji/lib/python3.8/site-packages/h5py/__init__.py", line 34, in <module>
    from . import version
  File "/home1/smaruj/miniconda3/envs/basenji/lib/python3.8/site-packages/h5py/version.py", line 19, in <module>
    import numpy
  File "/home1/smaruj/miniconda3/envs/basenji/lib/python3.8/site-packages/numpy/__init__.py", line 145, in <module>
    from . import core
  File "/home1/smaruj/miniconda3/envs/basenji/lib/python3.8/site-packages/numpy/core/__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.8 from "/home1/smaruj/miniconda3/envs/basenji/bin/python"
  * The NumPy version is: "1.20.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: PyCapsule_Import could not import module "datetime"

I’ve found some guidance here: numpy/numpy#14474
I’ve added a line os.environ['OPENBLAS_NUM_THREADS'] = '1' after importing os, and the problem disappeared.

[PSmaruj]

I've just checked and both my experiments work well when the line os.environ["OPENBLAS_NUM_THREADS"] = "1" is removed. There might have been changes in CARC jobs setup/limitations and now it works without this line (?).
@gfudenberg Can I assume that now I can remove this line from all my scripts?

[gfudenberg]

my guess is that CARC raised errors if scripts were run on head nodes w/o setting this environmental variable (otherwise the script would grab all the possible threads & lead to issues with login etc)

so yes, my guess is it is good to remove from all scripts (b/c on compute nodes you might want to grab all the threads allocated!)

[PSmaruj]

I think that this CARC limit still exists since I tried to run a multi-GPU script without the openblas line on the home node and I recreated exactly the same error. The error doesn't show up when I run the same script on the GPU setup (partition debug).
I usually do run a multi-GPU script on the home node since it submits the GPU slurm-jobs and only collects the data at the end. I do agree that the regular scripts that are run on GPU should not have this line: os.environ["OPENBLAS_NUM_THREADS"] = "1", although I think that it can be left in the multi-GPU scripts as long as they are run on the home node (?). I preferred this option so far since I was able to run them under screen.
@Kamulegeya-Fahad How do you run multi-GPU scripts?