Return "interrupt" on importing python additional module (e.g. numpy, scipy)

[falatak]

Hello,
My problem may be same as the last comment in #865, and it looks unresolved.
I may understand python is executed with -S option. So instead of modification of python_executer.py suggested in #865, I set PYTHONPATH and python on docker judge container could import numpy as expected.

judge@b346f839e882:/usr/bin$ export PYTHONPATH=/usr/local/lib/python3.9/dist-packages
judge@b346f839e882:/usr/bin$ python3.9 -S
Python 3.9.10 (main, Feb 22 2022, 13:54:07)
[GCC 11.2.0] on linux
>>> import numpy
>>> numpy.pi
3.141592653589793

But, judge ended up saying "interrupt" same as #865. It occur on importing numpy, like

import numpy as np #<----- raise interruption
print("hello world")

Can you please tell us how to solve this problem?

Thank you for your cooperation!

[Xyene]

Sorry, this is an unsupported configuration, and we don't have the resources to track down what's going on. Very likely something is running into the sandbox, and adding debug prints inside it may help you track down what. If you figure it out, feel free to leave a comment in case it's useful for others.

[falatak]

Thank you for reply. I fought with debug print and finally it works (probably). The error on importing numpy was

Self-testing executors
OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 8: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 0 current, 0 max
....
OpenBLAS blas_thread_init: pthread_create failed for thread 7 of 8: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 0 current, 0 max
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/usr/local/lib/python3.9/dist-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/usr/local/lib/python3.9/dist-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: PyCapsule_Import could not import module "datetime"

This was not "datetime importing" issue as the message saying, but the solution is adding environment variable OPENBLAS_NUM_THREADS. It was referred on numpy/numpy#14474 (comment)
But as I reported above, without OPENBLAS_NUM_THREAD normally it works on the server. It confused me but anyway my solution is modifying python_executer.py like below so far.

class PythonExecutor(CompiledExecutor):
    loader_script = """\
import runpy, sys, os
os.environ["OPENBLAS_NUM_THREADS"] = "1" #<----- ADDED
del sys.argv[0]
import numpy
sys.stdin = os.fdopen(0, 'r', 65536)
sys.stdout = os.fdopen(1, 'w', 65536)
runpy.run_path(sys.argv[0], run_name='__main__')
"""

I think the reason why the environment variable is required is that the executed process is detached, but I don't know clearly . And I’m wondering adding directly to python_executer.py is not good idea. If there is a better way and/or knowledge, I would appreciate if you could share it.

[Xyene]

Good sleuthing.

OpenBLAS is likely running into the RLIMIT_NPROC DMOJ sets. I suspect if you disable this protection, things would work. Setting nproc = -1 allows processes to spawn arbitrarily many threads, see

judge-server/dmoj/executors/mono_executor.py

Line 22 in 82871ee

nproc = -1

for example.

(Likely, you wouldn't need to set it to -1, but to something positive but enough for OpenBLAS to load. In either case, making OpenBLAS use only one thread seems like the right solution for online grading.)

Regarding your modification to python_executor.py, the judge clears the environment at execution time, to avoid accidentally leaking bits of the environment you didn't mean to.

Rather than modify the loader_script to inject OPENBLAS_NUM_THREADS=1, you can override get_env. See

judge-server/dmoj/executors/mono_executor.py

Lines 49 to 56 in 82871ee

	def get_env(self) -> Dict[str, str]:
	env = super().get_env()
	# Disable Mono's usage of /dev/shm, so we don't have to deal with
	# its extremely messy access patterns to it.
	env['MONO_DISABLE_SHARED_AREA'] = '1'
	# Disable Mono's generation of core dump files (e.g. on MLE).
	env['MONO_CRASH_NOFILE'] = '1'
	return env

for an example.

[falatak]

Many thanks!

[ktikhonov]

Hi, @falatak as a follow up: when you say "I set PYTHONPATH and python on docker judge container could import numpy as expected.", at which stage one should do this actually?

[falatak]

@ktikhonov ,This process was performed to ensure that python and numpy were correctly installed in the judge container for troubleshooting. Therefore, it should be done at the initial stage.