progress stalled; perhaps related to too many open files

[wideglide]

FACT version

aaa3c4c

Environment

• Ubuntu 22.04.4 LTS
• (venv) Python 3.10.12

Steps to reproduce

• processing firmware images
  • ASUS RT-AX82U v. 3.0.0.4.388.22158
  • TP-Link Archer C59(US)_V2_220113.zip

Observed Behavior

Processing stalled.

system health

Expeced Behavior

Processed firmware image normally.

Installation logs

no issues during install.

Backend logs

fact_main_backend.log

[2024-03-19 10:32:11][unpacking_scheduler][ERROR]: Exception occurred during unpacking of 2df36886966bb3f3ce60c849fed95e7ff02d6a0ae1c2d8889b8442cfaddd5790_5392
Traceback (most recent call last):
  File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 187, in _work_thread_wrapper
    self.work_thread(task, container)
  File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 195, in work_thread
    with TemporaryDirectory(dir=container.tmp_dir.name) as tmp_dir:
  File "/usr/lib/python3.10/tempfile.py", line 1008, in __exit__
    self.cleanup()
  File "/usr/lib/python3.10/tempfile.py", line 1012, in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
  File "/usr/lib/python3.10/tempfile.py", line 994, in _rmtree
    _rmtree(name, onerror=onerror)
  File "/usr/lib/python3.10/shutil.py", line 725, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.10/shutil.py", line 633, in _rmtree_safe_fd
    onerror(os.scandir, path, sys.exc_info())
  File "/usr/lib/python3.10/shutil.py", line 629, in _rmtree_safe_fd
    with os.scandir(topfd) as scandir_it:
OSError: [Errno 24] Too many open files: '/tmp/fact-docker-mount-base-dir/tmprpukq11b/tmpbm_gz_v9/reports'
[2024-03-19 10:32:11][unpacking_scheduler][ERROR]: Exception occurred during unpacking of cc528127f922195150ad66dd7254c6275392a06beeab9f8a7905d2ec3a8e2562_7756
Traceback (most recent call last):
  File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 187, in _work_thread_wrapper
    self.work_thread(task, container)
  File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 195, in work_thread
    with TemporaryDirectory(dir=container.tmp_dir.name) as tmp_dir:
  File "/usr/lib/python3.10/tempfile.py", line 1008, in __exit__
    self.cleanup()
  File "/usr/lib/python3.10/tempfile.py", line 1012, in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
  File "/usr/lib/python3.10/tempfile.py", line 994, in _rmtree
    _rmtree(name, onerror=onerror)
  File "/usr/lib/python3.10/shutil.py", line 725, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.10/shutil.py", line 633, in _rmtree_safe_fd
    onerror(os.scandir, path, sys.exc_info())
  File "/usr/lib/python3.10/shutil.py", line 629, in _rmtree_safe_fd
    with os.scandir(topfd) as scandir_it:
OSError: [Errno 24] Too many open files: '/tmp/fact-docker-mount-base-dir/tmpgs5tv66w/tmp22hy3plt/reports'
[2024-03-19 10:32:11][unpacking_scheduler][ERROR]: Exception occurred during unpacking of 0ecf29ed3821a55d6dfa80f75ad6777e64f71d47a305bbe9fc50979352dba0f8_1248
Traceback (most recent call last):
  File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 197, in work_thread
    extracted_objects = self.unpacker.unpack(task, tmp_dir, container)
  File "/home/kyle/FACT_core/src/unpacker/unpack.py", line 43, in unpack
    extracted_files = self.extract_files_from_file(current_fo.file_path, tmp_dir, container)
  File "/home/kyle/FACT_core/src/unpacker/unpack_base.py", line 35, in extract_files_from_file
    shutil.copy2(file_path, str(Path(tmp_dir, 'input', Path(file_path).name)))
  File "/usr/lib/python3.10/shutil.py", line 434, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.10/shutil.py", line 256, in copyfile
    with open(dst, 'wb') as fdst:
OSError: [Errno 24] Too many open files: '/tmp/fact-docker-mount-base-dir/tmpgu0kfqtb/tmpmtxr_4gt/input/0ecf29ed3821a55d6dfa80f75ad6777e64f71d47a305bbe9fc50979352dba0f8_1248'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 187, in _work_thread_wrapper
    self.work_thread(task, container)
  File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 195, in work_thread
    with TemporaryDirectory(dir=container.tmp_dir.name) as tmp_dir:
  File "/usr/lib/python3.10/tempfile.py", line 1008, in __exit__
    self.cleanup()
  File "/usr/lib/python3.10/tempfile.py", line 1012, in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
  File "/usr/lib/python3.10/tempfile.py", line 994, in _rmtree
    _rmtree(name, onerror=onerror)
  File "/usr/lib/python3.10/shutil.py", line 725, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.10/shutil.py", line 633, in _rmtree_safe_fd
    onerror(os.scandir, path, sys.exc_info())
  File "/usr/lib/python3.10/shutil.py", line 629, in _rmtree_safe_fd
    with os.scandir(topfd) as scandir_it:
OSError: [Errno 24] Too many open files: '/tmp/fact-docker-mount-base-dir/tmpgu0kfqtb/tmpmtxr_4gt'
[2024-03-19 10:32:11][unpacking_scheduler][ERROR]: Exception occurred during unpacking of 8401139708fdc9407e43b05d0df712fc89061946fab799917e11937032745f5d_950

Frontend logs

fact_main_frontend.log

Not sure if this is related?

[2024-03-17 14:35:32][app][ERROR]: Exception on /admin/logs [GET]
Traceback (most recent call last):
  File "/home/kyle/.local/lib/python3.10/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/kyle/.local/lib/python3.10/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/kyle/.local/lib/python3.10/site-packages/flask_restx/api.py", line 674, in error_router
    return original_handler(e)
  File "/home/kyle/.local/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/kyle/.local/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/kyle/FACT_core/src/./web_interface/security/decorator.py", line 13, in decorated_view
    return fn(*args, **kwargs)
  File "/home/kyle/FACT_core/src/./web_interface/components/miscellaneous_routes.py", line 116, in show_logs
    backend_logs = '\n'.join(self.intercom.get_backend_logs())
TypeError: can only join an iterable

Other information

Maybe this is a system configuration issue?

[jstucke]

This is a problem that has haunted us for quite a while now. There is a limit on how many files can be opened on a system at the same time and the default value is quite low on some systems. You can find out what the limit is by running ulimit -n and you can set a new value by running ulimit -n <new_limit> (but this only applies to the current shell, so you may want to put it in your .bashrc/.zshrc/config.fish/etc.). Depending on how many worker processes you configure, you may need more. I'm not running into issues with a value of 9999 but I don't have that many workers configured.