The analysis fails to start and ends immediately. [Extraction container could not be reached.] #1277
Labels
Comments
This means that the extractor container could not be reached. The extractor is responsible for unpacking the firmware image and runs inside a docker container. Could you make sure the docker image is there? It could also be the case that extractor containers are still running from another time when you did not shut down FACT cleanly. Then you can run |
|
When I start the tool, i can see these logs (and I guess there is no error here.) oguzhanozgur@oguzhanozgur:~/FACT_core$ ./start_all_installed_fact_components And here is the docker output you want : Also, there is an error about the command you provided. Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] Stop one or more running containers |
|
There is no log entry that suggest an extractor container was started. Did you change anything in the configuration file (
That is actually good, since it means there are no orphaned extractor containers running. It does not explain what the problem is, though. |
|
I didnt change anything in the conf flies. When i try to execute your command: |
|
I'm still not sure what the underlying problem is. Everything looks fine apart from the extraction containers not starting in the unpacking scheduler. Could you try running the scheduler tests? pytest src/test/integration/schedulerAlso could you try starting only the backend with Normally the extractor containers should show up in the output, e.g. |
|
Here is the all results : |
|
Did you maybe forget to activate your virtualenv? There are some import errors:
|
|
Yes, i forgot :(
|
This could be the root cause. FACT needs a lot of open files at once. What do you get when you run |
|
Hi again, |
|
It seems an exception occurred in the extraction container, but we need to activate additional logging to see what it was. Could you try to run this command? pytest -vvv --log-cli-level=DEBUG -s src/test/integration/scheduler/test_unpack_and_analyse.py |
|
Sadly, this was not really helpful: It seems the container did not start at all and therefore did not produce any error log. This is really puzzling. What happens if you try to start the container manually with the parameters used in FACT: docker run --rm -it -p "9990:5000/tcp" --entrypoint gunicorn fkiecad/fact_extractor:latest --timeout 600 -w 1 -b 0.0.0.0:5000 server:appDoes it start? Are you able to do a |
|
|
|
So the container starts fine, just not when started from FACT using the docker python API it seems. What is the version of the python docker package? (you can get it by running Also is does not look like you are really using a venv judging by the package path python3 -m venv venv
. venv/bin/activateand then in that shell reinstall all python packages with and then try starting FACT again (also from this shell)? |
|
I thinks there is a problem with docker. Here are the new logs: (btw, i could start FACT successfully but same issue occured again) |
That's the only remaining source for this problem I can think of at this point. There are some problems that can occur if you install docker and then try to use it without restarting. Have you restarted the system since you installed docker (or FACT)? If not, that could also be worth a try. And if that wasn't the issue, could you try running this script to try starting the container manually using the python docker API? from time import sleep
import docker
DOCKER_CLIENT = docker.from_env()
container = DOCKER_CLIENT.containers.run(
image='fkiecad/fact_extractor',
ports={'5000/tcp': 9999},
volumes={'/dev': {'bind': '/dev', 'mode': 'rw'}},
privileged=True,
detach=True,
remove=True,
entrypoint='gunicorn --timeout 600 -w 1 -b 0.0.0.0:5000 server:app',
)
sleep(3)
print(container.logs().decode())
container.stop() |
|
Where should I write this script, couldnt understand |
|
You could either write it to a file and run the file with |
|
Here is the script result:
|
|
That also looks normal. I'm really not sure what to make of this: Starting the containers doesn't work when you start FACT, but it works when you do it manually. There could in theory be a problem with permissions on the mounted folders. Could you try it like this (so that it works exactly like it is called in FACT): from multiprocessing import Manager
from pathlib import Path
from tempfile import TemporaryDirectory
from time import sleep
from config import load, backend
from unpacker.extraction_container import ExtractionContainer
load()
Path(backend.docker_mount_base_dir).mkdir(exist_ok=True)
tmp_dir = TemporaryDirectory(dir=backend.docker_mount_base_dir)
try:
with Manager() as manager:
ec = ExtractionContainer(id_=1, tmp_dir=tmp_dir, value=manager.Value('i', 0))
ec.start()
sleep(3)
container = ec._get_container()
print(container.logs())
ec.stop()
finally:
tmp_dir.cleanup()There is also one thing I did not ask about: How much RAM do you have in the system where FACT is running? |
|
I think there is a problem here :
|
This script must be executed form the |
|
Hi again,
|
|
Sorry that was my fault: The directory does not exists yet which causes the error. I updated the script. Also you didn't answer this question:
This can be a problem, because FACT is rather memory hungry. Normally it shouldn't be a problem during startup, though. |
|
Sorry, didnt see that one. |
|
By the way, I dont know why this happened but I can not start FACT anymore. |
That shoud be plenty.
This is not a problem I encountered before. Maybe it is also related to the problems with starting the extractor containers. According to https://stackoverflow.com/questions/43720339/docker-error-could-not-find-an-available-non-overlapping-ipv4-address-pool-am this may have something to do with a VPN running in the background. Could this be the problem in your case? |
|
Yes, you are right. I disconnected from VPN and tool worked again but our problem is still continuing. |
I'm sorry that the problem still persists. I'm still puzzled what could be the underlying issue here. |
|
Hi again, Vagrant is also has same issue. Also I want to install the tool with normal way. I tried re-installation several times but nothing changed. What should I do? |
Are you sure that it is the same issue? Since it runs in a VM, this would not make a lot of sense. Maybe it has something to do with your hardware or your network |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
FACT version
4.3-dev
Environment
Distribution : Ubuntu 22.04.5 LTS
powered by FACT 4.3-dev
© Fraunhofer FKIE 2015-2024
Steps to reproduce
~/FACT_core/start_all_installed_fact_components
Observed Behavior
The analysis has ended immediately and nothing showed up at result screen.
Expeced Behavior
The analysis should start normally and the tool analyze the fields which I marked.
Installation logs
No response
Backend logs
[2024-10-01 13:29:01][connectionpool][WARNING]: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7dbd805f7c70>: Failed to establish a new connection: [Errno 111] Connection refused')': /status
[2024-10-01 13:29:01][connectionpool][WARNING]: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7dbd805f7ee0>: Failed to establish a new connection: [Errno 111] Connection refused')': /status
[2024-10-01 13:29:02][connectionpool][WARNING]: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7dbd805f71c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /status
[2024-10-01 13:29:02][unpacking_scheduler][ERROR]: Could not fetch unpacking container logs
[2024-10-01 13:29:02][unpacking_scheduler][ERROR]: Exception happened during extraction of 2bd7fcbb382db9223414bde8aefd4f7eab3299bc0084e43356e6c1ac26af3baf_4535.: Extraction container could not be reached.
Traceback (most recent call last):
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/connection.py", line 199, in _new_conn
sock = connection.create_connection(
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/connectionpool.py", line 789, in urlopen
response = self._make_request(
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/connectionpool.py", line 495, in _make_request
conn.request(
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/connection.py", line 441, in request
self.endheaders()
File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/connection.py", line 279, in connect
self.sock = self._new_conn()
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/connection.py", line 214, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7dbd80424160>: Failed to establish a new connection: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/requests/adapters.py", line 589, in send
resp = conn.urlopen(
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/connectionpool.py", line 873, in urlopen
return self.urlopen(
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/connectionpool.py", line 873, in urlopen
return self.urlopen(
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/connectionpool.py", line 873, in urlopen
return self.urlopen(
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/connectionpool.py", line 843, in urlopen
retries = retries.increment(
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=9900): Max retries exceeded with url: /status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7dbd80424160>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/oguzhanozgur/FACT_core/src/unpacker/unpack_base.py", line 55, in _extract_with_worker
response = container.start_unpacking(tmp_dir, timeout=WORKER_TIMEOUT)
File "/home/oguzhanozgur/FACT_core/src/unpacker/extraction_container.py", line 118, in start_unpacking
response = self._check_connection()
File "/home/oguzhanozgur/FACT_core/src/unpacker/extraction_container.py", line 133, in _check_connection
return session.get(url, timeout=5)
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/oguzhanozgur/ozzy/lib/python3.10/site-packages/requests/adapters.py", line 622, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=9900): Max retries exceeded with url: /status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7dbd80424160>: Failed to establish a new connection: [Errno 111] Connection refused'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/oguzhanozgur/FACT_core/src/scheduler/unpacking_scheduler.py", line 199, in work_thread
extracted_objects = self.unpacker.unpack(task, tmp_dir, container)
File "/home/oguzhanozgur/FACT_core/src/unpacker/unpack.py", line 42, in unpack
extracted_files = self.extract_files_from_file(current_fo.file_path, tmp_dir, container)
File "/home/oguzhanozgur/FACT_core/src/unpacker/unpack_base.py", line 41, in extract_files_from_file
self._extract_with_worker(file_path, container, tmp_dir)
File "/home/oguzhanozgur/FACT_core/src/unpacker/unpack_base.py", line 59, in _extract_with_worker
raise ExtractionError('Extraction container could not be reached.') from error
unpacker.unpack_base.ExtractionError: Extraction container could not be reached.
[2024-10-01 13:29:02][unpacking_scheduler][INFO]: Unpacking completed: 2bd7fcbb382db9223414bde8aefd4f7eab3299bc0084e43356e6c1ac26af3baf_4535 (extracted files: 0)
[2024-10-01 13:29:02][unpacking_scheduler][INFO]: Unpacking of firmware 2bd7fcbb382db9223414bde8aefd4f7eab3299bc0084e43356e6c1ac26af3baf_4535 completed.
/home/oguzhanozgur/FACT_core/src/bin/internal_symlink_magic, 7: Warning: using regular magic file
/home/oguzhanozgur/FACT_core/src/bin/firmware' /home/oguzhanozgur/FACT_core/src/bin/internal_symlink_magic, 7: Warning: using regular magic file/home/oguzhanozgur/FACT_core/src/bin/firmware'Process ExceptionSafeProcess-109:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/oguzhanozgur/FACT_core/src/helperFunctions/process.py", line 93, in run
raise exception
File "/home/oguzhanozgur/FACT_core/src/helperFunctions/process.py", line 87, in run
Process.run(self)
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/oguzhanozgur/FACT_core/src/scheduler/unpacking_scheduler.py", line 149, in extraction_loop
self.check_pending()
File "/home/oguzhanozgur/FACT_core/src/scheduler/unpacking_scheduler.py", line 173, in check_pending
container.restart()
File "/home/oguzhanozgur/FACT_core/src/unpacker/extraction_container.py", line 91, in restart
self.stop()
File "/home/oguzhanozgur/FACT_core/src/unpacker/extraction_container.py", line 67, in stop
raise RuntimeError('Container is not running.')
RuntimeError: Container is not running.
[2024-10-01 13:29:03][scheduler][INFO]: Analysis Completed: 2bd7fcbb382db9223414bde8aefd4f7eab3299bc0084e43356e6c1ac26af3baf_4535
[2024-10-01 13:29:03][analysis_status][INFO]: Analysis of firmware 2bd7fcbb382db9223414bde8aefd4f7eab3299bc0084e43356e6c1ac26af3baf_4535 completed
Frontend logs
No response
Other information
No response
The text was updated successfully, but these errors were encountered: