SCT fails with OSError: [Errno 98] Address already in use

[ilya-rarov]

Issue description

• This issue is a regression.
• It is unknown if this issue is a regression.

The artifacts-ubuntu2204-arm-test job failed with the error:

14:36:24  < t:2023-07-09 11:36:24,227 f:test_config.py  l:296  c:sdcm.test_config     p:INFO  > Initializing Argus connection...
14:36:24  < t:2023-07-09 11:36:24,703 f:tester.py       l:390  c:ArtifactsTest        p:INFO  > test_id 707e495b-0757-47c6-966e-4a613fa62c4f already exists in Argus with status: created
14:36:25  < t:2023-07-09 11:36:24,849 f:tester.py       l:405  c:ArtifactsTest        p:INFO  > sct_runner info in Argus TestRun is updated
14:36:26  Process SyncManager-3:
14:36:26  Traceback (most recent call last):
14:36:26    File "/usr/local/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
14:36:26      self.run()
14:36:26    File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run
14:36:26      self._target(*self._args, **self._kwargs)
14:36:26    File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 591, in _run_server
14:36:26      server = cls._Server(registry, address, authkey, serializer)
14:36:26    File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 156, in __init__
14:36:26      self.listener = Listener(address=address, backlog=16)
14:36:26    File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 453, in __init__
14:36:26      self._listener = SocketListener(address, family, backlog)
14:36:26    File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 596, in __init__
14:36:26      self._socket.bind(address)
14:36:26  OSError: [Errno 98] Address already in use
14:36:26  
14:36:26  Aborted!

The error happened at the very beginning of the execution of Run SCT Test () stage of the pipeline, when ClusterTester was being initialized - the test didn't really start.

How frequently does it reproduce?

It reproduced in many other artifacts jobs
scylla-master:

• artifacts-rocky9-test
• artifacts-ubuntu2204-nonroot-test
• artifacts-debian11-test
• artifacts-ubuntu2204-arm-test
• artifacts-debian11-nonroot-test
• artifacts-rocky9-nonroot-test

scylla-enterprise:

• artifacts-ubuntu2204-test
• artifacts-rocky8-test
• artifacts-centos8-arm-test
• artifacts-ubuntu2204-arm-test

and more

Installation details

Cluster size: 1 nodes (im4gn.xlarge)

Scylla Nodes used in this run:
No resources left at the end of the run

OS / Image: ami-022c8ce295ce9ac4c (aws: undefined_region)

Test: artifacts-ubuntu2204-arm-test
Test id: 707e495b-0757-47c6-966e-4a613fa62c4f
Test name: scylla-master/artifacts/artifacts-ubuntu2204-arm-test
Test config file(s):

• ubuntu2204.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor 707e495b-0757-47c6-966e-4a613fa62c4f
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs 707e495b-0757-47c6-966e-4a613fa62c4f

Logs:

• sct-runner-events-707e495b.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/707e495b-0757-47c6-966e-4a613fa62c4f/20230709_113636/sct-runner-events-707e495b.tar.gz
• sct-707e495b.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/707e495b-0757-47c6-966e-4a613fa62c4f/20230709_113636/sct-707e495b.log.tar.gz

Jenkins job URL
Argus

[soyacz]

@aleksbykov isn't it related to manager you added recently in #6302?
Maybe we can postpone starting it when needed only?

[aleksbykov]

@soyacz , yes looks it is related to that code. It failed when try to initialize the new process event for counter. But it seems internal python errro, so event if it will be initialized lately, it will get same error, because addr will in used anyway. May be i am wrong. As workaround, i can add try except on launching this event-process, so start of it doesn't fail the test? WDYT?

with perf and longevities i didn't catch such error.

[soyacz]

@soyacz , yes looks it is related to that code. It failed when try to initialize the new process event for counter. But it seems internal python errro, so event if it will be initialized lately, it will get same error, because addr will in used anyway. May be i am wrong. As workaround, i can add try except on launching this event-process, so start of it doesn't fail the test? WDYT?

We don't count events in ArtifactTests, so this could be a workaround.

with perf and longevities i didn't catch such error.

Because we run only one test at a time in sct runners.

How about resigning from using manager at all and use Queue/Pipe as a bus for add/get/remove counter (or even don't get counters, just get_stats straight from EventsCounter) and make counters register internal to EventsCounter. WDYT?

[fruch]

one thing is for sure in artifacts we could get into a situation we would be running multiple jobs on the same builder as the same time.

so the failure of address is already in use, sounds related to two test running at the same time.

we shouldn't have such a limit, it cloud backfire in multiple places.

[fruch]

sounds like it's a bug in python 3.9 and 3.10
and we are pinning to 3.10.0-slim-bullseye, we should move to use 3.10.12-slim-bullseye
(and also start looking at moving to python 3.11)

[fruch]

I'm opening a PR with rebuild of the image, I think it would solve this one.