🐛 Dynamic sidecar: failing service on freshly started node #3836

sanderegg · 2023-02-03T15:24:36Z

What do these changes do?

increase the COMPOSE_HTTP_TIMEOUT env variable when calling docker-compose from the dynamic sidecar on a freshly created node. NOTE: this will be deprecated with docker-compose V2
fixes the issue with starting s4l-lite on a fresh node

references:

Related issue/s

How to test

Checklist

odeimaiz

Awesome, thanks!

codeclimate · 2023-02-03T15:39:21Z

Code Climate has analyzed commit 8a4d1d3 and detected 0 issues on this pull request.

View more on Code Climate.

sonarqubecloud · 2023-02-03T15:39:42Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

pcrespov

Will fix the issue but we could export environs more reliably using the env and even add the values as contants? next refactoring perhapse :-)

codecov · 2023-02-03T15:52:53Z

Codecov Report

Merging #3836 (8a4d1d3) into master (4d340e2) will decrease coverage by 16.5%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##           master   #3836      +/-   ##
=========================================
- Coverage    84.9%   68.5%   -16.5%     
=========================================
  Files         927     416     -511     
  Lines       39701   21498   -18203     
  Branches      836     137     -699     
=========================================
- Hits        33741   14731   -19010     
- Misses       5740    6718     +978     
+ Partials      220      49     -171

Flag	Coverage Δ
integrationtests	`66.6% <ø> (-0.1%)`	⬇️
unittests	`87.0% <100.0%> (+4.8%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...rvice_dynamic_sidecar/core/docker_compose_utils.py	`94.9% <ø> (ø)`
...vice_dynamic_sidecar/modules/long_running_tasks.py	`95.4% <100.0%> (+<0.1%)`	⬆️
...-v2/src/simcore_service_director_v2/cli/_client.py	`0.0% <0.0%> (-100.0%)`	⬇️
...v2/src/simcore_service_director_v2/cli/__init__.py	`0.0% <0.0%> (-100.0%)`	⬇️
...r/src/simcore_service_webserver/storage_schemas.py	`0.0% <0.0%> (-100.0%)`	⬇️
...service_webserver/projects/_project_models_rest.py	`0.0% <0.0%> (-100.0%)`	⬇️
...service_director_v2/cli/_close_and_save_service.py	`0.0% <0.0%> (-100.0%)`	⬇️
...odules/dynamic_sidecar/scheduler/_core2/_errors.py	`0.0% <0.0%> (-100.0%)`	⬇️
...odules/dynamic_sidecar/scheduler/_core2/_marker.py	`0.0% <0.0%> (-100.0%)`	⬇️
...odules/dynamic_sidecar/scheduler/_core2/_models.py	`0.0% <0.0%> (-100.0%)`	⬇️
... and 715 more

pcrespov · 2023-02-07T10:25:14Z

This fix should solve logs like:

023-02-07T10:11:08.964392399Z Creating network "dy-sidecar_d1138a1d-948c-5fea-ae57-c136ec4366cb_back----end" with the default driver
2023-02-07T10:11:08.964392910Z Creating dy-sidecar-d1138a1d-948c-5fea-ae57-c136ec4366cb-2-s4l-core-stre ...
2023-02-07T10:11:08.964393589Z Creating dy-sidecar-d1138a1d-948c-5fea-ae57-c136ec4366cb-3-sym-server    ...
2023-02-07T10:11:08.964394231Z Creating dy-sidecar-d1138a1d-948c-5fea-ae57-c136ec4366cb-1-s4l-core      ...
2023-02-07T10:11:08.964394662Z
2023-02-07T10:11:08.964395069Z ERROR: for dy-sidecar-d1138a1d-948c-5fea-ae57-c136ec4366cb-3-sym-server  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
2023-02-07T10:11:08.964395596Z
2023-02-07T10:11:08.964395966Z ERROR: for dy-sidecar-d1138a1d-948c-5fea-ae57-c136ec4366cb-2-s4l-core-stre  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
2023-02-07T10:11:08.964396510Z
2023-02-07T10:11:08.964396895Z ERROR: for dy-sidecar-d1138a1d-948c-5fea-ae57-c136ec4366cb-1-s4l-core  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
2023-02-07T10:11:08.964397320Z
2023-02-07T10:11:08.964397812Z ERROR: for dy-sidecar-d1138a1d-948c-5fea-ae57-c136ec4366cb-3-sym-server  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
2023-02-07T10:11:08.964398121Z
2023-02-07T10:11:08.964398389Z ERROR: for dy-sidecar-d1138a1d-948c-5fea-ae57-c136ec4366cb-2-s4l-core-stre  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
2023-02-07T10:11:08.964398832Z
2023-02-07T10:11:08.964399095Z ERROR: for dy-sidecar-d1138a1d-948c-5fea-ae57-c136ec4366cb-1-s4l-core  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
2023-02-07T10:11:08.964399515Z An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
2023-02-07T10:11:08.964399901Z If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
2023-02-07T10:11:08.964400262Z '
2023-02-07T10:11:08.964400863Z   File "/home/scu/.venv/lib/python3.9/site-packages/servicelib/long_running_tasks/_task.py", line 418, in _progress_task
2023-02-07T10:11:08.964401255Z     return await handler(progress, **task_kwargs)
2023-02-07T10:11:08.964401680Z
2023-02-07T10:11:08.964402090Z   File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_dynamic_sidecar/modules/long_running_tasks.py", line 138, in task_create_service_containers
2023-02-07T10:11:08.964402680Z     await _retry_docker_compose_create(shared_store.compose_spec, settings)
2023-02-07T10:11:08.964403065Z
2023-02-07T10:11:08.964403447Z   File "/home/scu/.venv/lib/python3.9/site-packages/tenacity/_asyncio.py", line 86, in async_wrapped
2023-02-07T10:11:08.964403818Z     return await fn(*args, **kwargs)
2023-02-07T10:11:08.964404175Z
2023-02-07T10:11:08.964404783Z   File "/home/scu/.venv/lib/python3.9/site-packages/tenacity/_asyncio.py", line 48, in __call__
2023-02-07T10:11:08.964405251Z     do = self.iter(retry_state=retry_state)
2023-02-07T10:11:08.964405931Z
2023-02-07T10:11:08.964406312Z   File "/home/scu/.venv/lib/python3.9/site-packages/tenacity/__init__.py", line 349, in iter
2023-02-07T10:11:08.964406844Z     return fut.result()
2023-02-07T10:11:08.964407198Z
2023-02-07T10:11:08.964407505Z   File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
2023-02-07T10:11:08.964407864Z     return self.__get_result()
2023-02-07T10:11:08.964408147Z
2023-02-07T10:11:08.964408560Z   File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
2023-02-07T10:11:08.964408851Z     raise self._exception
2023-02-07T10:11:08.964409312Z
2023-02-07T10:11:08.964409685Z   File "/home/scu/.venv/lib/python3.9/site-packages/tenacity/_asyncio.py", line 51, in __call__
2023-02-07T10:11:08.964409980Z     result = await fn(*args, **kwargs)
2023-02-07T10:11:08.964410418Z
2023-02-07T10:11:08.964411013Z   File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_dynamic_sidecar/modules/long_running_tasks.py", line 89, in _retry_docker_compose_create
2023-02-07T10:11:08.964411413Z     _raise_for_errors(result, "up")
2023-02-07T10:11:08.964411763Z
2023-02-07T10:11:08.964412169Z   File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_dynamic_sidecar/modules/long_running_tasks.py", line 61, in _raise_for_errors
2023-02-07T10:11:08.964412489Z     raise RuntimeError(command_result.message)
2023-02-07T10:11:08.964412942Z
2023-02-07T10:11:14.758789102Z WARNING:simcore_service_director_v2.modules.dynamic_sidecar.scheduler._core._events_utils:Could not remove service containers for dy-sidecar_d1138a1d-948c-5fea-ae57-c136ec4366cb
2023-02-07T10:11:14.758795127Z Task simcore_service_dynamic_sidecar.modules.long_running_tasks.task_runs_docker_compose_down.189254e2-9608-4fae-a398-2b8c053c2857 finished with exception: ''dy-sidecar-d1138a1d-948c-5fea-ae57-c136ec4366cb-0-rt-web-lite''
2023-02-07T10:11:14.758795866Z   File "/home/scu/.venv/lib/python3.9/site-packages/servicelib/long_running_tasks/_task.py", line 418, in _progress_task
2023-02-07T10:11:14.758796565Z     return await handler(progress, **task_kwargs)
2023-02-07T10:11:14.758797072Z
2023-02-07T10:11:14.758797677Z   File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_dynamic_sidecar/modules/long_running_tasks.py", line 175, in task_runs_docker_compose_down
2023-02-07T10:11:14.758798109Z     await stop_log_fetching(app, container_name)
2023-02-07T10:11:14.758798608Z
2023-02-07T10:11:14.758799030Z   File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_dynamic_sidecar/core/docker_logs.py", line 94, in stop_log_fetching
2023-02-07T10:11:14.758800223Z     await background_log_fetcher.stop_log_fetching(container_name)
2023-02-07T10:11:14.758800612Z
2023-02-07T10:11:14.758800894Z   File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_dynamic_sidecar/core/docker_logs.py", line 65, in stop_log_fetching
2023-02-07T10:11:14.758801388Z     task = self._log_processor_tasks.pop(container_name)
2023-02-07T10:11:14.758801793Z . Will continue to save the data from the service!

sanderegg added the changelog:🐛bugfix label Feb 3, 2023

sanderegg added this to the Resistance Is Futile milestone Feb 3, 2023

sanderegg requested a review from GitHK as a code owner February 3, 2023 15:24

sanderegg self-assigned this Feb 3, 2023

sanderegg requested review from pcrespov, odeimaiz, mrnicegyu11 and Surfict February 3, 2023 15:31

odeimaiz approved these changes Feb 3, 2023

View reviewed changes

sanderegg added 3 commits February 3, 2023 16:37

fixes startup of sim4life

556b260

improve logs in case of retrial

6bae28e

added more info

8a4d1d3

sanderegg force-pushed the dynamic-sidecar/bugfix/failing_service_on_fresh_node branch from 6383fc4 to 8a4d1d3 Compare February 3, 2023 15:38

pcrespov approved these changes Feb 3, 2023

View reviewed changes

pcrespov reviewed Feb 3, 2023

View reviewed changes

sanderegg merged commit 23308d1 into ITISFoundation:master Feb 3, 2023

sanderegg deleted the dynamic-sidecar/bugfix/failing_service_on_fresh_node branch February 3, 2023 16:27

GitHK mentioned this pull request Apr 21, 2023

Docker compose timing out #4144

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Dynamic sidecar: failing service on freshly started node #3836

🐛 Dynamic sidecar: failing service on freshly started node #3836

sanderegg commented Feb 3, 2023 •

edited

Loading

odeimaiz left a comment

codeclimate bot commented Feb 3, 2023

sonarqubecloud bot commented Feb 3, 2023

pcrespov left a comment

codecov bot commented Feb 3, 2023 •

edited

Loading

pcrespov commented Feb 7, 2023

🐛 Dynamic sidecar: failing service on freshly started node #3836

🐛 Dynamic sidecar: failing service on freshly started node #3836

Conversation

sanderegg commented Feb 3, 2023 • edited Loading

What do these changes do?

Related issue/s

How to test

Checklist

odeimaiz left a comment

Choose a reason for hiding this comment

codeclimate bot commented Feb 3, 2023

sonarqubecloud bot commented Feb 3, 2023

pcrespov left a comment

Choose a reason for hiding this comment

codecov bot commented Feb 3, 2023 • edited Loading

Codecov Report

pcrespov commented Feb 7, 2023

sanderegg commented Feb 3, 2023 •

edited

Loading

codecov bot commented Feb 3, 2023 •

edited

Loading