You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As part of the service shutdown proceed, director-v2 will try to remove the used volumes of the service from the node via the agent. If the node is missing for whatever reason, the below error would be raised.
This fix allows the director-v2 to no longer fail when the node with the required running agent instance is missing.
TaskClientResultError: Task simcore_service_director_v2.api.routes.dynamic_scheduler._task_cleanup_service_docker_resources.a8f28f40-2b63-4c86-99b7-fc6e71241cc6 finished with exception: 'Could not find a remote method named:
'docker_node_id_r77hxq9vws8tvo8wmyumug2y9-service_agent-swarm_stack_name_production-simcore.remove_volumes_without_backup_for_service'. Message from remote server was returned: IncomingMessage:{'app_id': None,
'body_size': 73,
'cluster_id': '',
'consumer_tag': None,
'content_encoding': None,
'content_type': 'application/python-pickle',
'correlation_id': '3d52bc9a-8805-43af-ae9b-b65ba374f012',
'delivery_mode': <DeliveryMode.NOT_PERSISTENT: 1>,
'delivery_tag': None,
'exchange': '',
'expiration': 3600.0,
'headers': {'From': 'amq_617014d185f64dd48bac7fcb1e899202'},
'message_id': 'c632f220bf3a44489ac00bca99977605',
'priority': 5,
'redelivered': None,
'reply_to': 'amq_617014d185f64dd48bac7fcb1e899202',
'routing_key': 'docker_node_id_r77hxq9vws8tvo8wmyumug2y9-service_agent-swarm_stack_name_production-simcore.remove_volumes_without_backup_for_service',
'timestamp': datetime.datetime(2025, 1, 14, 8, 53, 58, tzinfo=datetime.timezone.utc),
'type': 'call',
'user_id': None}. '
File "/home/scu/.venv/lib/python3.11/site-packages/servicelib/long_running_tasks/_task.py", line 422, in _progress_task
return await handler(progress, **task_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/api/routes/dynamic_scheduler.py", line 246, in _task_cleanup_service_docker_resources
await dynamic_sidecars_scheduler.remove_service_sidecar_proxy_docker_networks_and_volumes(
File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_task.py", line 63, in remove_service_sidecar_proxy_docker_networks_and_volumes
return await self.scheduler.remove_service_sidecar_proxy_docker_networks_and_volumes(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_scheduler.py", line 215, in remove_service_sidecar_proxy_docker_networks_and_volumes
await service_remove_sidecar_proxy_docker_networks_and_volumes(
File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_events_utils.py", line 242, in service_remove_sidecar_proxy_docker_networks_and_volumes
await remove_volumes_without_backup_for_service(
File "/home/scu/.venv/lib/python3.11/site-packages/servicelib/logging_utils.py", line 305, in _async_wrapper
result = await func_or_coro(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/scu/.venv/lib/python3.11/site-packages/servicelib/rabbitmq/rpc_interfaces/agent/volumes.py", line 24, in remove_volumes_without_backup_for_service
result = await rabbitmq_rpc_client.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/scu/.venv/lib/python3.11/site-packages/servicelib/rabbitmq/_client_rpc.py", line 99, in request
raise RemoteMethodNotRegisteredError(
Q: @GitHK So sometimes when I run manually director-v2 CLI to stop the hanging sidecars and it fails that the node ID is not in the DB anymore, this will fix it?
Q: @GitHK So sometimes when I run manually director-v2 CLI to stop the hanging sidecars and it fails that the node ID is not in the DB anymore, this will fix it?
@matusdrobuliak66 that is a different issue and unrelated to this one
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHK
added
a:director-v2
What do these changes do?
As part of the service shutdown proceed,
director-v2will try to remove the used volumes of the service from the node via the agent. If the node is missing for whatever reason, the below error would be raised.This fix allows the director-v2 to no longer fail when the node with the required running agent instance is missing.
Related issue/s
How to test
Dev-ops checklist