Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GCP/provisioner: Handle the RESOURCE_NOT_FOUND error. #1842

Merged
merged 2 commits into from
Apr 8, 2023
Merged

GCP/provisioner: Handle the RESOURCE_NOT_FOUND error. #1842

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1767083
GPC/provisioner: Mitigate RESOURCE_NOT_FOUND error for A100s.
concretevitamin Apr 8, 2023
116083d
Block zone.
concretevitamin Apr 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions sky/backends/cloud_vm_ray_backend.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -657,6 +657,15 @@ def _update_blocklist_on_gcp_error(
# {'code': 8, 'message': 'There is no more capacity in the zone "europe-west4-a"; you can try in another zone where Cloud TPU Nodes are offered (see https://cloud.google.com/tpu/docs/regions) [EID: 0x1bc8f9d790be9142]'} # pylint: disable=line-too-long
self._blocked_resources.add(
launchable_resources.copy(zone=zone.name))
elif code == 'RESOURCE_NOT_FOUND':
# https://github.com/skypilot-org/skypilot/issues/1797
# The VM may be alive on console. In the inner provision
# loop we have used retries to recover but failed. The
# provision loop will terminate the potentially live VMs
# and move onto the next zone. Since the VM may have been
# provisioned in this zone, it doesn't seem right to block
# the current zone.
pass
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we have done the retry in _gang_schedule_ray_up by adding that error in the need_ray_up, we should add the zone to the block list here, to avoid the outer failover loop go to that zone again.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC, without -r the outer loop will do a single pass over all zones, so this zone will not be tried again. So blocking vs. not blocking has the same effect. With -r, it's desirable to retry this zone. Is this the case?

Copy link
Collaborator

@Michaelvll Michaelvll Apr 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our yield zones will not loop through all the zones in a region for spot, as we currently rely on the optimizer to generate optimization result for per-zone. That is to say the failover for zones will happen here, but not here.
However, since we have add the to_provision in the block list here, it should explain why it does not go to that zone again. I still prefer to add that to the block list here for consistency with the other parts. Wdyt?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that's right. I got caught up thinking _yield_zones() will go through the zones. Added now.

else:
assert False, error
elif len(httperror_str) >= 1:
Expand Down Expand Up @@ -1524,6 +1533,18 @@ def need_ray_up(
'Retrying due to list request rate limit exceeded.')
return True

# https://github.com/skypilot-org/skypilot/issues/1797
# "The resource 'projects/xxx/zones/us-central1-b/instances/ray-yyy-head-<hash>-compute' was not found" # pylint: disable=line-too-long
pattern = (r'\'code\': \'RESOURCE_NOT_FOUND\'.*The resource'
r'.*instances\/.*-compute\' was not found')
result = re.search(pattern, stderr)
Michaelvll marked this conversation as resolved.
Show resolved Hide resolved
if result is not None:
# Retry. Unlikely will succeed if it's due to no capacity.
logger.info(
'Retrying due to the possibly flaky RESOURCE_NOT_FOUND '
'error.')
return True

if ('Processing file mounts' in stdout and
'Running setup commands' not in stdout and
'Failed to setup head node.' in stderr):
Expand Down