Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix "the resource was not found" instance errors in CI #5366

Closed
teor2345 opened this issue Oct 10, 2022 · 5 comments
Closed

Fix "the resource was not found" instance errors in CI #5366

teor2345 opened this issue Oct 10, 2022 · 5 comments
Labels
C-bug Category: This is a bug I-integration-fail Continuous integration fails, including build and test failures

Comments

@teor2345
Copy link
Contributor

Motivation

I'm seeing errors like this in CI:

  • The resource 'projects/zealous-zebra/zones/us-central1-a/instances/update-to-tip-5360-merge-818a593' was not found

https://github.com/ZcashFoundation/zebra/actions/runs/3203311358/jobs/5239744583#step:6:101

Command Reference

https://cloud.google.com/sdk/gcloud/reference/topic/filters

Designs

I'm not sure what is causing these bugs, we only delete instances after 3 days:

INSTANCES=$(gcloud compute instances list --sort-by=creationTimestamp --filter="name~-[0-9a-f]{7,}$ AND creationTimestamp < $DELETE_BEFORE_DATE" --format='value(NAME,ZONE)' | \

And after the job fails:

INSTANCE=$(gcloud compute instances list --filter=${{ inputs.test_id }}-${{ env.GITHUB_REF_SLUG_URL }}-${{ env.GITHUB_SHA_SHORT }} --format='value(NAME)')

Maybe we got a failed instance or outdated instance delete script wrong somewhere?
We could try using name={{key}} for failed job deletes, but it might not be causing this problem.

Let's check the Google Cloud logs to work out what is deleting these instances.

@teor2345 teor2345 added C-bug Category: This is a bug S-needs-triage Status: A bug report needs triage P-Medium ⚡ I-integration-fail Continuous integration fails, including build and test failures labels Oct 10, 2022
@teor2345
Copy link
Contributor Author

teor2345 commented Nov 4, 2022

Failed in PR #5541, halfway through the test run:
https://github.com/ZcashFoundation/zebra/actions/runs/3390759314/jobs/5636067584#step:6:106

@mpguerra
Copy link
Contributor

mpguerra commented Nov 8, 2022

@teor2345
Copy link
Contributor Author

@gustavovalverde did this get fixed by the recent changes?

@teor2345
Copy link
Contributor Author

teor2345 commented Dec 1, 2022

We haven't seen this bug for a while.

@teor2345
Copy link
Contributor Author

I haven't seen these in ages?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Category: This is a bug I-integration-fail Continuous integration fails, including build and test failures
Projects
None yet
Development

No branches or pull requests

2 participants