Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(pipeline): wait for Terraform lock to be released #876

Merged
merged 1 commit into from
Aug 25, 2022

Conversation

afeld
Copy link
Contributor

@afeld afeld commented Aug 25, 2022

Avoids race condition happening between pipeline runs.

@afeld afeld requested a review from a team as a code owner August 25, 2022 14:54
@github-actions github-actions bot added the infrastructure Terraform, Azure, etc. label Aug 25, 2022
Avoids race condition happening between pipeline runs.
@afeld afeld force-pushed the chore/terraform-race branch from 2403a60 to ba7eb81 Compare August 25, 2022 14:57
@afeld afeld mentioned this pull request Aug 25, 2022
15 tasks
thekaveman
thekaveman previously approved these changes Aug 25, 2022
Copy link
Member

@thekaveman thekaveman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great!

Should we have expected to see the pipeline succeed with this PR? Or not until the change makes it into the dev branch?

Edit: seems like the wait was part of the command, but it still failed:

image

@thekaveman thekaveman dismissed their stale review August 25, 2022 15:46

The wait didn't clear the lock

@afeld
Copy link
Contributor Author

afeld commented Aug 25, 2022

Appears the Terraform lock wasn't released from an operation on 8/22 for some reason. Forced unlock and re-ran the Job, which succeed.

In other words, this pull request seems to have "worked" since it waited two minutes for the lock to be released, but that never happened, so it exited. Why the lock wasn't released before is another question — let me know if you see the pipeline failing at other unexpected times and we can dig in.

@afeld afeld merged commit 82b76d8 into dev Aug 25, 2022
@afeld afeld deleted the chore/terraform-race branch August 25, 2022 16:33
@angela-tran
Copy link
Member

angela-tran commented Sep 27, 2022

Why the lock wasn't released before is another question — let me know if you see the pipeline failing at other unexpected times and we can dig in.

🤔 The Azure pipeline started failing again yesterday with the same error about not being able to acquire the lock.

For example, #956 failed (Azure pipeline log)

Other PRs failing:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Terraform, Azure, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants