Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bazel 8 intermittently hangs with remote cache enabled. #25308

Open
leungster opened this issue Feb 18, 2025 · 2 comments
Open

Bazel 8 intermittently hangs with remote cache enabled. #25308

leungster opened this issue Feb 18, 2025 · 2 comments
Assignees
Labels
team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug untriaged

Comments

@leungster
Copy link

Description of the bug:

Bazel has been randomly hanging for us after upgrading to v8. We're currently on v8.1.0 and still seeing hangs.

The hanging logs manifest as (4 actions, 0 running) and never progresses. If we disable remote caching then it appears to stop hanging.

When we reverted to 7.5.0, the hangs went away.

Which category does this issue belong to?

Local Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

We have a jstack of a hanging build. I can't reliably make it happen since it seems to be random. If we cancel and retry the build it will work.

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

release 8.1.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?


If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

Seems to be somewhere between 7 and 8 but not sure which commit.

Have you found anything relevant by searching the web?

The fix for #25233 doesn't seem to help for us.

Any other information, logs, or outputs that you want to share?

jstack.txt

@fmeum
Copy link
Collaborator

fmeum commented Feb 18, 2025

Based on the thread dump, it looks like your client is stuck querying the capabilities of the server. It's not an obvious deadlock. When you say that it never progresses, how long have you waited for?

@leungster
Copy link
Author

When it gets into the hanging scenario, it'll hang for up to 6 hours and then our CI times out the process.

@sgowroji sgowroji added the team-Remote-Exec Issues and PRs for the Execution (Remote) team label Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug untriaged
Projects
None yet
Development

No branches or pull requests

6 participants