Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

./pants run crashes when remote_cache_eager_fetch = false #15954

Closed
danxmoran opened this issue Jun 27, 2022 · 2 comments · Fixed by #15979
Closed

./pants run crashes when remote_cache_eager_fetch = false #15954

danxmoran opened this issue Jun 27, 2022 · 2 comments · Fixed by #15979
Assignees
@stuhood
Labels
bug

Comments

@danxmoran
Copy link
Contributor

Describe the bug

After setting remote_cache_eager_fetch = false, a CI step that ./pants runs a pex_binary has started (intermittently) failing with errors like:

 Exception: Was not present in either the local or remote store: Digest { hash: Fingerprint<46683dec8706b7ac7c4f6011f68b4b8c10ad423ae8ba57745a6f5e01ba5b2f7b>, size_bytes: 11827 }

Pants version

PANTS_SHA=5d8a328d72209863986c8959b20305505bc068ba

OS

Linux

Additional info

Some BuildSense links where we've seen the failure:
@danxmoran danxmoran added the bug label Jun 27, 2022
@danxmoran
Copy link
Contributor Author

cc @stuhood maybe another race condition with lazy remote fetches

@stuhood stuhood self-assigned this Jun 27, 2022
@stuhood
Copy link
Member

stuhood commented Jun 27, 2022 •
edited
Loading

Mm, thanks: I see what this is. Yet another TODO that I should have assumed would be a blocker!

Sorry for the trouble. We're prioritizing getting the 2.12.x release out early this week, but I should be able to get a patch into 2.13.x for this by EOD Wednesday-ish.

@stuhood stuhood mentioned this issue Jun 28, 2022
Merged
stuhood added a commit that referenced this issue Jun 29, 2022
@stuhood
Backtrack through calls from @rules to synchronous engine methods (#…
ba9f863 
…15979)

As reported in #15954, backtracking doesn't currently work when a synchronous method like `Workspace.write_digests` is the source of a `MissingDigest` error. This was due to a TODO left behind in #15761, where we did not propagate the `Failure` type "through" `@rule` bodies.

To fix this, we add a conversion from `Failure` to `PrErr` which wraps in a well known exception type, and then look for that type when converting back from `PyErr` to `Failure`.

Fixes #15954.
tdyas pushed a commit to tdyas/pants that referenced this issue Jun 29, 2022
@stuhood
Backtrack through calls from @rules to synchronous engine methods (p…
0729907 
…antsbuild#15979)

As reported in pantsbuild#15954, backtracking doesn't currently work when a synchronous method like `Workspace.write_digests` is the source of a `MissingDigest` error. This was due to a TODO left behind in pantsbuild#15761, where we did not propagate the `Failure` type "through" `@rule` bodies.

To fix this, we add a conversion from `Failure` to `PrErr` which wraps in a well known exception type, and then look for that type when converting back from `PyErr` to `Failure`.

Fixes pantsbuild#15954.
# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
@tdyas tdyas mentioned this issue Jun 29, 2022
Merged
stuhood added a commit to stuhood/pants that referenced this issue Jun 29, 2022
@stuhood
Backtrack through calls from @rules to synchronous engine methods (p…
0397543 
…antsbuild#15979)

As reported in pantsbuild#15954, backtracking doesn't currently work when a synchronous method like `Workspace.write_digests` is the source of a `MissingDigest` error. This was due to a TODO left behind in pantsbuild#15761, where we did not propagate the `Failure` type "through" `@rule` bodies.

To fix this, we add a conversion from `Failure` to `PrErr` which wraps in a well known exception type, and then look for that type when converting back from `PyErr` to `Failure`.

Fixes pantsbuild#15954.
# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
@stuhood stuhood mentioned this issue Jun 29, 2022
Closed
stuhood added a commit that referenced this issue Jun 29, 2022
@stuhood
Backtrack through calls from @rules to synchronous engine methods (…
48a6a53 
…Cherry-pick of #15979) (#16001)

As reported in #15954, backtracking doesn't currently work when a synchronous method like `Workspace.write_digests` is the source of a `MissingDigest` error. This was due to a TODO left behind in #15761, where we did not propagate the `Failure` type "through" `@rule` bodies.

To fix this, we add a conversion from `Failure` to `PrErr` which wraps in a well known exception type, and then look for that type when converting back from `PyErr` to `Failure`.

Fixes #15954.

[ci skip-build-wheels]

Co-authored-by: Stu Hood <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stuhood stuhood

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Backtrack through calls from @rules to synchronous engine methods stuhood/pants
2 participants
@stuhood @danxmoran