Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Workspace APIs "block" tokio threads #11329

Open
stuhood opened this issue Dec 17, 2020 · 2 comments
Open

Workspace APIs "block" tokio threads #11329

stuhood opened this issue Dec 17, 2020 · 2 comments

Comments

@stuhood
Copy link
Member

stuhood commented Dec 17, 2020 •
edited
Loading

The Workspace and InteractiveProcess APIs execute some acrobatics in order to use blocking APIs (waiting for a future to complete) on a tokio thread (because @rules run on a tokio Runtime). This can cause deadlocks with lower thread counts, and in general is less efficient.

(#10291 was a previous tracking issue for this problem)
@stuhood stuhood mentioned this issue Jun 23, 2021
Open
@stuhood stuhood changed the title Workspace and InteractiveProcess APIs "block" tokio threads Workspace APIs "block" tokio threads Dec 7, 2021
@stuhood
Copy link
Member Author

stuhood commented Dec 7, 2021

Resolving in favor of #10542.

@stuhood stuhood closed this as completed Dec 7, 2021
@stuhood stuhood reopened this May 6, 2024
@stuhood
Copy link
Member Author

stuhood commented May 6, 2024

Hm. I don't think that #10542 fully covers this: better to think of this issue as blocking #10542.

@stuhood stuhood mentioned this issue May 6, 2024
Merged
stuhood added a commit that referenced this issue May 31, 2024
@stuhood
Expose intrinsics as decorated Python functions instead (#20874)
f8b6c42 
Remove `intrinsics` facility from the engine, and instead expose
intrinsics in `intrinsics.py` via a new native call response for
generators. This solves an issue in #19730 that there were no obvious
`@rule` functions to call for intrinsics.

The native call response wraps a Rust future, which we trampoline to
execute on the Rust runtime, and then return a value to the generator.
It should work generically for any use case where we would like `@rule`
code to await native code, which opens the door to fixing #11329 by
making
[`Workspace.write_digest`](https://github.com/pantsbuild/pants/blob/00c757e77f3509dec2849982b148d41d95d6e629/src/python/pants/engine/fs.py#L290-L308)
async.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@stuhood