Adjust plugin thread-local APIs to account for WorkunitStore state
#15887
Milestone
Comments
stuhood
added a commit
that referenced
this issue
Jun 23, 2022
…te stdio-specific methods (#15890) As described in #15887: `StreamingWorkunit` plugins have never been able to set thread-local `WorkunitStore` state, but that apparently didn't matter until #11331 made it possible for the `StreamingWorkunitContext` file-fetching methods to encounter data which had not yet been fetched (and thus needed to create a workunit for the fetching). This change updates and "deprecates" the existing `stdio_thread_[gs]et_destination` methods (although it doesn't have access to a decorator to do that), and introduces generic thread-local state methods which include all thread-local state required by engine APIs. Fixes #15887. [ci skip-build-wheels]
stuhood
added a commit
to stuhood/pants
that referenced
this issue
Jun 23, 2022
…te stdio-specific methods (pantsbuild#15890) As described in pantsbuild#15887: `StreamingWorkunit` plugins have never been able to set thread-local `WorkunitStore` state, but that apparently didn't matter until pantsbuild#11331 made it possible for the `StreamingWorkunitContext` file-fetching methods to encounter data which had not yet been fetched (and thus needed to create a workunit for the fetching). This change updates and "deprecates" the existing `stdio_thread_[gs]et_destination` methods (although it doesn't have access to a decorator to do that), and introduces generic thread-local state methods which include all thread-local state required by engine APIs. Fixes pantsbuild#15887. [ci skip-build-wheels]
stuhood
added a commit
that referenced
this issue
Jun 23, 2022
…te stdio-specific methods (Cherry-pick of #15890) (#15907) As described in #15887: `StreamingWorkunit` plugins have never been able to set thread-local `WorkunitStore` state, but that apparently didn't matter until #11331 made it possible for the `StreamingWorkunitContext` file-fetching methods to encounter data which had not yet been fetched (and thus needed to create a workunit for the fetching). This change updates and "deprecates" the existing `stdio_thread_[gs]et_destination` methods (although it doesn't have access to a decorator to do that), and introduces generic thread-local state methods which include all thread-local state required by engine APIs. Fixes #15887. [ci skip-build-wheels]
stuhood
added a commit
to stuhood/pants
that referenced
this issue
Jun 24, 2022
…te stdio-specific methods (pantsbuild#15890) As described in pantsbuild#15887: `StreamingWorkunit` plugins have never been able to set thread-local `WorkunitStore` state, but that apparently didn't matter until pantsbuild#11331 made it possible for the `StreamingWorkunitContext` file-fetching methods to encounter data which had not yet been fetched (and thus needed to create a workunit for the fetching). This change updates and "deprecates" the existing `stdio_thread_[gs]et_destination` methods (although it doesn't have access to a decorator to do that), and introduces generic thread-local state methods which include all thread-local state required by engine APIs. Fixes pantsbuild#15887. [ci skip-build-wheels]
stuhood
added a commit
that referenced
this issue
Jun 24, 2022
…te stdio-specific methods (Cherry-pick of #15890) (#15916) As described in #15887: `StreamingWorkunit` plugins have never been able to set thread-local `WorkunitStore` state, but that apparently didn't matter until #11331 made it possible for the `StreamingWorkunitContext` file-fetching methods to encounter data which had not yet been fetched (and thus needed to create a workunit for the fetching). This change updates and "deprecates" the existing `stdio_thread_[gs]et_destination` methods (although it doesn't have access to a decorator to do that), and introduces generic thread-local state methods which include all thread-local state required by engine APIs. Fixes #15887. [ci skip-build-wheels]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When
eager_fetch=False, it's possible that a workunit's "artifacts" containDigests which haven't actually been fetched. When that's the case for aDigest, and aStreamingWorkunitplugin is using any of the context methods which fetch files from a background thread, they will encounter a:...error. That's because our existing
native_engine.stdio_thread_set_destinationstatics only set the thread localstdiostate, and not also our workunit state.To fix this, we should adjust the existing method to additionally set the workunit store. But we should also deprecate the existing method and add a new one with a more accurate name (replacing #12295).
The text was updated successfully, but these errors were encountered: