You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It's common to have a canonical location for outputs, logs, files, and other miscelanea that may be needed by the SDK or for the prism runner, that needs to be accessible during the job lifetime.
eg. For loopback mode jobs for Go, where the binary is unused, it's simpler/and faster to have an empty "binary" file that is uploaded to the runner if needed than to compile and upload an unused binary. So it's useful for the SDK to have a specific place to put this empty file, along with other files for the job.
In the same vein, a canonical place to compile that binary to would avoid cluttering user code directories with binaries, vs something for the specific purpose of the job, and copying any manually specified binary as well.
Other uses would be to write the JSON version of the Pipeline proto, and other request information. Then printing out this directory path. This avoids unnecessary cruft being logged on job startup, and provides a cannon folder with artifacts to upload for debugging purposes.
Further, in prism, it would be good to have a canonical directory when it begins to have artifacts, for restarts and similar, inside the same folder tree.
I'm aware that Python and the Go xlang code probably already has this, so I've filed this largely so I can remember to hunt for it, and make it obviously canonical with a common package.
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components
Component: Python SDK
Component: Java SDK
Component: Go SDK
Component: Typescript SDK
Component: IO connector
Component: Beam examples
Component: Beam playground
Component: Beam katas
Component: Website
Component: Spark Runner
Component: Flink Runner
Component: Samza Runner
Component: Twister2 Runner
Component: Hazelcast Jet Runner
Component: Google Cloud Dataflow Runner
The text was updated successfully, but these errors were encountered:
What needs to happen?
It's common to have a canonical location for outputs, logs, files, and other miscelanea that may be needed by the SDK or for the prism runner, that needs to be accessible during the job lifetime.
eg. For loopback mode jobs for Go, where the binary is unused, it's simpler/and faster to have an empty "binary" file that is uploaded to the runner if needed than to compile and upload an unused binary. So it's useful for the SDK to have a specific place to put this empty file, along with other files for the job.
In the same vein, a canonical place to compile that binary to would avoid cluttering user code directories with binaries, vs something for the specific purpose of the job, and copying any manually specified binary as well.
Other uses would be to write the JSON version of the Pipeline proto, and other request information. Then printing out this directory path. This avoids unnecessary cruft being logged on job startup, and provides a cannon folder with artifacts to upload for debugging purposes.
Further, in prism, it would be good to have a canonical directory when it begins to have artifacts, for restarts and similar, inside the same folder tree.
I'm aware that Python and the Go xlang code probably already has this, so I've filed this largely so I can remember to hunt for it, and make it obviously canonical with a common package.
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components
The text was updated successfully, but these errors were encountered: