Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Task][Go SDK]: Determine canonical local temp directory. #27569

Open
1 of 15 tasks
lostluck opened this issue Jul 19, 2023 · 0 comments
Open
1 of 15 tasks

[Task][Go SDK]: Determine canonical local temp directory. #27569

lostluck opened this issue Jul 19, 2023 · 0 comments
Assignees

Comments

@lostluck
Copy link
Contributor

What needs to happen?

It's common to have a canonical location for outputs, logs, files, and other miscelanea that may be needed by the SDK or for the prism runner, that needs to be accessible during the job lifetime.

eg. For loopback mode jobs for Go, where the binary is unused, it's simpler/and faster to have an empty "binary" file that is uploaded to the runner if needed than to compile and upload an unused binary. So it's useful for the SDK to have a specific place to put this empty file, along with other files for the job.

In the same vein, a canonical place to compile that binary to would avoid cluttering user code directories with binaries, vs something for the specific purpose of the job, and copying any manually specified binary as well.

Other uses would be to write the JSON version of the Pipeline proto, and other request information. Then printing out this directory path. This avoids unnecessary cruft being logged on job startup, and provides a cannon folder with artifacts to upload for debugging purposes.


Further, in prism, it would be good to have a canonical directory when it begins to have artifacts, for restarts and similar, inside the same folder tree.


I'm aware that Python and the Go xlang code probably already has this, so I've filed this largely so I can remember to hunt for it, and make it obviously canonical with a common package.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant