Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provide mechanism to stream action output #7213

Open
picnoir opened this issue Jan 22, 2019 · 8 comments
Open

Provide mechanism to stream action output #7213

picnoir opened this issue Jan 22, 2019 · 8 comments
Labels
P3 We're not considering working on this, but happy to review a PR. (No assignee) team-Local-Exec Issues and PRs for the Execution (Local) team type: feature request

Comments

@picnoir
Copy link

picnoir commented Jan 22, 2019

Description of the feature request:

We currently cannot stream a build rule output to the terminal. We could replicate the analogous test flag --test_output=streamed in the build execution phase.

In that case, this would probably mean we would have to run the build actions sequentially in order to prevent any race condition on the terminal output.

Feature requests: what underlying problem are you trying to solve with this feature?

Some building tools [1] are not guaranteed to terminate. For such tools, having access to stdout/stderr is the only way we get to:

  1. Realize the build won't terminate.
  2. Investigate on what makes the build looping.

[1] Tools such as GHC, the Haskell compiler. See tweag/rules_haskell#603
@irengrig irengrig added untriaged team-Bazel General Bazel product/strategy issues labels Jan 22, 2019
@RNabel
Copy link
Contributor

RNabel commented Feb 8, 2019

This would be super useful for build rule authors in general, since debugging a hanging tool with no access to stdout/stderr is super difficult.

For what it's worth, one approach to debugging a sh_binary is to triage the error location within the script by adding exit 1 (which results in all output being printed).

@dslomov dslomov added team-Local-Exec Issues and PRs for the Execution (Local) team and removed team-Bazel General Bazel product/strategy issues labels Jul 5, 2019
@susinmotion
Copy link
Contributor

A possible workaround is using bazel build -s ... to print the subcommands that bazel is running. Then you can copy paste the one that gets stuck if you need to troubleshoot. You can also disable parallelism by passing --jobs=1. Does that help?

@susinmotion susinmotion added P3 We're not considering working on this, but happy to review a PR. (No assignee) type: feature request and removed untriaged labels Sep 25, 2019
@jerrymarino jerrymarino mentioned this issue Oct 23, 2019
Merged
@jmmv jmmv changed the title Cannot stream Bazel build output Provide mechanism to stream action output May 13, 2020
@uri-canva uri-canva mentioned this issue Oct 26, 2020
Open
@burdiyan
Copy link

I was so surprised when I realised there's no easy way to stream action build output in Bazel. It seems like a very necessary thing to be able to do. I have a very long build process, and it would be very helpful to be able to stream the output.

So definitely a +1 from me to implement this.

@pauldraper
Copy link
Contributor

  1. Realize the build won't terminate.

I am here.

@QuantamHD
Copy link

+1 For super long running actions. It's nice to be able to see where they are at progress wise

dgp1130 added a commit to dgp1130/rules_prerender that referenced this issue Mar 9, 2023
@dgp1130
Adds --debug_prerender flag.
917d976 
This makes it possible to interactively debug a renderer build action. There are a few DX challenges I'm not able to fully address unfortunately:
1. Debugging is a bit wonky from the Bazel cache as I couldn't find a good way to turn it off for a specific action (`no-cache` does _not_ work). I started a thread in [#rules](https://bazelbuild.slack.com/archives/CDCMRLS23/p1678340235374509), so hopefully someone can help me out here.
2. `bazel build` hides output from stderr until the action fails, and there is no equivalent of `--test_output=streamed` for individual build actions. This means the debugged target appears to hang and fails to print that it is waiting for the debugger. I tried to call this out in the README, but I can definitely see users getting tripped up from this. See bazelbuild/bazel#7213.
3. I used a proper flag target (`@rules_prerender//tools/flags:debug_prerender`), though this is quite a lot to type so I recommended an alias in the `.bazelrc` file. Unfortunately it's hard to automatically set that up for users so the best I can do is document it.
dgp1130 added a commit to dgp1130/rules_prerender that referenced this issue Mar 9, 2023
@dgp1130
Adds --debug_prerender flag.
33f6c94 
This makes it possible to interactively debug a renderer build action. There are a few DX challenges I'm not able to fully address unfortunately:
1. Debugging is a bit wonky from the Bazel cache as I couldn't find a good way to turn it off for a specific action (`no-cache` does _not_ work). I started a thread in [#rules](https://bazelbuild.slack.com/archives/CDCMRLS23/p1678340235374509), so hopefully someone can help me out here.
2. `bazel build` hides output from stderr until the action fails, and there is no equivalent of `--test_output=streamed` for individual build actions. This means the debugged target appears to hang and fails to print that it is waiting for the debugger. I tried to call this out in the README, but I can definitely see users getting tripped up from this. See bazelbuild/bazel#7213.
3. I used a proper flag target (`@rules_prerender//tools/flags:debug_prerender`), though this is quite a lot to type so I recommended an alias in the `.bazelrc` file. Unfortunately it's hard to automatically set that up for users so the best I can do is document it.
4. I added a `debug_target` attribute which allows a target to recognize itself as the target being debugged, based on a _different_ target label. This is nice because the actual target invoking the renderer is typically something like `:site_page_annotated` due to passing through a few macros of indirection. Using `debug_target` allows us to pass through the label users actually author in their `BUILD.bazel` files and would expect to debug like `:site`. It's a fairly nuanced API, but I decided to keep it public since it could be useful for macros outside of those in `@rules_prerender`.
5. I'm fairly sure this doesn't really work for inspecting external targets (`@foo//:site`), but neither does `prerender_pages()` in general right now, so I'm not too worried about it atm.
@github-actions
Copy link

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

@github-actions github-actions bot added the stale Issues or PRs that are stale (no activity for 30 days) label Jun 13, 2023
@domfarolino
Copy link

@bazelbuild/triage

@sgowroji sgowroji removed the stale Issues or PRs that are stale (no activity for 30 days) label Jun 13, 2023
@appthumb
Copy link

+1 on prioritizing this. This is the no.1 blocker for using Bazel extensively to manage slow and complex setups and preprocessing, e.g., for preparing machine learning training data etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P3 We're not considering working on this, but happy to review a PR. (No assignee) team-Local-Exec Issues and PRs for the Execution (Local) team type: feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests