`[command]…` output is prepended to exec `outStream` #649

Swatinem · 2020-11-25T08:51:00Z

Describe the bug
I want to redirect a commands output to a file, but I still want the command that I execute to be logged.
Currently the [command]… log is streamed into the outStream, and not to stdout as I would expect.

To Reproduce
Steps to reproduce the behavior:

await exec.exec("echo", ["foobar"], { outStream: fs.createWriteStream("foobar.txt") });

Expected behavior
I want the foobar.txt file to only have foobar in it.
Instead, it will have the [command]echo foobar command output prepended.

Additional context

Not quite sure if this is intentional?

toolkit/packages/exec/src/toolrunner.ts

Lines 424 to 428 in 2bf7365

    
           if (!optionsNonNull.silent && optionsNonNull.outStream) { 
        
             optionsNonNull.outStream.write( 
        
               this._getCommandString(optionsNonNull) + os.EOL 
        
             ) 
        
           }

The text was updated successfully, but these errors were encountered:

actions/toolkit#649

fearphage · 2021-08-12T17:23:15Z

It looks like the fix for this is to set the silent option to true, but that results in nothing being written to the outStream strangely.

I worked around this by doing two separate commands:

const { stdout } = await exec.getExecOutput(...);

fs.writeFile(filePath, stdout, options);

Presently if the exec'd tool is expected to produce an output (i.e. to write to stdout) that can be consumed in a Unix pipeline, assigned to a variable, or similar; the `silent` option must be set in order to avoid the output being mangled by `@actions/exec` to include the full command string that it spawns. This commit writes the spawned command string to stderr instead of stdout, as would be expected for logging/debug information as opposed to consumable output data. Closes actions#649.

OJFord · 2023-10-26T16:34:38Z

An example(s) of this causing a problem in the wild are available at hashicorp/setup-terraform#367.

Presently if the exec'd tool is expected to produce an output (i.e. to write to stdout) that can be consumed in a Unix pipeline, assigned to a variable, or similar; the `silent` option must be set in order to avoid the output being mangled by `@actions/exec` to include the full command string that it spawns (and then re-writing out from the provided stdout listener). This commit writes the spawned command string to stderr instead of stdout, as would be expected for logging/debug information as opposed to consumable output data. Closes actions#649.

@austinvalle

Presently using a command such as `terraform output -json | jq` does not work with the wrapper enabled, as it is by default. In order to consume terraform's output having set it up with this Action, it is necessary either to disable the wrapper (`with: terraform_wrapper: false`) or run it in its own Actions step with an explicit `id` (e.g. `id: foo`) so that it can be referred to and consumed (`${{steps.foo.outputs.stdout}}` et al.) in later steps. This seems to be the result of much confusion (issues passim) and is not at all easy (hashicorp#338) to debug/diagnose and come to the realisation that it's due to the wrapper, or even that such a thing exists. @austinvalle identified the issue as being due to the `@actions/exec` package writing the spawned command to stdout (along with then its actual stdout). This has previously been reported upstream in actions/toolkit#649; I've proposed actions/toolkit#1573 to fix it. This commit aims to address the issue for `setup-terraform` in the meantime by silencing `@actions/exec` and then writing out to stdout & stderr from the listener buffers, which it writes to without this additional logging. Closes hashicorp#20, hashicorp#80, hashicorp#85, hashicorp#149, hashicorp#338, and probably more.

@austinvalle

Presently using a command such as `terraform output -json | jq` does not work with the wrapper enabled, as it is by default. In order to consume terraform's output having set it up with this Action, it is necessary either to disable the wrapper (`with: terraform_wrapper: false`) or run it in its own Actions step with an explicit `id` (e.g. `id: foo`) so that it can be referred to and consumed (`${{steps.foo.outputs.stdout}}` et al.) in later steps. This seems to be the result of much confusion (issues passim) and is not at all easy (hashicorp#338) to debug/diagnose and come to the realisation that it's due to the wrapper, or even that such a thing exists. @austinvalle identified the issue as being due to the `@actions/exec` package writing the spawned command to stdout (along with then its actual stdout). This has previously been reported upstream in actions/toolkit#649; I've proposed actions/toolkit#1573 to fix it. This commit aims to address the issue for `setup-terraform` in the meantime by silencing `@actions/exec` and then writing out to stdout & stderr from the listener buffers, which it writes to without this additional logging. Closes hashicorp#20, hashicorp#80, hashicorp#85, hashicorp#149, hashicorp#338, and probably more.

@austinvalle

Presently using a command such as `terraform output -json | jq` does not work with the wrapper enabled, as it is by default. In order to consume terraform's output having set it up with this Action, it is necessary either to disable the wrapper (`with: terraform_wrapper: false`) or run it in its own Actions step with an explicit `id` (e.g. `id: foo`) so that it can be referred to and consumed (`${{steps.foo.outputs.stdout}}` et al.) in later steps. This seems to be the result of much confusion (issues passim) and is not at all easy (hashicorp#338) to debug/diagnose and come to the realisation that it's due to the wrapper, or even that such a thing exists. @austinvalle identified the issue as being due to the `@actions/exec` package writing the spawned command to stdout (along with then its actual stdout). This has previously been reported upstream in actions/toolkit#649; I've proposed actions/toolkit#1573 to fix it. This commit aims to address the issue for `setup-terraform` in the meantime by silencing `@actions/exec` and then writing out to stdout & stderr from the listener buffers, which it writes to without this additional logging. Closes hashicorp#20, hashicorp#80, hashicorp#85, hashicorp#149, hashicorp#338, and probably more.

@austinvalle

* Fix output malformed when wrapper enabled Presently using a command such as `terraform output -json | jq` does not work with the wrapper enabled, as it is by default. In order to consume terraform's output having set it up with this Action, it is necessary either to disable the wrapper (`with: terraform_wrapper: false`) or run it in its own Actions step with an explicit `id` (e.g. `id: foo`) so that it can be referred to and consumed (`${{steps.foo.outputs.stdout}}` et al.) in later steps. This seems to be the result of much confusion (issues passim) and is not at all easy (#338) to debug/diagnose and come to the realisation that it's due to the wrapper, or even that such a thing exists. @austinvalle identified the issue as being due to the `@actions/exec` package writing the spawned command to stdout (along with then its actual stdout). This has previously been reported upstream in actions/toolkit#649; I've proposed actions/toolkit#1573 to fix it. This commit aims to address the issue for `setup-terraform` in the meantime by silencing `@actions/exec` and then writing out to stdout & stderr from the listener buffers, which it writes to without this additional logging. Closes #20, #80, #85, #149, #338, and probably more. * add test for stdout with jq * update test name * remove debug lines and add changelog * add additional note about the bug fix to wrapper --------- Co-authored-by: Austin Valle <[email protected]>

matejdro · 2024-12-20T06:54:13Z

getExecOutput workaround is not suitable in many cases because it stores entire output of a command in memory, so it cannot be used for piping large things.

Swatinem added the bug Something isn't working label Nov 25, 2020

fearphage added a commit to glg-public/setup-ssh-agent that referenced this issue Aug 11, 2021

work around github library bug

75ea411

actions/toolkit#649

OJFord linked a pull request Oct 26, 2023 that will close this issue

Fix tool output mangled with spawned cmd string #1573

Open

OJFord mentioned this issue Oct 26, 2023

Fix output malformed when wrapper enabled hashicorp/setup-terraform#367

Merged

mcalhoun mentioned this issue Jan 4, 2024

fix bug with wrapper and stdout cloudposse/github-action-setup-atmos#27

Merged

austinvalle mentioned this issue Apr 25, 2024

wrapper: Write stdout/stderr data to stream when received hashicorp/setup-terraform#410

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`[command]…` output is prepended to exec `outStream` #649

`[command]…` output is prepended to exec `outStream` #649

Swatinem commented Nov 25, 2020

fearphage commented Aug 12, 2021

OJFord commented Oct 26, 2023

matejdro commented Dec 20, 2024

[command]… output is prepended to exec outStream #649

[command]… output is prepended to exec outStream #649

Comments

Swatinem commented Nov 25, 2020

fearphage commented Aug 12, 2021

OJFord commented Oct 26, 2023

matejdro commented Dec 20, 2024

`[command]…` output is prepended to exec `outStream` #649

`[command]…` output is prepended to exec `outStream` #649