[session-manager-plugin] "aws ssm start-session" should print debug output to stderr instead of stdout

[heidemn]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug

ssm start-session should enable the user to pipe stdout to a file or other command.
Currently, it dumps debug output to stdout, which is not helpful.

SDK version number
Same behavior with v1 and v2.
aws-cli/1.19.12 Python/3.8.5 Linux/4.19.128-microsoft-standard botocore/1.20.12
aws-cli/2.1.27 Python/3.7.3 Linux/4.19.128-microsoft-standard exe/x86_64.ubuntu.20 prompt/off

$ session-manager-plugin --version -> 1.2.54.0

Platform/OS/Hardware/Device
Ubuntu 20.04.1 LTS (in WSL2)

To Reproduce (observed behavior)

$ aws ssm start-session --target i-0123456789abcde --document-name AWS-StartInteractiveCommand --parameters 'command=["echo 42"]' > 42.txt
(no output)

$ cat 42.txt

Starting session with SessionId: root-0001234567abcde
42


Exiting session with sessionId: root-0001234567abcde.

Expected behavior

$ aws ssm start-session --target i-0123456789abcde --document-name AWS-StartInteractiveCommand --parameters 'command=["echo 42"]' > 42.txt

Starting session with SessionId: root-0001234567abcde

Exiting session with sessionId: root-0001234567abcde.

$ cat 42.txt
42

Logs/output
n/a

Additional context
Use case: Run a command on the instance, and do some follow-up actions (local or remote) based on the output.

Unix commands generally print additional info to stderr.
Only the main output / payload should go to stdout.

I could find the text "Starting session" / "session with SessionId" neither in this repo, nor in the botocore repo.
-> I assume that this needs to be fixed in the session-manager-plugin, which seems not to be available on Github.

[heidemn]

As requested by @kdaily , I'm re-opening aws/aws-cli#5965 here.

Hi @heidemn, thanks for the report. You're right, this would need to be fixed in the session manager plugin. I will alert the SSM team to this feature request.

You can open this issue here - unfortunately I do not have the permissions to transfer this issue to the other repository.
https://github.com/aws/amazon-ssm-agent
Thanks!

Thanks in advance for fixing this.

[nitikagoyal87]

Thanks for your feedback! We have noted this request.

[rmccue]

We are also interested in this feature, as using scp over a shell requires separating stderr from stdout in the stream. As SSM merges these streams together currently, it is not possible to use SSM for scp functionality.

[heidemn]

@nitikagoyal87 looking at the code, it seems that 42milez' PR should fix the issue.
Since the change is quite small, can we get it merged soon?

[ioquatix]

We have exactly the same problem.

@MehdiZonjy anyway we can get some more attention on this?

[ioquatix]

Even just an option to switch off this logging, or to specify a log device, e.g. --log /dev/null

[tapajos]

We have exactly the same problem.

@MehdiZonjy anyway we can get some more attention on this?

[ioquatix]

@tapajos you should file an internal support request in AWS.

[ioquatix]

@gianniLesl any chance we can merge this?

[cohesion-flotz]

Bumping this as I am having the same problem :)

[ioquatix]

Any progress on this?

[chancez]

aws/session-manager-plugin#94 should help with this if anyone wants to test it out.

[vergenzt]

I've made a client-side workaround for this via a ~/.aws/cli/alias config which wraps around aws ssm start-session.

It sends the SSM header and trailer to stderr, sends program stdout to stdout and stderr to stderr, and then exits with the exit code of the program. (I also had it add color to the SSM header/trailer to make them less visually prominent.)

Note that it utilizes https://github.com/jpmens/jo to easily produce JSON from shell script, so you'll need to have that installed.

# in file: ~/.aws/cli/alias

[command ssm]


run =
  !f() {
    SSM_TARGET=${1:?Missing arg 1: SSM_TARGET}
    shift 1
    EC_FILE=$(mktemp -u)
    AWK_RED=$([ -t 2 ] && echo 'printf "\033[0;31m" > "/dev/stderr";')
    AWK_BLK=$([ -t 2 ] && echo 'printf "\033[0m" > "/dev/stderr";')
    aws ssm start-session \
      --target="$SSM_TARGET" \
      --document-name="AWS-StartNonInteractiveCommand" \
      --parameters="$(jo \
        command="$(jo -aB -- "$(bash -c 'echo -n "${*@Q}"' -- "$@")")" \
        separateOutputStream="$(jo -a -- -s true)" \
        stdoutSeparatorPrefix="$(jo -a -- "STDOUT:"$'\n')" \
        stderrSeparatorPrefix="$(jo -a -- "STDERR:"$'\n')" \
      )" \
    | awk '
      BEGIN         { '"$AWK_RED"' dest = "/dev/stderr" }
      /^STDERR:$/   { '"$AWK_BLK"' dest = "/dev/stderr"; next }
      /^STDOUT:$/   { '"$AWK_BLK"' dest = "/dev/stdout"; next }
      /^EXIT_CODE:/ { '"$AWK_RED"' dest = "/dev/stderr"; printf $2 > "'"$EC_FILE"'"; next }
      { print $0 > dest }
      END { '"$AWK_BLK"' }
    '
    return "$(cat $EC_FILE)"
  }; f

Invoke it with aws ssm run <target> <command> <args...>.

[heidemn]

I've made a client-side workaround for this via a ~/.aws/cli/alias config which wraps around aws ssm start-session.

On the one side, nice workaround.
On the other side... 😄 https://miro.medium.com/v2/resize:fit:1200/1*Z5ZPEyfqyzRZPNbBI_Rv7Q.jpeg

Would be nice if AWS fixes this properly.

[bbhoss]

This is a very frustrating limitation, as it makes it impossible to cleanly capture the output of running anything in the containers.