Panic / segfault (SIGSEGV) during an active EC2 ssh session (with aws ssm & SSO)

[cristconst]

I am using ssh to connect to EC2 instances; the ssh configuration uses:

• ec2-instance-connect to send the public ssh key to the host;
• ssm start-session to actually start the ssh session
  Authentication for the aws cli is done using SSO (provided by Azure and configured using aws configure sso).
  Session is sometimes cut-off by an go application panic (segfault), I removed the token value and the email address domain name:

panic: runtime error: invalid memory address or nil pointer dereference
     [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x865016]

                                                                           goroutine 67 [running]:
                                                                                                  github.com/aws/aws-sdk-go/aws/credentials/ssocreds.(*SSOTokenProvider).refreshToken(0xc000912100, {{{0xc000360000, 0xe8}, 0xc00000f0c8, {0xc0003600f0, 0xe9}, {0xc00002cc00, 0x22}, {0xc0000e2900, 0x808}}, ...})
                                                                                                /local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/aws/credentials/ssocreds/token_provider.go:115 +0x276
                           github.com/aws/aws-sdk-go/aws/credentials/ssocreds.(*SSOTokenProvider).RetrieveBearerToken(0xc000912100, {0x0?, 0x0?})
                                        /local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/aws/credentials/ssocreds/token_provider.go:86 +0x185
                                                                            github.com/aws/aws-sdk-go/aws/credentials/ssocreds.(*Provider).RetrieveWithContext(0xc000035180, {0xc52390, 0xc00048c640})
                                                                                                /local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/aws/credentials/ssocreds/provider.go:107 +0x3b3
                     github.com/aws/aws-sdk-go/aws/credentials.(*Credentials).singleRetrieve(0xc000035200, {0xc52390, 0xc00048c640})
                                /local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/aws/credentials/credentials.go:277 +0x373
                                                         github.com/aws/aws-sdk-go/aws/credentials.(*Credentials).GetWithContext.func1()
                                /local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/aws/credentials/credentials.go:255 +0x79
                                                        github.com/aws/aws-sdk-go/internal/sync/singleflight.(*Group).doCall(0xc000035200, 0xc000091ec0, {0x0, 0x0}, 0xc0000636b8?)
                                                                                /local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/internal/sync/singleflight/singleflight.go:97 +0x3b
         created by github.com/aws/aws-sdk-go/internal/sync/singleflight.(*Group).DoChan
                                                                                                /local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/internal/sync/singleflight/singleflight.go:90 +0x2eb

                          Command '['session-manager-plugin', '{"SessionId": "cristian.constantin@xxx", "TokenValue": "", "StreamUrl": "wss://ssmmessages.eu-west-1.amazonaws.com/v1/data-channel/cristian.constantin@xxx?role=publish_subscribe&cell-number=AAEAAUABAFXX0PIRv3PwHtPxpAXGlFSRWre8nKY5BYk8vVN5AAAAAGXKAGXPKw+Kv1XCNdgOiygRzoviEWWL0J5BHut0X0Wc+Wel9Q==", "ResponseMetadata": {"RequestId": "2b7a6e63-9fc3-4733-a5eb-53154bb696c7", "HTTPStatusCode": 200, "HTTPHeaders": {"server": "Server", "date": "Mon, 12 Feb 2024 11:26:29 GMT", "content-type": "application/x-amz-json-1.1", "content-length": "1061", "connection": "keep-alive", "x-amzn-requestid": "2b7a6e63-9fc3-4733-a5eb-53154bb696c7"}, "RetryAttempts": 0}}', 'eu-west-1', 'StartSession', '', '{"Target": "", "DocumentName": "AWS-StartSSHSession", "Parameters": {"portNumber": ["22"]}}', 'https://ssm.eu-west-1.amazonaws.com']' returned non-zero exit status 2.
                                               client_loop: send disconnect: Broken pipe

Session manager plugin version:

cat /usr/local/sessionmanagerplugin/VERSION
1.2.497.0

OS:

cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"

[yuting-fan]

Did you see this error when trying to start a session, or during a successfully established session?

If it is when starting a session, you would need to refer to SSO troubleshooting guide to ensure the credentials are valid.

If it is the latter, I'd recommend you to open up a support case, where you can share your session id and logs from your Session Manager plugin with us through the private case. Here's the documentation of how to acquire the logs https://docs.aws.amazon.com/systems-manager/latest/userguide/install-plugin-configure-logs.html.