Cannot start session from sdk with version 1.2.688.0

[yongzhang]

Hi, team

We're using golang sdk to start ssm session, but it is broken after session-manager-plugin upgraded to 1.2.688.0, I got error:

2024-11-13 18:41:45 ERROR [getV4SignatureHeader @ websocketchannel.go.138] Failed to sign websocket, NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2024-11-13 18:41:45 ERROR [Open @ websocketchannel.go.199] Failed to get the v4 signature, NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

verbose message:

2024-11-14 17:54:50 ERROR [getV4SignatureHeader @ websocketchannel.go.141] Failed to sign websocket, NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Seems like it is caused by this change.

And I'm afraid this line broken it:

session-manager-plugin/src/sessionmanagerplugin/session/sessionhandler.go

Line 41 in 80869bb

sess, err := sdkutil.GetNewSessionWithEndpoint(s.Endpoint)

I do not reply on local config or env to get credentials in my code, in my case, I created aws config like this with aws-go-sdk-v2:

	cfg, err := config.LoadDefaultConfig(ctx,
		config.WithRegion(resp.GetRegion()),
		config.WithCredentialsProvider(credentials.StaticCredentialsProvider{
			Value: aws.Credentials{
				AccessKeyID:     resp.GetAccessKeyId(),
				SecretAccessKey: resp.GetSecretAccessKey(),
				SessionToken:    resp.GetSessionToken(),
			},
		}),
	)

...
StartSession(ssm.NewFromConfig(cfg) ...

[hatchetaustralia]

Not sure if directly related to the same issue as you, but we saw a breaking change for us between 1.2.677.0 and 1.2.688.0. Specifically, when establishing an SSH connection via the data channel over SSM, the handshake would fail using the latest plugin. Logs from the target machines running SSM pointed us to the handshake failure.

Rolling back to the previous version 1.2.677.0 resolved the issue.

2024-11-15 03:58:11.3022 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Setting up datachannel for session: botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy, requestId: c14233a5-2e23-42b1-b09b-7182694300c5, clientId:
2024-11-15 03:58:11.3609 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Opening websocket connection to: wss://ssmmessages.ap-southeast-2.amazonaws.com/v1/data-channel/botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy?role=publish_subscribe
2024-11-15 03:58:11.4023 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Successfully opened websocket connection to: 99.83.82.120:443
2024-11-15 03:58:11.4024 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Starting websocket pinger
2024-11-15 03:58:11.4025 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Initiating Handshake
2024-11-15 03:58:11.4026 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Starting websocket listener
2024-11-15 03:58:26.4037 ERROR [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Encountered error while initiating handshake. Handshake timed out. Please ensure that you have the latest version of the session manager plugin.
2024-11-15 03:58:28.4041 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Closing datachannel with channel Id botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy
2024-11-15 03:58:28.4084 INFO [ssm-agent-worker] [MessageService] [EngineProcessor] [BasicExecuter] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] requested terminate messaging worker, destroying the channel
2024-11-15 03:58:28.4041 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Closing websocket channel connection to: wss://ssmmessages.ap-southeast-2.amazonaws.com/v1/data-channel/botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy?role=publish_subscribe
2024-11-15 03:58:28.4043 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Ending websocket pinger
2024-11-15 03:58:28.4044 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Ending the channel listening routine since the channel is closed
2024-11-15 03:58:28.4044 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Ending websocket listener
2024-11-15 03:58:28.4044 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Successfully closed websocket connection to: 99.83.82.120:443

[Yangtao-Hua]

Hi @yongzhang @hatchetaustralia,

Thanks for raising this issue. We are actively researching this issue and will provide an update as soon as we have more findings.

Thanks!
Yangtao

[Yangtao-Hua]

Hi,
We noticed a potential timeout issue when credentials are missing locally. Please try the latest version 1.2.694.0, which has just been released. Thank you!

Thanks,
Yangtao

[hatchetaustralia]

1.2.694.0 has fixed the issue for me.

[et304383]

Hi, We noticed a potential timeout issue when credentials are missing locally. Please try the latest version 1.2.694.0, which has just been released. Thank you!

Thanks, Yangtao

Please update brew asap. The old version is still the latest there and it has this problem.

[Yangtao-Hua]

Hi @et304383,
Thanks for calling out it! There is some delay on public document update, after that brew will check and upgrade it to latest version. Document will be updated by the end of today.

Thanks,
Yangtao

[LucioMS]

The public document update has been updated with the latest version.