Change in behavior on how aws_iid uses the creds in the environment between 12.1 and 12.2

[caleygoff-invitae]

• Version: 12.1-12.2
• Platform: Linux spire-server-0 4.14.225-169.362.amzn2.x86_64 CLI authentication #1 SMP Mon Mar 22 20:14:50 UTC 2021 x86_64 Linux
• Subsystem: root-spire-server,root-spire-agent,nested-spire-server,nested-spire-agent

Basically I have observed that in 12.1 when using the aws_iid plugin to attest my root-spire-agent with my root-spire-server I could get the root-spire-server to spit out it was using my OIDC assigned Pod IAM role that we attach to our individual workloads.

Whereas, I observed that in 12.2 I could get it to fail in the exact same spot as 12.1 but have it spit out that it was using the InstanceProfile that is attached to the ec2 nodes in my node.group.

This is problematic for our shop because we actually require that our IAM permissions be attached to our workloads and not the instanceprofile that is attached to the running instances in that node.group. If we attached permissions to the InstanceProfile such as DescribeInstance or GetInstanceProfile then any workload running in the node.group with the InstanceProfile that we have attached to it would be able to do that aws query, which is problematic for us.

Steps to Reproduce
12.1
First ensure that we are using pod level IAM permissions for both of these tests. You can do this now with AWS directly via their OIDC connect here.
We are actually using KIAM to do our OIDC federation and how to set that is here.
They both achieve the same thing, pod level IAM policies.
Then set up a spire-server and ensure that is 12.1 and make sure the above IAM policy is attached to the pod correctly.
Then set up a spire-agent that needs to use aws_iid to attest to a spire-server, IAM permissions aren't needed really on the spire-agent.
Then on the IAM permission that is attached to the server, even though aws_iid requires both ec2:DescribeInstances and iam:GetInstanceProfile , to get the code to spit the error we want, just make sure the role can run ec2:DescribeInstances. Leave off iam:GetInstanceProfile intentionally.
Also make sure there is a trust relationship and sts:AssumeRole is allowed on this ARN in which the workload is running (our cluster node.group arn.

Restart the spire-server with the updated IAM policy attached to it, and watch the logs. You should see something like

time="2021-05-06T20:37:56Z" level=error msg="Failed to attest" caller-addr="XX.XX.XX.XXX:27052" error="rpc error: code = Unknown desc = aws-iid: AccessDenied: User: arn:aws:sts::XXXXXXXXXXXX:assumed:assumed-role/spire-controller-pod.utils.stg.some.corp.net/kiam-kiam is not authorized to perform: iam:GetInstanceProfile on resource: instance profile eks-129bd4d6-aeab-11eb-91a8-7bdf79d66bc3\n\tstatus code: 403, request id: d81e98e5-2ada-484d-a49c-3f51da8e8d36" method=AttestAgent node_attestor_type=aws_iid service=agent.v1.Agent subsystem_name=api

note my user here User: arn:aws:sts::XXXXXXXXXXXX:assumed:assumed-role/spire-controller-pod.utils.stg.some.corp.net/kiam-kiam

12.2
Then set up a spire-server and ensure that is 12.2 and make sure the above IAM policy is attached to the pod correctly you can use the same one from the last steps.
Then set up a spire-agent that needs to use aws_iid to attest to a spire-server, IAM permissions aren't needed really on the spire-agent.
Make sure on the IAM permission that is attached to the server, that even though aws_iid requires both ec2:DescribeInstances and iam:GetInstanceProfile , to get the code to spit the error we want, just make sure the role can run ec2:DescribeInstances. Leave off iam:GetInstanceProfile intentionally.

Restart the spire-server with the IAM policy attached to it, and watch the logs. You should see something like
time="2021-05-06T19:55:05Z" level=error msg="Failed to attest" caller-addr="XX.XX.XX.XXX:16162" error="rpc error: code = Unknown desc = aws-iid: AccessDenied: User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/eks.workers.utils.stg.some.corp.net/i-03bf49d1171ebd129 is not authorized to perform: iam:GetInstanceProfile on resource: instance profile eks-129bd4d6-aeab-11eb-91a8-7bdf79d66bc3\n\tstatus code: 403, request id: d7af6bad-3618-401b-842c-dd3299873cd1" method=AttestAgent node_attestor_type=aws_iid service=agent.v1.Agent subsystem_name=api

note my user here User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/eks.workers.utils.stg.some.corp.net/i-03bf49d1171ebd129

Notice how in 12.2 it uses the InstanceProfile first, whereas in 12.1 it appears that its "failing back" to my AssumeRole policy, which is ideally what I want. I mentioned in #2218 how handy it would be to have a configurable on the aws_iid plugin for something like assume_role_arn which would allow the plugin to assume the role we specify or else it could fallback on to the InstanceProfile versus the otherway around.

[rturner3]

Hi @caleygoff-invitae, we added support for the assume_role configurable in the aws_iid NodeAttestor plugin in #2387. Since it sounds like this is enough to resolve the issue you are describing, we are going to close this. Please let us know if you are still facing this issue after using this configurable.