Add connection assume_role configuration

[brittandeyoung]

Is your feature request related to a problem? Please describe.

Currently in order to configure a connection to an AWS account, you need to provide a aws config profile or the credentials in the configuration file. This can make connecting to a large number of accounts difficult to maintain the configuration.

Describe the solution you'd like
It would be ideal to be able to provide a assume_role to the connection config in place of credentials and this role would be assumed for the connection.

Describe alternatives you've considered
A way to dynamically generate the needed aws profiles and dynamically generate the connection configuration file. This is not ideal in any way.

Additional context
This type of configuration would make dynamically adjusting the connection information simple and scale well with large multi account configurations.

[p5]

This looks like it would be a great addition!
Our main difficulty is generating this .aws/config file in ECS, but having the ability to assume role directly from the connection configuration would make things so much easier!

Happy to help out to push this through if needed, although not contributed here before either. Let me know 😄

[cbruno10]

@brittandeyoung Thanks for opening the issue and submitting the PR!

I like the idea of being able to specify a role ARN for easier creds setup, but I do have some concerns around how to pass options/config when assuming the role. For instance, if you want to assume a role with an external ID or MFA serial number in the AWS CLI or the AWS plugin today, you'd setup the AWS profile like:

[profile myrole]
role_arn = arn:aws:iam::234567890123:role/SomeRole
mfa_serial = arn:aws:iam::234567890123:mfa/saanvi
external_id = 123456

If you were to use Steampipe today to authenticate to an AWS account and reference your profile in aws.spc, the AWS SDK would automatically take in this profile information and pass it along to the STS AssumeRole API call.

So if we wanted to offer the same functionality in the AWS plugin, we'd most likely need to offer additional config args, e.g., assme_role_mfa_serial = '...', or have the assume role config arg be an object and essentially re-create the AWS profile format, e.g.,

assume_role_config = {
  role_arn = '...'
  mfa_serial = '...'
}

Is your use case just for assuming a role with no other configuration/options specified, or do you have use cases for this as well?

@johnsmyth Any thoughts on how we could support a feature like this?

[p5]

@cbruno10

My thoughts on this:

If we were deploying Steampipe to ECS or similar, we would need to have:

1. The .aws/config configuration stored in a Docker image
2. AWS CLI installed (since I don't believe Steampipe directly reads .aws/config, it resolves the values through AWS CLI)

Whereas with this implementation, the roles are assumed within Steampipe itself, and does not depend on AWS CLI at all. These roles would be assumable directly from the values in the environment variables through AWS SDK.

I don't think you will need to support mfa_serial since the user would have already authenticated with their default profile - a profile that already has permissions to assume roles, be it using AWS CLI or EC2/ECS IAM roles.

Terraform AWS provider is a good example of what to support and what not to. They support assume_role and access keys, but not mfa_serial or the others you mentioned.
https://github.com/hashicorp/terraform-provider-aws/blob/c2271cf2c464d1c8b751be52448afaf83a89f7b0/internal/provider/provider.go#L37-L49

Edit:
The parts about AWS CLI seems incorrect. It is not needed.
However it is more overhead having to maintain both .aws/config and the connector configuration.

[cbruno10]

@p5 Thanks for the examples!

I don't think AWS CLI would need to be installed, as the AWS plugin uses the AWS SDK Go v2, which I don't believe relies on the AWS CLI (but I could be wrong on this), as it handles creds and API calls directly in its code.

The Terraform AWS provider provides an assume_role block, which allows users to pass in other options, e.g.,:

provider "aws" {
  assume_role {
    role_arn     = "arn:aws:iam::123456789012:role/ROLE_NAME"
    session_name = "SESSION_NAME"
    external_id  = "EXTERNAL_ID"
  }
}

So we could provide something similar in this plugin (along with assume_role_with_web_identity now or in the future).

[brittandeyoung]

@p5 covered exactly why I raised this issue. Having to both manage the .aws/config and the connection settings inside steampipe is redundant and makes it difficult to manage.

I will adjust my PR to use the assume_role block.

[brittandeyoung]

From what I can see, the steampipe plugin schema does not have a method for doing a configuration block within the schema. I will be making root configuration items for each assume role config.

[cbruno10]

@brittandeyoung We have an example in the Kubernetes plugin you can follow:

• https://github.com/turbot/steampipe-plugin-kubernetes/blob/main/config/kubernetes.spc#L63-L71
• https://github.com/turbot/steampipe-plugin-kubernetes/blob/main/kubernetes/connection_config.go#L14

In this example, we allow any top level key in the map, e.g., "my_chart_name", but for this case, we would know all of the key names.

[brittandeyoung]

@cbruno10 Thank you for the example I am currently looking to see if i can make the needed adjustments to move it to a block using your example.

[cbruno10]

@brittandeyoung Can you please also test what happens when you try to run a query past the role session expiration time when using your new config arg(s)? For instance, if you run the following steps:

• Setup aws.spc to have an AWS connection that specifies an assume role timeout of 15 mins
• Start Steampipe (steampipe service start)
• Run a query (steampipe query "select name from aws_iam_role")
• Wait 20 minutes
• Run another query (steampipe query "select name from aws_s3_bucket")

Do you get an error, does the request succeed, or is there another outcome?

I believe this scenario is already handled in the AWS plugin by the AWS SDK when pointing to a profile that has configuration details in the credentials or config file (except when a static MFA token is passed in), so it should be supported for this new config arg as well in order to allow users to use long running Steampipe processes.

[brittandeyoung]

@cbruno10 I am struggling to get the configuration working in the config block. The configuration that I currently have pushed to the PR works great with the root level configurations.

One thing I am struggling with is that the schema for the plugin you are referencing is dynamic, while the schema of this plugin is statically defined. I do not see a way in the static definition to define a block, in other projects like terraform providers this is typically done by a List of one, but I do not see that as an option in the plugin schema.

Any tips from here are appreciated.

Here is what I tried, I removed the static schema config, and set the schema to be dynamic. This alone causes the plugin to crash even without adding any additional configuration.

This is the error when attempting to run with dynamic schema:

ERROR: failed to start plugin 'hub.steampipe.io/plugins/turbot/aws@latest': timed out waiting for hub.steampipe.io/plugins/turbot/aws@latest to startup after 20 seconds (SQLSTATE HV000)

I am very new to steampipe development, so i could be making a simple rookie mistake =) .

This is the configuration structures I was testing with:

type awsConfig struct {
	Regions               []string          `hcl:"regions,optional"`
	DefaultRegion         *string           `hcl:"default_region,optional"`
	Profile               *string           `hcl:"profile,optional"`
	AssumeRole            *assumeRoleConfig `hcl:"assume_role,block"`
	AccessKey             *string           `hcl:"access_key,optional"`
	SecretKey             *string           `hcl:"secret_key,optional"`
	SessionToken          *string           `hcl:"session_token,optional"`
	MaxErrorRetryAttempts *int              `hcl:"max_error_retry_attempts,optional"`
	MinErrorRetryDelay    *int              `hcl:"min_error_retry_delay,optional"`
	IgnoreErrorCodes      []string          `hcl:"ignore_error_codes,optional"`
	EndpointUrl           *string           `hcl:"endpoint_url,optional"`
	S3ForcePathStyle      *bool             `hcl:"s3_force_path_style,optional"`
}

type assumeRoleConfig struct {
	RoleARN     *string `hcl:"role_arn"`
	Duration    *string `hcl:"duration,optional"`
	ExternalId  *string `hcl:"external_id,optional"`
	SessionName *string `hcl:"session_name,optional"`
}

[Subhajit97]

Hi @brittandeyoung, thanks for submitting the PR. The changes look great!

For the above config example, I think you need to mention the cty tags as well to provide type hints to certain keys for parsing HCL configurations.

Could you please try the below config struct?

type awsConfig struct {
	Regions               []string          `hcl:"regions,optional"`
	DefaultRegion         *string           `hcl:"default_region,optional"`
	Profile               *string           `hcl:"profile,optional"`
	AssumeRole            *assumeRoleConfig `hcl:"assume_role,optional"`
	AccessKey             *string           `hcl:"access_key,optional"`
	SecretKey             *string           `hcl:"secret_key,optional"`
	SessionToken          *string           `hcl:"session_token,optional"`
	MaxErrorRetryAttempts *int              `hcl:"max_error_retry_attempts,optional"`
	MinErrorRetryDelay    *int              `hcl:"min_error_retry_delay,optional"`
	IgnoreErrorCodes      []string          `hcl:"ignore_error_codes,optional"`
	EndpointUrl           *string           `hcl:"endpoint_url,optional"`
	S3ForcePathStyle      *bool             `hcl:"s3_force_path_style,optional"`
}

type assumeRoleConfig struct {
	RoleARN     *string `hcl:"role_arn" cty:"role_arn"`
	Duration    *string `hcl:"duration,optional" cty:"duration"`
	SessionName *string `hcl:"session_name,optional" cty:"session_name"`
}

Let us know if you need any help, thanks!

[RafPe]

@p5 Thanks for the examples!

I don't think AWS CLI would need to be installed, as the AWS plugin uses the AWS SDK Go v2, which I don't believe relies on the AWS CLI (but I could be wrong on this), as it handles creds and API calls directly in its code.

The Terraform AWS provider provides an assume_role block, which allows users to pass in other options, e.g.,:

provider "aws" {
  assume_role {
    role_arn     = "arn:aws:iam::123456789012:role/ROLE_NAME"
    session_name = "SESSION_NAME"
    external_id  = "EXTERNAL_ID"
  }
}

So we could provide something similar in this plugin (along with assume_role_with_web_identity now or in the future).

To join the party :) I think adding both the assume role & the assume role with webidentity would open all the doors not to use the CLI itself but potentially to use it CICD pipelines ( like Github Actions with OIDC )

@brittandeyoung I have been through developing the above mentioned credentials setup so you are more than welcome to take any parts for your PR https://github.com/RaftechNL/qbconf/blob/main/main.go#L294

[brittandeyoung]

I have updated the PR with a working assume role configuraiton. Below is an example config that will work for assuming roles:

connection "aws" {
  plugin  = "aws"
  assume_role = {
    role_arn = "arn:aws:iam::1111111111111:role/steampipe"
    duration = "300s"
    external_id = "testing"
    session_name = "steampipe"
  } 
  // default_region = "us-east-1"
  regions = ["us-east-1"]
}

I am still having issues with the optional config items not actually being read into the config. In this example, eexternal_id , session_name and duration. I am assuming there is something missing with my structure tags. If I add a cty tag for the optional fields, steam pipe crashes and fails to load at all. If I remove the optional tag and add a cty tag, they load, but then they MUST be provided in the config.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[p5]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

Still would love to have this available.

[bigdatasourav]

Closing the issue for now, we plan to resume work on it once the Steampipe SDK update is released.

Progress and updates will continue to be tracked at the following -
turbot/steampipe-plugin-sdk#692
turbot/steampipe-plugin-sdk#691