- AWS Account Operator
The operator is responsible for creating and maintaining a pool of AWS accounts and assigning accounts to AccountClaims. The operator creates the account in AWS, does initial setup and configuration of the those accounts, creates IAM resources and expose credentials for a IAM user with enough permissions to provision an OpenShift 4.x cluster.
The operator is deployed to an OpenShift cluster in the aws-account-operator namespace.
The operator requires a secret named aws-account-operator-credentials in the aws-account-operator namespace, containing credentials to the AWS payer account you wish to create accounts in. The secret should contain credentials for an IAM user in the payer account with the data fields aws_access_key_id and aws_secret_access_key. The user should have the following IAM permissions:
Permissions to allow the user to assume the OrganizationAccountAccessRole role in any account created:
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::*:role/OrganizationAccountAccessRole"
}
}
Permissions to allow the user to interact with the support center:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"support:*"
],
"Resource": "*"
}
]
}
First, an AccountPool must be created to specify the number of desired accounts to be ready. The operator then goes and creates that number of accounts. When a Hive cluster has a new cluster request, an AccountClaim is created with the name equal to the desired name of the cluster in a unique workspace. The operator links the AccountClaim to an Account in the pool, and creates the required k8s secrets, placing them in the AccountClaim's unique namespace. The AccountPool is then filled up again by the operator. Hive then uses the secrets to create the AWS resources for the new cluster.
For more information on how this process is done, please refer to the controllers section.
The below commands can be used to test payer account credentials where we create new accounts inside the payer accounts organization. Once the account is created in the first step we wait until the account is created with step 2 and retrieve its account ID. Using the account ID we can then test our IAM user has sts:AssumeRole permissions to Assume the OrganizationAccountAccessRole in the new account. The OrganizationAccountAccessRole is created automatically when a new account is created under the organization.
aws organizations create-account --email "[email protected]" --account-name "username-cli-test" --profile=orgtestaws organizations list-accounts --profile=orgtest | jq '.[][] | select(.Name=="username-cli-test")'aws sts assume-role --role-arn arn:aws:iam::<ID>:role/OrganizationAccountAccessRole --role-session-name username-cli-test --profile=orgtest
The operator can be installed locally on a Minishift or Code-Ready-Containers (CRC) cluster (or a private OpenShift cluster). On a new local cluster, running the Makefile deploy target (passing the Access Key ID and Secret Access Key) will install the operator, and create all the necessary OpenShift CRDs and secrets for it to work locally.
You must be logged into the cluster as an administrator, or otherwise have permissions to create namespaces and deploy CRDs. For Minishift, this can be done:
oc login -u system:admin
OPERATOR_ACCESS_KEY_ID="YOUR_ACCESS_KEY_ID" OPERATOR_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY" make deployThis only needs to be done once for an individual local cluster.
When developing locally using operator-sdk you may not have a metrics service running and other stages of the code may not want to be run. We have introduced the ability to set the environment variable FORCE_DEV_MODE to account for these edge cases. Set the dev mode to local when running locally.
ex: FORCE_DEV_MODE=local operator-sdk up local --namespace=aws-account-operator
The AccountPool CR holds the information about the available number of accounts that can be claimed for cluster provisioning.
apiVersion: aws.managed.openshift.io/v1alpha1
kind: AccountPool
metadata:
name: example-accountpool
namespace: aws-account-operator
spec:
poolSize: 50The Account CR holds the details about the AWS account that was created, where the account is in the process of becoming ready, and whether its linked to an AccountClaime, i.e. claimed.
apiVersion: aws.managed.openshift.io/v1alpha1
kind: Account
metadata:
name: osd-{accountName}
namespace: aws-account-operator
spec:
awsAccountID: "0000000000"
claimLink: example-link
iamUserSecret: osd-{accountName}-secretThe AccountClaim CR links to an available account and stores the name of the associated secret with AWS credentials for that account.
apiVersion: aws.managed.openshift.io/v1alpha1
kind: AccountClaim
metadata:
name: example-link
namespace: {NameSpace cluster is being built in}
spec:
accountLink: osd-{accountName} (From AccountClaim)
aws:
regions:
- name: us-east-1
awsCredentialSecret:
name: aws
namespace: {NameSpace cluster is being built in}
legalEntity:
id: 00000000000000
name: {Legal Entity Name}The AWSFederatedRole CR contains a definition of a desired AWS Role, with both managed and custom Policies included
apiVersion: aws.managed.openshift.io/v1alpha1
kind: AWSFederatedRole
metadata:
name: example-role
namespace: aws-account-operator
spec:
roleDisplayName: Example Role
roleDescription: This is an example Role
# Custom Policy definition
awsCustomPolicy:
name: ExampleCustomPolicy
description: Description of Example Custom Policy
# list of statements for the policy
awsStatements:
- effect: Allow
action:
- "aws-portal:ViewAccount"
- "aws-portal:ViewBilling"
resource:
- "*"
# list of AWS managed
awsManagedPolicies:
- "AWSAccountUsageReportAccess"
- "AmazonEC2ReadOnlyAccess"
- "AmazonS3ReadOnlyAccess"
- "IAMReadOnlyAccess"The AWSFederatedAccountAccess CR creates an instance of an AWSFederatedRole in AWS and allows the target IAM account to assume it
apiVersion: aws.managed.openshift.io/v1alpha1
kind: AWSFederatedAccountAccess
metadata:
name: example-account-access
namespace: aws-account-operator
spec:
awsCustomerCredentialSecret:
name: {Name for secret with osdManagedAdmin credentials}
namespace: {Namespace for the secret with osdManagedAdmin credentials}
externalCustomerAWSIAMARN: arn:aws:iam::${EXTERNAL_AWS_ACCOUNT_ID}:user/${EXTERNAL_AWS_IAM_USER}
awsFederatedRole:
name: {Name of desired AWSFederatedRole}
namespace: aws-account-operator The accountpool-controller is triggered by a create or change to an accountpool CR or an account CR. It is responsible for filling the Acccount Pool by generating new account CRs.
It looks at the accountpool CR spec.poolSize and it ensures that the number of unclaimed accounts matches the number of the poolsize. If the number of unclaimed accounts is less then the poolsize it creates a new account CR for the account-controller to process.
emailID = "osd-creds-mgmt"Updates accountPool CR
claimedAccounts: 4
poolSize: 3
unclaimedAccounts: 3claimedAccounts are any accounts with the status.Claimed=true
unclaimedAccounts are any accounts with status.Claimed=false and status.State!="Failed".
poolSize is the poolsize from the accountPool spec
Updated in the accountPool-controller
MetricTotalAccountCRs
MetricTotalAccountCRsUnclaimed
MetricTotalAccountCRsClaimed
MetricTotalAccountPendingVerification
MetricTotalAccountCRsFailed
MetricTotalAccountCRsReady
MetricTotalAccountClaimCRsThe account-controller is triggered by creating or changing an account CR. It is responsible for following behaviors:
If the awsLimit set in the constants is not exceeded:
- Creates a new account in the organization belonging to credentials in secret
aws-account-operator-credentials - Configures two AWS IAM users from iamUserNameUHC and iamUserNameSRE as their respective usernames
- Creates IAM user in new account
- Attaches Admin policy
- Generates a secret access key for the user
- Stores user secret in a AWS secret
- Creates STS CLI tokens
- Creates Federated webconsole URL using the iamUserNameSRE user
- Creates and Destroys EC2 instances
- Creates aws support case to increase account limits
Note: iamUserNameUHC is used by Hive to provision clusters iamUserNameSRE is used to generate Federated console URL
- If
status.RotateCredentials == truethe account-controller will refresh the STS Cli Credentials - If the account's
status.State == "Creating"and the account is older then the createPendTime constant the account will be put into afailedstate - If the account's
status.State == AccountReady && spec.ClaimLink != ""it setsstatus.Claimed = true
awsLimit = 2000
awsCredsUserName = "aws_user_name"
awsCredsSecretIDKey = "aws_access_key_id"
awsCredsSecretAccessKey = "aws_secret_access_key"
iamUserNameUHC = "osdManagedAdmin"
iamUserNameSRE = "osdManagedAdminSRE"
awsSecretName = "aws-account-operator-credentials"
awsAMI = "ami-000db10762d0c4c05"
awsInstanceType = "t2.micro"
createPendTime = 10 * time.Minute
// Fields used to create/monitor AWS case
caseCategoryCode = "other-account-issues"
caseServiceCode = "customer-account"
caseIssueType = "customer-service"
caseSeverity = "high"
caseDesiredInstanceLimit = 25
caseStatusResolved = "resolved"
intervalAfterCaseCreationSecs = 30
intervalBetweenChecksSecs = 30
// AccountPending indicates an account is pending
AccountPending = "Pending"
// AccountCreating indicates an account is being created
AccountCreating = "Creating"
// AccountFailed indicates account creation has failed
AccountFailed = "Failed"
// AccountReady indicates account creation is ready
AccountReady = "Ready"
// AccountPendingVerification indicates verification (of AWS limits and Enterprise Support) is pending
AccountPendingVerification = "PendingVerification"
// IAM Role name for IAM user creating resources in account
accountOperatorIAMRole = "OrganizationAccountAccessRole"
var awsAccountID string
var desiredInstanceType = "m5.xlarge"
var coveredRegions = []string{
"us-east-1",
"us-east-2",
"us-west-1",
"us-west-2",
"ca-central-1",
"eu-central-1",
"eu-west-1",
"eu-west-2",
"eu-west-3",
"ap-northeast-1",
"ap-northeast-2",
"ap-south-1",
"ap-southeast-1",
"ap-southeast-2",
"sa-east-1",
}Updates the Account CR
spec:
awsAccountID: "000000112120"
claimLink: "claim-name"
iamUserSecret: accountName-secretawsAccountID is updated with the account ID of the aws account that is created by the account controller
claimLink holds the name of the accountClaim that has claimed this account CR
iamUserSecret holds the name of the secret containing IAM user credentials for the AWS account
Updates the Account CR
status:
claimed: false
conditions:
- lastProbeTime: 2019-07-18T22:04:38Z
lastTransitioNTime: 2019-07-18T22:04:38Z
message: Attempting to create account
reason: Creating
status: "True"
type: Creating
rotateCredentials: false
state: Failed
supportCaseID: "00000000"state can be any f the account states defined in the constants below:
- AccountPending indicates an account is pending
- AccountCreating indicates an account is being created
- AccountFailed indicates account creation has failed
- AccountReady indicates account creation is ready
- AccountPendingVerification indicates verification (of AWS limits and Enterprise Support) is pending
claimed is true if currentAcctInstance.Status.State == AccountReady && currentAcctInstance.Spec.ClaimLink != "
rotateCredentials updated by the secretwatcher pkg which will set the bool to true triggering an reconcile of this controller to rotate the STS credentials
supportCaseID is the ID of the aws support case to increase limits
conditions indicates the last state the account had and supporting details
Update in the account-controller
MetricTotalAWSAccountsThe accountClaim-controller is triggered when an accountClaim is created in any namespace. It is responsible for following behaviours:
- Sets account
spec.ClaimLinkto the name of the accountClaim - Sets accountClaim
spec.AccountLinkto the name of an unclaimed Account - Creates a secret in the accountClaim namespace that contains the credentials tied to the aws account in the accountCR
- Sets accountClaim
status.State = "Ready"
AccountClaimed = "AccountClaimed"
AccountUnclaimed = "AccountUnclaimed"
awsCredsUserName = "aws_user_name"
awsCredsAccessKeyID = "aws_access_key_id"
awsCredsSecretAccessKey = "aws_secret_access_key"Updates the accountClaim CR
spec:
accountLink: osd-{accountName}
aws:
regions:
- name: us-east-1
awsCredentialSecret:
name: aws
namespace: {NameSpace}
legalEntity:
id: 00000000000000
name: {Legal Entity Name}awsCredentialSecret holds the name and namespace of the secret with the credentials created for the accountClaim
Updates the accountClaim CR
status:
conditions:
- lastProbeTime: 2019-07-16T13:52:02Z
lastTransitionTime: 2019-07-16T13:52:02Z
message: Attempting to claim account
reason: AccountClaimed
status: "True"
type: Unclaimed
- lastProbeTime: 2019-07-16T13:52:03Z
lastTransitionTime: 2019-07-16T13:52:03Z
message: Account claimed by osd-creds-mgmt-fhq2d2
reason: AccountClaimed
status: "True"
type: Claimed
state: Readystate can be any of the ClaimStatus strings defined in accountclaim_types.go conditions indicates the last state the account had and supporting details
Updated in the accountClaim-controller
MetricTotalAccountClaimCRsThe AWSFederatedRole-controller is triggered when an AWSFederatedRoke is created in any namespace. It is responsible for following behaviours:
- Building AWS Policy Doc from Role definition in the spec
- Attempting to validate the Role in AWS by creating the Role, and deleting it if successful
- Setting the status to Valid or Failed
- If the status is Valid or Failed, stop all reconciling
- If an AWSFederatedRole is deleted, cleaning up any instance of the Role in AWS by cleaning up any AWSFederatedAccountAccesses using the AWSFederatedRole
None
spec:
roleDisplayName: Example Role
roleDescription: This is an example Role
# Custom Policy definition
awsCustomPolicy:
name: ExampleCustomPolicy
description: Description of Example Custom Policy
# list of statements for the policy
awsStatements:
- effect: Allow
action:
- "aws-portal:ViewAccount"
- "aws-portal:ViewBilling"
resource:
- "*"
# list of AWS managed
awsManagedPolicies:
- "AWSAccountUsageReportAccess"
- "AmazonEC2ReadOnlyAccess"
- "AmazonS3ReadOnlyAccess"
- "IAMReadOnlyAccess"roleDisplayName is a human-readable name for the Role roleDescription is a human-readable description for what the Role does awsCustomPolicy is a representation of an AWS Policy to be created as part of the Role. It contains a Policy name a description, and a list of AWS Statements which Allow or Deny specific actions on specific resources. Please refer to the following documentation for more information: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html awsManagedPolicies is a list of AWS pre-defined policies to add to the Role. Please refer to the following documentation for more information: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#aws-managed-policies
conditions:
- lastProbeTime: {Time Stamp}
lastTransitionTime: {Time Stamp}
message: All managed and custom policies are validated
reason: AllPoliciesValid
status: "True"
type: Valid
state: Validconditions indicates the last states the AWSFederatedRole had and supporting details. In general for AWSFederatedRoles, only one condition is expected, and it should match the state. state is the current state of the CR. Possible values are Valid and Failed
None
The AWSFederatedAccountAccess-controller is triggered when an accountClaim is created in any namespace. It is responsible for following behaviours:
- Ensures the requested AWSFederatedRole exists
- Converts the AWSFederatedRole spec into an AWS Policy Doc
- Creates a unique AWS Role in the AWS containing the OSD cluster using the AWSFederatedRole definition
- Creates a unique AWS Policy if the AWSFederatedRole has awsCustomPolicy defined and attaches it to the Role
- Attaches any specified AWS Managed Policies to the Role
None
spec:
awsCustomerCredentialSecret:
name: {Name for secret with osdManagedAdmin credentials}
namespace: {Namespace for the secret with osdManagedAdmin credentials}
externalCustomerAWSIAMARN: arn:aws:iam::${EXTERNAL_AWS_ACCOUNT_ID}:user/${EXTERNAL_AWS_IAM_USER}
awsFederatedRole:
name: {Name of desired AWSFederatedRole}
namespace: aws-account-operator awsCustomerCredentialSecret is the secret reference for the osdManagedAdmin IAM user in the AWS account where OSD is installed externalCustomerAWSIAMARN is the AWS ARN for the desired IAM user that will use the AWS role when created. This should be in an AWS account external to the one where OSD is installed awsFederatedRole is the reference to the target AWSFederatedRole CR to create an instance of
status:
conditions:
- lastProbeTime: {Time Stamp}
lastTransitionTime: {Time Stamp}
message: Account Access Ready
reason: Ready
status: "True"
type: Ready
consoleURL: https://signin.aws.amazon.com/switchrole?account=701718415138&roleName=network-mgmt-5dhkmd
state: Readyconditions indicates the states the AWSFederatedAccountAccess had and supporting details consoleURL is a generated URL that directly allows the targeted IAM user to access the AWS Role state is the current state of the CR
None
- Starts a metric server with custom metrics defined in
localmetricspkg
metricsPort = "8080"
metricsPath = "/metrics"
secretWatcherScanInterval = time.Duration(10) * time.MinutemetricsPort is the port used to start the metrics port
metricsPath it the path used as the metrics endpoint
secretWatcherScanInterval sets the interval at which the secret watcher will look for secrets that are expiring