Skip to content

Commit

Permalink
NE-1325: External DNS Operator support for Shared VPCs
Browse files Browse the repository at this point in the history 
Update the aws-cross-account-dns-zone.md enhancement to include
details on how we are updating the External DNS Operator to support
cross account DNS record creation in AWS shard VPCs.
  • Loading branch information
@gcs278
gcs278 committed Jul 19, 2023
1 parent 15cfd89 commit f1b3af9
Showing 1 changed file with 97 additions and 8 deletions.
105 changes: 97 additions & 8 deletions enhancements/installer/aws-cross-account-dns-zone.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ api-approvers:
- "@deads2k"
- "@JoelSpeed"
creation-date: 2023-05-08
last-updated: 2023-05-08
last-updated: 2023-07-17
tracking-link: # link to the tracking ticket (for example: Jira Feature or Epic ticket) that corresponds to this enhancement
- https://issues.redhat.com/browse/CORS-2613
- https://issues.redhat.com/browse/OCPBU-558
Expand Down Expand Up @@ -139,8 +139,6 @@ type AWSDNSSpec struct {

```

The name of the field is discussed further in open questions.

#### Install Config

`platform.aws.hostedZoneRole` would be added to complement the existing `platform.aws.hostedZone` field.
Expand All @@ -162,9 +160,55 @@ platform:
hostedZone: Z00147933I3NWOQ6M4699
hostedZoneRole: arn:aws:iam::<account-a>:role/<role-name>
```
Please see Open Questions for further discussion of the install config.
### API: ExternalDNS Operator
[ExternalDNS Operator](https://github.com/openshift/external-dns-operator) allows you to deploy and manage [ExternalDNS](https://github.com/kubernetes-sigs/external-dns),
a cluster-internal component which makes Kubernetes resources discoverable through public DNS servers. This enhancement
extends the ExternalDNS Operator to support cross-account DNS zones in AWS. We extend the`ExternalDNS` API object by adding
the field `RoleARN` to `ExternalDNSAWSProviderOptions` struct (`spec.provider.aws`).
Refer to [`externaldns_types.go`](https://github.com/openshift/external-dns-operator/blob/main/api/v1beta1/externaldns_types.go)
for the existing API structure of the `ExternalDNS` object.

In addition to adding `RoleARN`, the `Credentials` field was updated to be optional. Prior to this change, the `Credentials`
field was required if `spec.provider.aws` was specified. However, OpenShift clusters are exempt from explicitly
providing `Credentials` since the ExternalDNS Operator generates a `CredentialRequest` automatically. Consequently, to
enable OpenShift clusters to specify `RoleARN` without `Credentials`, it is necessary to make the `Credentials` field
optional. The validation webhook takes care of guaranteeing that non-OpenShift clusters include credentials through the
API.

The ExternalDNS Operator's validation webhook will use the existing [IsARN](https://pkg.go.dev/github.com/aws/aws-sdk-go/service/s3/internal/arn#IsARN)
function to validate the `RoleARN` field. The approach mentioned above differs from the API design documented in the
[API: DNS](#API-DNS) section, which employs CRD validation utilizing a regular expression. Using the existing `IsARN`
function provides more flexibility and reliability since it is maintained by AWS themselves, which helps to mitigate
potential issues with edge cases.

```go
type ExternalDNSAWSProviderOptions struct {
// Credentials is a reference to a secret containing
// the following keys (with corresponding values):
//
// * aws_access_key_id
// * aws_secret_access_key
//
// See
// https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md
// for more information.
//
// +kubebuilder:validation:Optional
// +optional
Credentials SecretReference `json:"credentials"`

// RoleARN contains the ARN of a IAM role that will be assumed when using the AWS API.
// It provides the ability to use a hosted zone in another AWS account.
//
// +kubebuilder:validation:Optional
// +optional
RoleARN *string `json:"roleARN,omitempty"`
}
```

See [ExternalDNS Operator Details](#External-DNS-Operator-Details) for more implementation details.

### Implementation Details/Notes/Constraints [optional]

Expand Down Expand Up @@ -218,9 +262,30 @@ An example IAM Trust Policy:

```

#### ExternalDNS Operator Details

Prior to this enhancement, the [ExternalDNS Operator API](https://github.com/openshift/external-dns-operator/blob/main/api)
did not allow users to configure an AWS role ARN field. As a result, there wasn't a supported way to use the External
DNS Operator to create DNS records in another AWS account within a shared VPC.

The ExternalDNS Operator configures and runs the [ExternalDNS](https://github.com/openshift/external-dns) binary.
Conveniently, ExternalDNS already supports the [`--aws-assume-role`](https://github.com/openshift/external-dns/blob/fe00b4b83c2263282a9068655e8e3fbbc167b653/docs/faq.md#can-external-dns-manageaddremove-records-in-a-hosted-zone-which-is-setup-in-different-aws-account)
argument, which uses the specified AWS role ARN when creating new DNS records. Therefore, to support shared VPCs with
the ExternalDNS Operator, we update the [API](#API-External-DNS-Operator) to expose the `--aws-assume-role` argument
for the ExternalDNS binary.

By design, ExternalDNS can update both public and private DNS zones using credentials that the cluster-admin provides.
Therefore, ExternalDNS Operator users are able to use the new [API](#API-External-DNS-Operator) to update both public
and private DNS zones in other accounts, provided the role ARN has appropriate permissions.

More details on using a role ARN to create DNS records in another AWS account will be documented in ExternalDNS
Operator's [Docs](https://github.com/openshift/external-dns-operator/tree/main/docs).

### Risks and Mitigations

Add-on operators needing permissions will not work unless updated to use the role ARN.
Add-on operators needing permissions will not work unless updated to use the role ARN. The following add-on operators
have been addressed:
- ExternalDNS Operator

### Drawbacks

Expand All @@ -235,14 +300,23 @@ platform dependencies in this config, but it should be carefully considered.

An e2e test will be setup which utilizes both AWS CI accounts.

For the ExternalDNS Operator, we will add e2e tests utilizing both AWS CI accounts using the RoleARN to create DNS
records.

### Graduation Criteria

This functionality is targeted for 4.14 GA and for backporting to previous releases.
This functionality is targeted for 4.14 GA and for backporting to previous releases.

Note that the ExternalDNS Operator updates to support Shared VPC are out of payload and are therefore not aligned with
OpenShift releases and will not need backporting.

#### Dev Preview -> Tech Preview

Behind `PrivateHostedZoneAWS` feature gate.

Updates to the ExternalDNS Operator will be made available without being restricted by a feature gate. These updates
will be considered GA at the time of merging the feature.

#### Tech Preview -> GA

- E2E CI coverage with satisfactory quality signal is needed to lift feature gate.
Expand All @@ -258,7 +332,12 @@ TODO: there should be minimal impact here.

### Version Skew Strategy

n/a
There are potential version skew scenarios for the ExternalDNS Operator support of Shared VPC. The updates to the
ExternalDNS Operator are decoupled from the standard release schedule of OpenShift Container Platform. Consequently, in
a Shared VPC cluster using cross-account DNS record creation with the Ingress Operator, there is a possibility
of installing a version of the ExternalDNS Operator that lacks support for this same feature. However, it is important
to note that the ExternalDNS Operator and the installer or Ingress Operator do not rely on each other's logic.
Hence, this mismatch does not pose a concern.

### Operational Aspects of API Extensions

Expand All @@ -267,7 +346,7 @@ n/a
#### Failure Modes

- Installation will fail if there are any issues with role during install.
- If there are issues with the role during day-2 operations, the cluster ingress operator will not
- If there are issues with the role during day-2 operations, the Cluster Ingress Operator will not
be able to create DNS records and should log appropriate error messages.

#### Support Procedures
Expand All @@ -286,6 +365,16 @@ profile directly in the AWS creds. The implementation suggested in this enhancem
preferred as it allows users to declare intent of a cross account install,
rather than OpenShift trying to infer that from the credentials.

### External ID

We considered supporting configuration of an [external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html)
for both the Ingress Operator and the ExternalDNS Operator when using an assumed role. ExternalDNS provides support for
an external ID via the [`--aws-assume-role-external-id`](https://github.com/kubernetes-sigs/external-dns/blob/22da9f231dbc6faa3a668b507a4a06823a129609/pkg/apis/externaldns/types.go#L460)
command line argument. AWS suggests the use of an external ID in specific scenarios to mitigate the confused deputy
problem. However, for this effort, we opted not to include this functionality at the moment. The reason behind this
decision is that incorporating support for external ID throughout all the components that enable Shared VPC (including
API, Install, and Ingress Operator) would be necessary, but there hasn't been a specific customer need for it thus far.

### API: Infrastructure

A `privateHostedZoneRole` field containing the role ARN would be added to `AWSPlatformStatus` in the [cluster infrastructure object](https://github.com/openshift/api/blob/894b49f57a15cbce3869961e20cd9d52df6f8b0f/config/v1/types_infrastructure.go#L424).
Expand Down

0 comments on commit f1b3af9

Please sign in to comment.