Skip to content

Commit

Permalink
NE-1325: External DNS Operator support for Shared VPCs
Browse files Browse the repository at this point in the history 
Update the aws-cross-account-dns-zone.md enhancement to include
details on how we are updating the External DNS Operator to support
cross account DNS record creation in AWS shard VPCs.
  • Loading branch information
@gcs278
gcs278 committed Jul 17, 2023
1 parent 15cfd89 commit 4cba6b6
Showing 1 changed file with 94 additions and 8 deletions.
102 changes: 94 additions & 8 deletions enhancements/installer/aws-cross-account-dns-zone.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ api-approvers:
- "@deads2k"
- "@JoelSpeed"
creation-date: 2023-05-08
last-updated: 2023-05-08
last-updated: 2023-07-17
tracking-link: # link to the tracking ticket (for example: Jira Feature or Epic ticket) that corresponds to this enhancement
- https://issues.redhat.com/browse/CORS-2613
- https://issues.redhat.com/browse/OCPBU-558
Expand Down Expand Up @@ -139,8 +139,6 @@ type AWSDNSSpec struct {

```

The name of the field is discussed further in open questions.

#### Install Config

`platform.aws.hostedZoneRole` would be added to complement the existing `platform.aws.hostedZone` field.
Expand All @@ -162,9 +160,55 @@ platform:
hostedZone: Z00147933I3NWOQ6M4699
hostedZoneRole: arn:aws:iam::<account-a>:role/<role-name>
```
Please see Open Questions for further discussion of the install config.
### API: ExternalDNS Operator
[ExternalDNS Operator](https://github.com/openshift/external-dns-operator) allows you to deploy and manage [ExternalDNS](https://github.com/kubernetes-sigs/external-dns),
a cluster-internal component which makes Kubernetes resources discoverable through public DNS servers. This enhancement
extends the ExternalDNS Operator to support cross-account DNS zones in AWS. We extend the`ExternalDNS` API object by adding
the field `RoleARN` to `ExternalDNSAWSProviderOptions` struct (`spec.provider.aws`).
Refer to [`externaldns_types.go`](https://github.com/openshift/external-dns-operator/blob/main/api/v1beta1/externaldns_types.go)
for the existing API structure of the `ExternalDNS` object.

In addition to adding `RoleARN`, the `Credentials` field was updated to be optional. Prior to this change, the `Credentials`
field was required if `spec.provider.aws` was specified. However, OpenShift clusters are exempt from explicitly
providing `Credentials` since the ExternalDNS Operator generates a `CredentialRequest` automatically. Consequently, to
enable OpenShift clusters to specify `RoleARN` without `Credentials`, it is necessary to make the `Credentials` field
optional. The validation webhook takes care of guaranteeing that non-OpenShift clusters include credentials through the
API.

The ExternalDNS Operator's validation webhook will use the existing [IsARN](https://pkg.go.dev/github.com/aws/aws-sdk-go/service/s3/internal/arn#IsARN)
function to validate the `RoleARN` field. The approach mentioned above differs from the API design documented in the
[API: DNS](#API-DNS) section, which employs CRD validation utilizing a regular expression. Using the existing `IsARN`
function provides more flexibility and reliability since it is maintained by AWS themselves, which helps to mitigate
potential issues with edge cases.

```go
type ExternalDNSAWSProviderOptions struct {
// Credentials is a reference to a secret containing
// the following keys (with corresponding values):
//
// * aws_access_key_id
// * aws_secret_access_key
//
// See
// https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md
// for more information.
//
// +kubebuilder:validation:Optional
// +optional
Credentials SecretReference `json:"credentials"`

// RoleARN contains the ARN of a IAM role that will be assumed when using the AWS API.
// It provides the ability to use a hosted zone in another AWS account.
//
// +kubebuilder:validation:Optional
// +optional
RoleARN *string `json:"roleARN,omitempty"`
}
```

See [ExternalDNS Operator Details](#External-DNS-Operator-Details) for more implementation details.

### Implementation Details/Notes/Constraints [optional]

Expand Down Expand Up @@ -218,9 +262,37 @@ An example IAM Trust Policy:

```

#### ExternalDNS Operator Details

Prior to this enhancement, the [ExternalDNS Operator API](https://github.com/openshift/external-dns-operator/blob/main/api)
did not allow users to configure an AWS role ARN field. As a result, there wasn't a supported way to use the External
DNS Operator to create DNS records in another AWS account within a shared VPC.

The ExternalDNS Operator configures and runs the [ExternalDNS](https://github.com/openshift/external-dns) binary.
Conveniently, ExternalDNS already supports the [`--aws-assume-role`](https://github.com/openshift/external-dns/blob/fe00b4b83c2263282a9068655e8e3fbbc167b653/docs/faq.md#can-external-dns-manageaddremove-records-in-a-hosted-zone-which-is-setup-in-different-aws-account)
argument, which uses the specified AWS role ARN when creating new DNS records. Therefore, to support shared VPCs with
the ExternalDNS Operator, we update the [API](#API-External-DNS-Operator) to expose the `--aws-assume-role` argument
for the ExternalDNS binary.

By design, ExternalDNS can update both public and private DNS zones using credentials that the cluster-admin provides.
Therefore, ExternalDNS Operator users are able to use the new [API](#API-External-DNS-Operator) to update both public
and private DNS zones in other accounts, provided the role ARN has appropriate permissions.

For this effort, we took into consideration the [`--aws-assume-role-external-id`](https://github.com/kubernetes-sigs/external-dns/blob/22da9f231dbc6faa3a668b507a4a06823a129609/pkg/apis/externaldns/types.go#L460)
flag which associates an `ExternalID` with an assumed role. AWS [suggests](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html)
the use of `ExternalID` in specific scenarios to mitigate the confused deputy problem. However, for this effort, we
opted not to include this functionality at the moment. The reason behind this decision is that incorporating support
for ExternalID throughout all the components that enable Shared VPC (including API, Install, and Ingress Operator) would
be necessary, but there hasn't been a specific customer need for it thus far.

More details on using a role ARN to create DNS records in another AWS account will be documented in ExternalDNS
Operator's [Docs](https://github.com/openshift/external-dns-operator/tree/main/docs).

### Risks and Mitigations

Add-on operators needing permissions will not work unless updated to use the role ARN.
Add-on operators needing permissions will not work unless updated to use the role ARN. The following add-on operators
have been addressed:
- ExternalDNS Operator

### Drawbacks

Expand All @@ -235,14 +307,23 @@ platform dependencies in this config, but it should be carefully considered.

An e2e test will be setup which utilizes both AWS CI accounts.

For the ExternalDNS Operator, we will add e2e tests utilizing both AWS CI accounts using the RoleARN to create DNS
records.

### Graduation Criteria

This functionality is targeted for 4.14 GA and for backporting to previous releases.
This functionality is targeted for 4.14 GA and for backporting to previous releases.

Note that the ExternalDNS Operator updates to support Shared VPC are out of payload and are therefore not aligned with
OpenShift releases and will not need backporting.

#### Dev Preview -> Tech Preview

Behind `PrivateHostedZoneAWS` feature gate.

Updates to the ExternalDNS Operator will be made available without being restricted by a feature gate. These updates
will be considered GA at the time of merging the feature.

#### Tech Preview -> GA

- E2E CI coverage with satisfactory quality signal is needed to lift feature gate.
Expand All @@ -258,7 +339,12 @@ TODO: there should be minimal impact here.

### Version Skew Strategy

n/a
There are potential version skew scenarios for the ExternalDNS Operator support of Shared VPC. The updates to the
ExternalDNS Operator are decoupled from the standard release schedule of OpenShift Container Platform. Consequently, in
a Shared VPC cluster using cross-account DNS record creation with the Ingress Operator, there is a possibility
of installing a version of the ExternalDNS Operator that lacks support for this same feature. However, it is important
to note that the ExternalDNS Operator and the installer or Ingress Operator do not rely on each other's logic.
Hence, this mismatch does not pose a concern.

### Operational Aspects of API Extensions

Expand All @@ -267,7 +353,7 @@ n/a
#### Failure Modes

- Installation will fail if there are any issues with role during install.
- If there are issues with the role during day-2 operations, the cluster ingress operator will not
- If there are issues with the role during day-2 operations, the Cluster Ingress Operator will not
be able to create DNS records and should log appropriate error messages.

#### Support Procedures
Expand Down

0 comments on commit 4cba6b6

Please sign in to comment.