Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set custom cluster tag on egress stacks [2/2] #4939

Merged
merged 1 commit into from
Jan 27, 2022

Conversation

mikkeloscar
Copy link
Contributor

Step 2 of #4910 (Described in detail below)


Currently the controller will set a tag with the format: kubernetes.io/cluster/<id>=owned on the stack and this will be propagated down to the subnets. The tag has two purposes:

  1. Identify resources as belonging to a particular subnet
  2. Find the egress stack belonging to a particular cluster in order to manage the right stack in case you have multiple clusters in one AWS account/region.

The downside of the tag is that having such a tag on the subnets means that Kubernetes (Service Type Load Balancer logic) will consider the NAT subnets when finding subnets for ELBs. We never want to use the NAT subnets for this purpose so it's desirable to avoid tagging the subnets at all to prevent this from happening no matter how all the other subnets in the account are tagged.

The idea to avoid this issue is to change the tag used to identify the cluster id. Instead of using the tag format: kubernetes.io/cluster/<id>=owned we instead use a custom tag format: zalando.org/cluster/<id>=owned so we still can track the cluster ID ownership, but not interfere with the "Service Type Load Balancer logic" which is in core Kubernetes.

The change is done in two steps:

  1. Add the additional tag: zalando.org/cluster/<id>=owned
  2. Change the tag prefix to zalando.org/cluster/.

We need to do it in two steps to avoid that we loose track of which stacks belong to a cluster as this is identified via the clusterID tag.

Signed-off-by: Mikkel Oscar Lyderik Larsen <[email protected]>
@mikkeloscar
Copy link
Contributor Author

👍

1 similar comment
@gargravarr
Copy link
Contributor

gargravarr commented Jan 27, 2022

👍

@gargravarr gargravarr merged commit a388880 into dev Jan 27, 2022
@gargravarr gargravarr deleted the egress-controller-cluster-id-tag branch January 27, 2022 08:30
This was referenced Jan 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants