Use the kubeApiServerConfig clientCAFile field

[slu2011]

Hello!

We have a usecase that would need to be able to specify a different client-CA file for the kube-apiserver. Also, it seems that there were prior discussion about this requirement too(#3243), although it was closed for inactivity.

This change just enables the ClientCAFile field of the KubeAPIServerConfig struct, So that the user can specify the ClientCAFile and the apiserver config yaml would reflect it.

Thanks a lot!

[k8s-ci-robot]

Welcome @slu2011!

It looks like this is your first PR to kubernetes/kops 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kops has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @slu2011. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
• Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[rifelpet]

/ok-to-test

Thanks @slu2011! do you mind explaining your use-case for setting this field?

[slu2011]

/test pull-kops-e2e-kubernetes-aws

[slu2011]

/test pull-kops-e2e-k8s-containerd

[slu2011]

/ok-to-test

Thanks @slu2011! do you mind explaining your use-case for setting this field?

Thanks a lot @rifelpet.

Our usecase is that we need to have each human admin to have short-lived client cert to access the cluster.. The admin client cert are generated separately using another CA (ca_admin). Thus we would like to have a separate client CA file for the apiserver, which contains both the K8s CA and the ca_admin.

We cannot put multiple certs in ca.crt as ca.crt must contain one cert.

There are also older issues regarding this: #3243

[olemarkus]

We have kubeAPIServer := b.Cluster.Spec.KubeAPIServer above, and then you set kubeAPIServer.ClientCAFile = b.Cluster.Spec.KubeAPIServer.ClientCAFile

Is this rather redundant or am I missing something?

[slu2011]

Thanks for your comment @olemarkus.

Before the change, line 300 would overwrite kubeAPIServer.ClientCAFile to filepath.Join(b.PathSrvKubernetes(), "ca.crt") regardless.

I did not realize we have kubeAPIServer := b.Cluster.Spec.KubeAPIServer above. A better change might be overwrite the kubeAPIServer.ClientCAFile to filepath.Join(b.PathSrvKubernetes(), "ca.crt") only when b.Cluster.Spec.KubeAPIServer.ClientCAFile is empty string (default)? Updated the change.

[olemarkus]

Can you add some docs on how to use this as well? I assume you want to populate the actual file using fileAssets

[slu2011]

Can you add some docs on how to use this as well? I assume you want to populate the actual file using fileAssets

Would that be OK if add a section in docs/cluster_spec.md?

In our usecase, we would like the client-ca file to contain two certs (K8s ca.crt and our own admin_ca.crt). We embed the admin_ca.crt in the AMI and use kops hooks to create the combined file.

[olemarkus]

Thanks for adding the docs. Can you mention that the file must include the cluster ca?

Alternatively I wonder if you can always append the cluster CA. If people do this manually, it will break the cluster CA rotation functionality I am working on.

[slu2011]

Thanks for adding the docs. Can you mention that the file must include the cluster ca?

Alternatively I wonder if you can always append the cluster CA. If people do this manually, it will break the cluster CA rotation functionality I am working on.

Thanks for the info regarding CA rotation. Yes we currently do append the ca.crt in the end of client-ca.crt. The append work is done using a kops-hook. In my opinion, using a Kops hook to do the append is at least an OK solution (since the additional CA is provided by the user, the work to configure it should also be handled by the user -- in this case via a kops hook).

Also, from the user's point of view, using a kops-hook has the best flexibility that they can put any logic in it.

Using kops hook can also cover the case that the CA is rotated -- the hook can be set to run continuously, and it also compared the content of the ca.crt and the appended ca.crt copy. If diff is found, it can append the updated ca.crt.

Moreover, I think overriding the client-ca file also means that the user need to be responsible to manage the client-ca file, which includes making sure that the rotated ca.crt should be appended, as well as that the user provided CA is also rotated correctly.

Do you think this is a OK solution? If so I will update the doc to cover the above append and update ca.crt logic, and the recommended way to setup the kops hook.

[olemarkus]

Yes, I think it is sufficient to add that for now

[olemarkus]

Nit: it will have :) It is still a PR. But you could just refer to https://kops.sigs.k8s.io/rotate-secrets/

[olemarkus]

Thanks. The docs could have used relative links. But not important. Can clean this up later.

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: olemarkus, slu2011

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [olemarkus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[slu2011]

@olemarkus many thanks for your help!!

On the other hand, what is the procedure I should follow to cherry-pick this change to release 1.19?