Scaling machines with CAPH

[robertlemke]

I spent the last couple of days to try out this Cluster API provider and I'm very happy that such a project exists for Hetzner Cloud – thanks a lot for creating it!

One question I have is related to scaling, as I couldn't find anything about it in this project's documentation.

According to the Cluster API docs, I should be able to manually scale my scaling up MachineDeployment:

kubectl scale machinedeployment foo --replicas=5

When I do that, I see that a new HCloudMachine is created, a new node is created … but then after a very short time, it is scaled down again. The "desired" machine deployments stay at "1".

I also took a look at MachineSetand checked the log output of the controllers, but couldn't make much sense of it.

So, which mechanism / controller takes influence on this? What's the proper way to scale manually or, bonus question: does the autoscaler work as well?

[batistein]

Hello, the cluster-api is responsible for scaling machines. As a user, you interact with the machinedeployment or another high-level API Object (e.g cluster).
The machinedeployment then sizes the machineset. The machineset creates machine objects and then the machinecontroller uses the ProviderMachineTemplate to create ProviderMachine objects. The ProviderMachine objects (e.g. hcloudMachine) are then reconciled by caph.
I have never tried to scale a machine deployment using kubectl scale. In our product "Syself Autopilot", all the scaling is done by changing the cluster object as we use the clusterclass feature of capi for our Kubernetes platform.

I would suggest to directly edit the machinedeployment field replicas and check again if this solves your issue.
If it scales down again, there might be another issue with your cluster like it's not fully initialized...

And yes, cluster-autoscaler works fine, we use it extensively.

[batistein]

Yes, we use it extensively for our platform. We have also developed a solution based on ClusterClass for the Sovereign Cloud Stack: https://scs.community/

[robertlemke]

After some holidays I now finally migrated my setup to ClusterClass / cluster topology. I can scale my worker machine deployments manually, for example by setting the replicas in the respective Cluster API object.

Now I'm trying to setup the cluster-autoscaler. I have seen the general documentation about setting up cluster-autoscaler with Cluster API, however, I'm unsure about the concrete (best-practice) setup.

Is there any example you are aware of, which demonstrates how to configure the cluster-autoscaler for CAPH setups? What's the best practice regarding connecting the management and workload server (https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/clusterapi#connecting-cluster-autoscaler-to-cluster-api-management-and-workload-clusters)?

[batistein]

We run the autoscaler in the workload cluster. The cluster autoscaler requires a lot of tuning and is not a generic solution for every cluster. Since the configuration is done on the cluster autoscaler itself, it is best to have per-cluster deployments.

[Preisschild]

I run multiple cluster-autoscaler deployments (one for each workload cluster) in my management cluster. This works great too.

You just have to configure cluster-autoscaler in the kubeconfig-incluster mode. The pods can then use the <clustername>-kubeconfig secret that is generated by CAPI to connect to the workload clusters.

cloudProvider: clusterapi
clusterAPIMode: kubeconfig-incluster
clusterAPIKubeconfigSecret: <clustername>-kubeconfig
autoDiscovery:
      clusterName: <clustername>

[robertlemke]

Thanks a lot to both of you for sharing your experience with this – it's a bit hard to find related information apart from the reference docs. Maybe I can contribute some documentation when I have a bigger picture …

@Preisschild I started implementing your choice, mostly for security reasons: In my setup, at this point no client needs access to the management cluster's Kube API. And since the management cluster already has a kubeconf for all workload clusters, no additional credentials need to be maintained to grant access into that direction.

What bugged me a bit was that apparently everything needs to be maintained in one single namespace in the management cluster, because reference to secrets, the ClusterClass and some other objects don't support namespaces other than the "current" one. Originally I planned to put all objects related to a workload cluster into its own namespace (for example, hcloud credentials etc). But well, that's not a big deal, it just feels a bit more messy.

Another hint to someone trying this with a management server running on Google Cloud (GKE): Don't try to install the cluster autoscaler in kube-system – it will try to use the leader election lease from GKE, which also uses the autoscaler …