Conceptually, all Gardener components are designed to run as a Pod inside a Kubernetes cluster. The Gardener API server extends the Kubernetes API via the user-aggregated API server concepts. However, if you want to develop it, you may want to work locally with the Gardener without building a Docker image and deploying it to a cluster each and every time. That means that the Gardener runs outside a Kubernetes cluster which requires providing a Kubeconfig in your local filesystem and point the Gardener to it when starting it (see below).
Further details can be found in
This guide is split into three main parts:
- Preparing your setup by installing all dependencies and tools
- Building and starting Gardener components locally
- Using your local Gardener setup to create a Shoot
You can run Gardener (API server, controller manager, scheduler, gardenlet) against any local Kubernetes cluster, however, your seed and shoot clusters must be deployed to a cloud provider. Currently, it is not possible to run Gardener entirely isolated from any cloud provider. This means that to be able create Shoot clusters you need to register an external Seed cluster (e.g., one created in AWS).
The copy-paste instructions in this guide are designed for macOS and use the package manager Homebrew.
On macOS run
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"We use git as VCS which you need to install. On macOS run
brew install gitFor other OS, please check the Git installation documentation.
Install the latest version of Go. On macOS run
brew install goFor other OS, please check Go installation documentation.
Install kubectl. Please make sure that the version of kubectl is at least v1.11.x. On macOS run
brew install kubernetes-cliFor other OS, please check the kubectl installation documentation.
You also need the Helm CLI. On macOS run
brew install helmFor other OS please check the Helm installation documentation.
We use OpenVPN to establish network connectivity from the control plane running in the Seed cluster to the Shoot's worker nodes running in private networks.
To harden the security we need to generate another secret to encrypt the network traffic (details).
Please install the openvpn binary. On macOS run
brew install openvpn
export PATH=$(brew --prefix openvpn)/sbin:$PATHFor other OS, please check the OpenVPN downloads page.
You need to have docker installed and running. On macOS run
brew install --cask dockerFor other OS please check the docker installation documentation.
iproute2 provides a collection of utilities for network administration and configuration. On macOS run
brew install iproute2macbrew install jqGNU Parallel is a shell tool for executing jobs in parallel, used by the code generation scripts (make generate). On macOS run
brew install parallelWhen running on macOS, install the GNU core utilities and friends:
brew install coreutils gnu-sed gnu-tar grepThis will create symbolic links for the GNU utilities with g prefix in /usr/local/bin, e.g., gsed or gbase64. To allow using them without the g prefix please put /usr/local/opt/coreutils/libexec/gnubin etc. at the beginning of your PATH environment variable, e.g., export PATH=/usr/local/opt/coreutils/libexec/gnubin:$PATH (brew will print out instructions for each installed formula).
Apart from Linux distributions and macOS, the local gardener setup can also run on the Windows Subsystem for Linux 2.
While WSL1, plain docker for windows and various Linux distributions and local Kubernetes environments may be supported, this setup was verified with:
- WSL2
- Docker Desktop WSL2 Engine
- Ubuntu 18.04 LTS on WSL2
- Nodeless local garden (see below)
The Gardener repository and all the above-mentioned tools (git, golang, kubectl, ...) should be installed in your WSL2 distro, according to the distribution-specific Linux installation instructions.
Clone the repository from GitHub into your $GOPATH.
mkdir -p $GOPATH/src/github.com/gardener
cd $GOPATH/src/github.com/gardener
git clone [email protected]:gardener/gardener.git
cd gardenerNote: Gardener is using Go modules and cloning the repository into
$GOPATHis not a hard requirement. However it is still recommended to clone into$GOPATHbecausek8s.io/code-generatordoes not work yet outside of$GOPATH- kubernetes/kubernetes#86753.
ℹ️ In the following guide, you have to define the configuration (CloudProfiles, SecretBindings, Seeds, etc.) manually for the infrastructure environment you want to develop against.
Additionally, you have to register the respective Gardener extensions manually.
If you are rather looking for a quick start guide to develop entirely locally on your machine (no real cloud provider or infrastructure involved) then you should rather follow this guide.
For the development of Gardener you need a Kubernetes API server on which you can register Gardener's own Extension API Server as APIService. This cluster doesn't need any worker nodes to run pods, though, therefore, you can use the "nodeless Garden cluster setup" residing in hack/local-garden. This will start all minimally required components of a Kubernetes cluster (etcd, kube-apiserver, kube-controller-manager)
and an etcd Instance for the gardener-apiserver as Docker containers. This is the easiest way to get your
Gardener development setup up and running.
Using the nodeless cluster setup
Use the provided Makefile rules to start your local Garden:
make local-garden-up
[...]
Starting gardener-dev kube-etcd cluster..!
Starting gardener-dev kube-apiserver..!
Starting gardener-dev kube-controller-manager..!
Starting gardener-dev gardener-etcd cluster..!
namespace/garden created
clusterrole.rbac.authorization.k8s.io/gardener.cloud:admin created
clusterrolebinding.rbac.authorization.k8s.io/front-proxy-client created
[...]ℹ️ [Optional] If you want to develop the SeedAuthorization feature then you have to run make ACTIVATE_SEEDAUTHORIZER=true local-garden-up. However, please note that this forces you to start the gardener-admission-controller via make start-admission-controller.
To tear down the local Garden cluster and remove the Docker containers, simply run:
make local-garden-downAlternative: Using a local kubernetes cluster
Instead of starting a kubernetes API server and etcd as docker containers, you can also opt for running a local kubernetes cluster, provided by e.g. minikube, kind or docker desktop.
Note: Gardener requires self-contained kubeconfig files because of a security issue. You can configure your minikube to create self-contained kubeconfig files via:
minikube config set embed-certs trueor when starting the local cluster
minikube start --embed-certs
Alternative: Using a remote kubernetes cluster
For some testing scenarios, you may want to use a remote cluster instead of a local one as your Garden cluster.
To do this, you can use the "remote Garden cluster setup" residing in hack/remote-garden. This will start an etcd instance for the gardener-apiserver as a Docker container, and open tunnels for accessing local gardener components from the remote cluster.
To avoid mistakes, the remote cluster must have a garden namespace labeled with gardener.cloud/purpose=remote-garden.
You must create the garden namespace and label it manually before running make remote-garden-up as described below.
Use the provided Makefile rules to bootstrap your remote Garden:
export KUBECONFIG=<path to kubeconfig>
make remote-garden-up
[...]
# Start gardener etcd used to store gardener resources (e.g., seeds, shoots)
Starting gardener-dev-remote gardener-etcd cluster!
[...]
# Open tunnels for accessing local gardener components from the remote cluster
[...]To close the tunnels and remove the locally-running Docker containers, run:
make remote-garden-downNote: The minimum K8S version of the remote cluster that can be used as Garden cluster is
1.19.x.
ℹ️ [Optional] If you want to use the remote Garden cluster setup with the SeedAuthorization feature you have to adapt the kube-apiserver process of your remote Garden cluster. To do this, perform the following steps after running make remote-garden-up:
-
Create an authorization webhook configuration file using the IP of the
garden/quic-serverpod running in your remote Garden cluster and port 10444 that tunnels to your locally runninggardener-admission-controllerprocess.apiVersion: v1 kind: Config current-context: seedauthorizer clusters: - name: gardener-admission-controller cluster: insecure-skip-tls-verify: true server: https://<quic-server-pod-ip>:10444/webhooks/auth/seed users: - name: kube-apiserver user: {} contexts: - name: seedauthorizer context: cluster: gardener-admission-controller user: kube-apiserver
-
Change or add the following command line parameters to your
kube-apiserverprocess:--authorization-mode=<...>,Webhook--authorization-webhook-config-file=<path to config file>--authorization-webhook-cache-authorized-ttl=0--authorization-webhook-cache-unauthorized-ttl=0
-
Delete the cluster role and rolebinding
gardener.cloud:system:seedsfrom your remote Garden cluster.
If your remote Garden cluster is a Gardener shoot, and you can access the seed on which this shoot is scheduled, you can automate the above steps by running the enable-seed-authorizer script and passing the kubeconfig of the seed cluster and the shoot namespace as parameters:
hack/local-development/remote-garden/enable-seed-authorizer <seed kubeconfig> <namespace>Note: The configuration changes introduced by this script result in a working
SeedAuthorizationfeature only on shoots for which theReversedVPNfeature is not enabled. If the corresponding feature gate is enabled ingardenlet, add the annotationalpha.featuregates.shoot.gardener.cloud/reversed-vpn: 'false'to the remote Garden shoot to disable it for that particular shoot.
To prevent Gardener from reconciling the shoot and overwriting your changes, add the annotation shoot.gardener.cloud/ignore: 'true' to the remote Garden shoot. Note that this annotation takes effect only if it is enabled via the constollers.shoot.respectSyncPeriodOverwrite: true option in the gardenlet configuration.
To disable the seed authorizer again, run the same script with -d as a third parameter:
hack/local-development/remote-garden/enable-seed-authorizer <seed kubeconfig> <namespace> -dIf the seed authorizer is enabled, you also have to start the gardener-admission-controller via make start-admission-controller.
⚠️ In the remote garden setup all Gardener components run with administrative permissions, i.e., there is no fine-grained access control via RBAC (as opposed to productive installations of Gardener).
Now, that you have started your local cluster, we can go ahead and register the Gardener API Server.
Just point your KUBECONFIG environment variable to the cluster you created in the previous step and run:
make dev-setup
[...]
namespace/garden created
namespace/garden-dev created
deployment.apps/etcd created
service/etcd created
service/gardener-apiserver created
service/gardener-admission-controller created
endpoints/gardener-apiserver created
endpoints/gardener-admission-controller created
apiservice.apiregistration.k8s.io/v1alpha1.core.gardener.cloud created
apiservice.apiregistration.k8s.io/v1beta1.core.gardener.cloud created
apiservice.apiregistration.k8s.io/v1alpha1.seedmanagement.gardener.cloud created
apiservice.apiregistration.k8s.io/v1alpha1.settings.gardener.cloud createdℹ️ [Optional] If you want to enable logging, in the Gardenlet configuration add:
logging:
enabled: trueThe Gardener exposes the API servers of Shoot clusters via Kubernetes services of type LoadBalancer.
In order to establish stable endpoints (robust against changes of the load balancer address), it creates DNS records pointing to these load balancer addresses. They are used internally and by all cluster components to communicate.
You need to have control over a domain (or subdomain) for which these records will be created.
Please provide an internal domain secret (see this for an example) which contains credentials with the proper privileges. Further information can be found here.
kubectl apply -f example/10-secret-internal-domain-unmanaged.yaml
secret/internal-domain-unmanaged createdNext, run the Gardener API Server, the Gardener Controller Manager (optionally), the Gardener Scheduler (optionally), and the Gardenlet in different terminal windows/panes using rules in the Makefile.
make start-apiserver
[...]
I0306 15:23:51.044421 74536 plugins.go:84] Registered admission plugin "ResourceReferenceManager"
I0306 15:23:51.044523 74536 plugins.go:84] Registered admission plugin "DeletionConfirmation"
[...]
I0306 15:23:51.626836 74536 secure_serving.go:116] Serving securely on [::]:8443
[...](Optional) Now you are ready to launch the Gardener Controller Manager.
make start-controller-manager
time="2019-03-06T15:24:17+02:00" level=info msg="Starting Gardener controller manager..."
time="2019-03-06T15:24:17+02:00" level=info msg="Feature Gates: "
time="2019-03-06T15:24:17+02:00" level=info msg="Starting HTTP server on 0.0.0.0:2718"
time="2019-03-06T15:24:17+02:00" level=info msg="Acquired leadership, starting controllers."
time="2019-03-06T15:24:18+02:00" level=info msg="Starting HTTPS server on 0.0.0.0:2719"
time="2019-03-06T15:24:18+02:00" level=info msg="Found internal domain secret internal-domain-unmanaged for domain nip.io."
time="2019-03-06T15:24:18+02:00" level=info msg="Successfully bootstrapped the Garden cluster."
time="2019-03-06T15:24:18+02:00" level=info msg="Gardener controller manager (version 1.0.0-dev) initialized."
time="2019-03-06T15:24:18+02:00" level=info msg="ControllerRegistration controller initialized."
time="2019-03-06T15:24:18+02:00" level=info msg="SecretBinding controller initialized."
time="2019-03-06T15:24:18+02:00" level=info msg="Project controller initialized."
time="2019-03-06T15:24:18+02:00" level=info msg="Quota controller initialized."
time="2019-03-06T15:24:18+02:00" level=info msg="CloudProfile controller initialized."
[...](Optional) Now you are ready to launch the Gardener Scheduler.
make start-scheduler
time="2019-05-02T16:31:50+02:00" level=info msg="Starting Gardener scheduler ..."
time="2019-05-02T16:31:50+02:00" level=info msg="Starting HTTP server on 0.0.0.0:10251"
time="2019-05-02T16:31:50+02:00" level=info msg="Acquired leadership, starting scheduler."
time="2019-05-02T16:31:50+02:00" level=info msg="Gardener scheduler initialized (with Strategy: SameRegion)"
time="2019-05-02T16:31:50+02:00" level=info msg="Scheduler controller initialized."
[...]The Gardener should now be ready to operate on Shoot resources. You can use
kubectl get shoots
No resources found.to operate against your local running Gardener API Server.
Note: It may take several seconds until the Gardener API server has been started and is available.
No resources foundis the expected result of our initial development setup.
The steps below describe the general process of creating a Shoot. Have in mind that the steps do not provide full example manifests. The reader needs to check the provider documentation and adapt the manifests accordingly.
The next steps require modifications of the example manifests. These modifications are part of local setup and should not be git push-ed. To do not interfere with git, let's copy the example manifests to dev/ which is ignored by git.
cp example/*.yaml dev/Every Shoot is associated with a Project. Check the corresponding example manifests dev/00-namespace-garden-dev.yaml and dev/05-project-dev.yaml. Adapt them and create them.
kubectl apply -f dev/00-namespace-garden-dev.yaml
kubectl apply -f dev/05-project-dev.yamlMake sure that the Project is successfully reconciled:
$ kubectl get project dev
NAME NAMESPACE STATUS OWNER CREATOR AGE
dev garden-dev Ready [email protected] kubernetes-admin 6sThe CloudProfile resource is provider specific and describes the underlying cloud provider (available machine types, regions, machine images, etc.). Check the corresponding example manifest dev/30-cloudprofile.yaml. Check also the documentation and example manifests of the provider extension. Adapt dev/30-cloudprofile.yaml and apply it.
kubectl apply -f dev/30-cloudprofile.yamlThe Known Extension Implementations section contains a list of available extension implementations. You need to create a ControllerRegistration and ControllerDeployment for
- at least one infrastructure provider
- a dns provider (if the DNS for the Seed is not disabled)
- at least one operating system extension
- at least one network plugin extension
As a convention, the example ControllerRegistration manifest (containing also the necessary ControllerDeployment) for an extension is located under example/controller-registration.yaml in the corresponding repository (for example for AWS the ControllerRegistration can be found here). An example creation for provider-aws (make sure to replace <version> with the newest released version tag):
kubectl apply -f https://raw.githubusercontent.com/gardener/gardener-extension-provider-aws/<version>/example/controller-registration.yamlInstead of updating extensions manually you can use Gardener Extensions Manager to install and update extension controllers. This is especially useful if you want to keep and maintain your development setup for a longer time. Also, please refer to this document for further information about how extensions are registered in case you want to use other versions than the latest releases.
Shoot controlplanes run in seed clusters, so we need to create our first Seed now.
Check the corresponding example manifest dev/40-secret-seed.yaml and dev/50-seed.yaml. Update dev/40-secret-seed.yaml with base64 encoded kubeconfig of the cluster that will be used as Seed (the scope of the permissions should be identical to the kubeconfig that the Gardenlet creates during bootstrapping - for now, cluster-admin privileges are recommended).
kubectl apply -f dev/40-secret-seed.yamlAdapt dev/50-seed.yaml - adjust .spec.secretRef to refer the newly created Secret, adjust .spec.provider with the Seed cluster provider and revise the other fields.
kubectl apply -f dev/50-seed.yamlOnce the Seed is created, start the Gardenlet to reconcile it. The make start-gardenlet command will automatically configure the local Gardenlet process to use the Seed and its kubeconfig. If you have multiple Seeds, you have to specify which to use by setting the SEED_NAME environment variable like in make start-gardenlet SEED_NAME=my-first-seed.
make start-gardenlet
time="2019-11-06T15:24:17+02:00" level=info msg="Starting Gardenlet..."
time="2019-11-06T15:24:17+02:00" level=info msg="Feature Gates: HVPA=true, Logging=true"
time="2019-11-06T15:24:17+02:00" level=info msg="Acquired leadership, starting controllers."
time="2019-11-06T15:24:18+02:00" level=info msg="Found internal domain secret internal-domain-unmanaged for domain nip.io."
time="2019-11-06T15:24:18+02:00" level=info msg="Gardenlet (version 1.0.0-dev) initialized."
time="2019-11-06T15:24:18+02:00" level=info msg="ControllerInstallation controller initialized."
time="2019-11-06T15:24:18+02:00" level=info msg="Shoot controller initialized."
time="2019-11-06T15:24:18+02:00" level=info msg="Seed controller initialized."
[...]The Gardenlet will now reconcile the Seed. Check the progess from time to time until it's Ready:
kubectl get seed
NAME STATUS PROVIDER REGION AGE VERSION K8S VERSION
seed-aws Ready aws eu-west-1 4m v1.11.0-dev v1.18.12A Shoot requires a SecretBinding. The SecretBinding refers to a Secret that contains the cloud provider credentials. The Secret data keys are provider specific and you need to check the documentation of the provider to find out which data keys are expected (for example for AWS the related documentation can be found here). Adapt dev/70-secret-provider.yaml and dev/80-secretbinding.yaml and apply them.
kubectl apply -f dev/70-secret-provider.yaml
kubectl apply -f dev/80-secretbinding.yamlAfter the SecretBinding creation, you are ready to proceed with the Shoot creation. You need to check the documentation of the provider to find out the expected configuration (for example for AWS the related documentation and example Shoot manifest can be found here). Adapt dev/90-shoot.yaml and apply it.
To make sure that a specific Seed cluster will be chosen or to skip the scheduling (the sheduling requires Gardener Scheduler to be running), specify the .spec.seedName field (see here).
kubectl apply -f dev/90-shoot.yamlWatch the progress of the operation and make sure that the Shoot will be successfully created.
watch kubectl get shoot --all-namespaces