From ae6c18cad7f4f854a91a7baf5b0d33f7ddec31a4 Mon Sep 17 00:00:00 2001 From: Elias Lindkvist Date: Thu, 5 Dec 2024 12:56:25 +0100 Subject: [PATCH 1/5] docs: Improve readme steps for installation of apps --- README.md | 65 ++++++++++++++++++++++++++++++++++++--------- bin/update-ips.bash | 4 +-- 2 files changed, 55 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 22dc9acf8..5084c3fde 100644 --- a/README.md +++ b/README.md @@ -114,6 +114,21 @@ There are two options when managing DNS records, manually or ExternalDNS. Assuming you already have everything needed to install the apps, this is what you need to do. +> [!NOTE] +> Depending on your infrastructure, you might utilize a Service of type LoadBalancer for the ingress controller. This means you will not have an IP for the domains before installing the ingress controller. After configuring and validating the config, you can install just the ingress controller before the rest of apps with the following command +> +> ```bash +> ./bin/ck8s ops helmfile sc apply -lapp=ingress-nginx --include-transitive-needs +> ./bin/ck8s ops helmfile wc apply -lapp=ingress-nginx --include-transitive-needs +> ``` +> +> The IP is then available on the ingress controller Service +> +> ```bash +> ./bin/ck8s ops kubectl sc -n ingress-nginx get svc ingress-nginx-controller +> ./bin/ck8s ops kubectl wc -n ingress-nginx get svc ingress-nginx-controller +> ``` + The other option is to let ExternalDNS manage your DNS records, currently only AWS Route 53 is supported. You configure ExternalDNS later in the process. @@ -132,10 +147,10 @@ You configure ExternalDNS later in the process. export CK8S_K8S_INSTALLER=[kubespray|capi] # set this to whichever installer was used for the kubernetes layer ``` - > [!NOTE] - > The `air-gapped` flavor has a lot of the same settings as the `prod` flavor but with some additional variables that you need to configure yourself (these are set to `set-me`). +> [!NOTE] +> The `air-gapped` flavor has a lot of the same settings as the `prod` flavor but with some additional variables that you need to configure yourself (these are set to `set-me`). -1. Then set the path to where the ck8s configuration should be stored and the PGP fingerprint of the key(s) to use for encryption: +2. Then set the path to where the ck8s configuration should be stored and the PGP fingerprint of the key(s) to use for encryption: ```bash export CK8S_CONFIG_PATH=${HOME}/.ck8s/my-ck8s-cluster @@ -154,22 +169,27 @@ You configure ExternalDNS later in the process. ./bin/ck8s init both ``` - > [!NOTE] - > It is possible to initialize `wc` and `sc` clusters separately by replacing `both` when running the `init` command: - > - > ```bash - > ./bin/ck8s init wc - > ./bin/ck8s init sc - > ``` +> [!NOTE] +> It is possible to initialize `wc` and `sc` clusters separately by replacing `both` when running the `init` command: +> +> ```bash +> ./bin/ck8s init wc +> ./bin/ck8s init sc +> ``` -1. Edit the configuration files that have been initialized in the configuration path. +4. Edit the configuration files that have been initialized in the configuration path. Make sure that the `objectStorage` values are set in `common-config.yaml` or `sc-config.yaml` and `wc-config.yaml`, as well as required credentials in `secrets.yaml` according to your `objectStorage.type`. The type may already be set in the default configuration found in the `defaults/` directory depending on your selected cloud provider. Set `objectStorage.s3.*` if you are using S3 or `objectStorage.gcs.*` if you are using GCS. Enable ExternalDNS `externalDns.enabled` and set the required variables, if you want ExternalDNS to manage your records from inside your cluster. It requires credentials to route53, `txtOwnerId`, `endpoints` if `externalDns.sources.crd` is enabled. -1. Create S3 buckets - optional +> [!NOTE] +> One important configuration is whether or not you need to use proxy protocol for the ingress controller which depends on what infrastructure you use. You enable it and need to set an annotation depending on your infrastructure. Example for openstack +> `ingressNginx.controller.config.useProxyProtocol: true` +> `ingressNginx.controller.service.annotations: { loadbalancer.openstack.org/proxy-protocol: "true" }` + +5. Create S3 buckets - optional If you have set `objectStorage.type: s3`, then you need to create the buckets specified under `objectStorage.buckets` in your configuration files. You can run the script `scripts/S3/entry.sh create` to create the buckets required. The script uses `s3cmd` in the background and it uses the `${HOME}/.s3cfg` file for configuration and authentication for your S3 provider. @@ -205,6 +225,27 @@ You configure ExternalDNS later in the process. ) ``` +1. Update Network Policies + + ```bash + ./bin/ck8s update-ips both apply + ``` + +1. Validate config and fill in missing values + This should indicate any missing configuration that still needs to be set. + + ```bash + ./bin/ck8s validate sc + ./bin/ck8s validate wc + ``` + +1. **Optional step**, install ingress controller and setup DNS as explained above. + + ```bash + ./bin/ck8s ops helmfile sc apply -lapp=ingress-nginx --include-transitive-needs + ./bin/ck8s ops helmfile wc apply -lapp=ingress-nginx --include-transitive-needs + ``` + 1. **Note**, for this step each cluster need to be up and running already. Deploy the apps: diff --git a/bin/update-ips.bash b/bin/update-ips.bash index bdb508240..c8853e3c9 100755 --- a/bin/update-ips.bash +++ b/bin/update-ips.bash @@ -161,8 +161,8 @@ get_dns_ips() { local -a ips mapfile -t ips < <(dig +short "${domain}" | grep '^[.0-9]*$') if [ ${#ips[@]} -eq 0 ]; then - log_error "No IPs for ${domain} was found" - exit 1 + log_error "No IPs for ${domain} was found. Will block all IPs" + echo "0.0.0.0" fi echo "${ips[@]}" } From 3f5f76ef1dd50824d3ec1d0b355ac74acd58bc7a Mon Sep 17 00:00:00 2001 From: Elias Lindkvist Date: Thu, 5 Dec 2024 15:26:03 +0100 Subject: [PATCH 2/5] another update-ips reminder. --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 5084c3fde..24c6eb019 100644 --- a/README.md +++ b/README.md @@ -246,6 +246,12 @@ You configure ExternalDNS later in the process. ./bin/ck8s ops helmfile wc apply -lapp=ingress-nginx --include-transitive-needs ``` + After configuring the DNS, update the Network Policies again. + + ```bash + ./bin/ck8s update-ips both apply + ``` + 1. **Note**, for this step each cluster need to be up and running already. Deploy the apps: From 7119e1274d0d1630657ae996a20aed709efbe434 Mon Sep 17 00:00:00 2001 From: Elias Lindkvist Date: Tue, 10 Dec 2024 13:09:20 +0100 Subject: [PATCH 3/5] Add update-ips dry-run --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 24c6eb019..0fccf01e5 100644 --- a/README.md +++ b/README.md @@ -228,6 +228,7 @@ You configure ExternalDNS later in the process. 1. Update Network Policies ```bash + ./bin/ck8s update-ips both dry-run ./bin/ck8s update-ips both apply ``` @@ -249,6 +250,7 @@ You configure ExternalDNS later in the process. After configuring the DNS, update the Network Policies again. ```bash + ./bin/ck8s update-ips both dry-run ./bin/ck8s update-ips both apply ``` From 4f90578de8fa004cc46449d75ed3852aa41bdfec Mon Sep 17 00:00:00 2001 From: Elias Lindkvist Date: Wed, 18 Dec 2024 14:43:23 +0100 Subject: [PATCH 4/5] Forest before the trees --- README.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 0fccf01e5..f1c953ef3 100644 --- a/README.md +++ b/README.md @@ -35,8 +35,14 @@ This repository installs all the applications of ck8s on top of already created To setup the clusters see [compliantkubernetes-kubespray][compliantkubernetes-kubespray]. A service-cluster (sc) or workload-cluster (wc) can be created separately but all of the applications will not work correctly unless both are running. -All config files will be located under `CK8S_CONFIG_PATH`. +We follow the "configuration as code" principle which means that all configuration necessary to configure and operate the platform resides in the `CK8S_CONFIG_PATH` directory. There will be four config files: `common-config.yaml`, `wc-config.yaml`, `sc-config.yaml` and `secrets.yaml`. + +We strongly suggest to make your config directory part of a git repository so that it is stored safely and to allow you to rollback previously applied changes. +We additionally suggest that you make Apps a submodule of your config repository in order to properly track which version of Apps you have applied and make your config repository the single source of truth of your environment. + +All operations are done through the `./bin/ck8s` command line tool. Run `./bin/ck8s help` for a complete set of possible commands. + See [Quickstart](#Quickstart) for instructions on how to initialize the repo ### :cloud: Cloud providers :cloud: From 81b021cd373790d87dd52d90a6fd0f74adcc4a9b Mon Sep 17 00:00:00 2001 From: Elias Lindkvist Date: Wed, 18 Dec 2024 14:50:38 +0100 Subject: [PATCH 5/5] remove update-ips change --- bin/update-ips.bash | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/bin/update-ips.bash b/bin/update-ips.bash index c8853e3c9..bdb508240 100755 --- a/bin/update-ips.bash +++ b/bin/update-ips.bash @@ -161,8 +161,8 @@ get_dns_ips() { local -a ips mapfile -t ips < <(dig +short "${domain}" | grep '^[.0-9]*$') if [ ${#ips[@]} -eq 0 ]; then - log_error "No IPs for ${domain} was found. Will block all IPs" - echo "0.0.0.0" + log_error "No IPs for ${domain} was found" + exit 1 fi echo "${ips[@]}" }