diff --git a/docs/airgap.md b/docs/airgap.md index c75eac0802e..336f8f76309 100644 --- a/docs/airgap.md +++ b/docs/airgap.md @@ -12,6 +12,283 @@ This section describes how to use Harvester in an air gapped environment. Some u The Harvester ISO image contains all the packages to make it work in an air gapped environment. +## Working without HTTP Proxy + +In this example, we use KVM to provision Harvester and integrate it with Rancher. + +### Setup offline Harvester + +1. Clone [harvester/ipxe-examples](https://github.com/harvester/ipxe-examples). +1. Go to `vagrant-pxe-harvester` folder. +1. Edit [settings.yml](https://github.com/harvester/ipxe-examples/blob/c8267b6270660255bf71149a1fef3a7a914550de/vagrant-pxe-harvester/settings.yml#L44) and set `offline: true`. +1. Execute [setup_harvester.sh](https://github.com/harvester/ipxe-examples/blob/main/vagrant-pxe-harvester/setup_harvester.sh) script. + +### Deploy Rancher on k3s and setup private registry in another VM. + +1. Create another KVM with two network interfaces `harvester` and `vagrant-libvirt`. `harvester` network interface is for intranet and `vagrant-libvirt` is for internet. We need `vagrant-libvirt` to download all required resources. We will remove it before we start Rancher. Also, please give this VM at least 300GB to save all required images. +1. Install [Docker](https://www.docker.com/) and [Helm](https://helm.sh/). +1. Create `certs` folder. +``` +mkdir -p certs +``` +1. Generate private registry certificate files. +``` +openssl req \ + -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key \ + -addext "subjectAltName = DNS:myregistry.local" \ + -x509 -days 365 -out certs/domain.crt +``` +1. Move certificate files to `/etc/docker/certs.d`. +``` +sudo mkdir -p /etc/docker/certs.d/myregistry.local:5000 +sudo cp certs/domain.crt /etc/docker/certs.d/myregistry.local:5000/domain.crt +``` +1. Start private registry. +``` +docker run -d \ + -p 5000:5000 \ + --restart=always \ + --name registry \ + -v "$(pwd)"/certs:/certs \ + -v "$(pwd)"/registry:/var/lib/registry \ + -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \ + -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \ + registry:2 +``` +1. Add `myregistry.local` to `/etc/hosts`. Remember to change `192.168.0.50` to your private ip. +``` +# vim /etc/hosts +192.168.0.50 myregistry.local +``` +1. Create `get-rancher` script. +``` +# vim get-rancher +#!/bin/bash +if [[ $# -eq 0 ]] ; then + echo 'This requires you to pass a version for the url like "v2.6.4"' + exit 1 +fi +wget https://github.com/rancher/rancher/releases/download/$1/rancher-images.txt +wget https://github.com/rancher/rancher/releases/download/$1/rancher-load-images.sh +wget https://github.com/rancher/rancher/releases/download/$1/rancher-save-images.sh +chmod +x ./rancher-save-images.sh +chmod +x ./rancher-load-images.sh +``` +1. Make `get-rancher` script be excutable. +``` +chmod +x get-rancher +``` +1. Download `rancher-images.txt`, `rancher-load-images.sh` and `rancher-save-images.sh`. +``` +./get-rancher v2.6.4 +``` +1. Add cert-manager images to `rancher-images.txt`. +``` +helm repo add jetstack https://charts.jetstack.io/ +helm repo update +helm fetch jetstack/cert-manager --version v1.7.1 +helm template ./cert-manager-v1.7.1.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt +``` +1. Sort `rancher-images.txt`. +``` +sort -u rancher-images.txt -o rancher-images.txt +``` +1. Get images. This step may take 1 to 2 hours. It depends on your network speed. +``` +./rancher-save-images.sh --image-list ./rancher-images.txt +``` +1. Create `get-k3s` script. +``` +# vim get-k3s +#!/bin/bash +if [[ $# -eq 0 ]] ; then + echo 'This requires you to pass a version for the url like "v1.23.4+k3s1"' + exit 1 +fi +wget https://github.com/k3s-io/k3s/releases/download/$1/k3s-airgap-images-amd64.tar +wget https://github.com/k3s-io/k3s/releases/download/$1/k3s +wget https://get.k3s.io/ -O install.sh +chmod +x ./k3s +chmod +x ./install.sh +``` +1. Make `get-k3s` be excutable. +``` +chmod +x get-kes +``` +1. Download `k3s-airgap-images-amd64.tar`, `k3s` and `install.sh`. +``` +./get-k3s v1.23.4+k3s1 +``` +1. Download Rancher. +``` +helm repo add rancher-latest https://releases.rancher.com/server-charts/latest +helm fetch rancher-latest/rancher --version=v2.6.4 +``` +1. Download `cert-manager-crds.yaml`. +``` +mkdir cert-manager +curl -L -o cert-manager/cert-manager-crd.yaml https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml +``` +1. Remove `vagrant-libvirt` network interface. We've downloaded all required resources. We can cut network from this step. +1. Move `k3s-airgap-images-amd64.tar` to `/var/lib/rancher/k3s/agent/images/`. +``` +sudo mkdir -p /var/lib/rancher/k3s/agent/images/ +sudo cp k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images/ +``` +1. Create `/etc/rancher/k3s` folder. +``` +mkdir -p /etc/rancher/k3s +``` +1. Add `registries.yaml` to `/etc/rancher/k3s`. +``` +# vim /etc/rancher/k3s/registries.yaml +mirrors: + docker.io: + endpoint: + - "https://myregistry.local:5000/" +configs: + "myregistry.local:5000": + tls: + insecure_skip_verify: true +``` +1. Install k3s. +``` +INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh +``` +1. Generate cert-manager YAML files. +``` +helm template cert-manager ./cert-manager-v1.7.1.tgz --output-dir . \ + --namespace cert-manager \ + --set image.repository=myregistry.local:5000/quay.io/jetstack/cert-manager-controller \ + --set webhook.image.repository=myregistry.local:5000/quay.io/jetstack/cert-manager-webhook \ + --set cainjector.image.repository=myregistry.local:5000/quay.io/jetstack/cert-manager-cainjector \ + --set startupapicheck.image.repository=myregistry.local:5000/quay.io/jetstack/cert-manager-ctl +``` +1. Move `/etc/rancher/k3s/k3s.yaml` to `~/.kube`. +``` +mkdir ~/.kube +sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config +sudo chown $USER ~/.kube/config +export KUBECONFIG=~/.kube/config +``` +1. Install cert-manager. +``` +kubectl create namespace cert-manager +kubectl apply -f cert-manager/cert-manager-crd.yaml +kubectl apply -R -f ./cert-manager +``` +1. Create CA private key and certifacate file. +``` +openssl genrsa -out cakey.pem 2048 +openssl req -x509 -sha256 -new -nodes -key cakey.pem -days 3650 -out cacerts.pem -subj "/CN=cattle-ca" +``` +1. Create `openssl.cnf`. Remember to change `192.168.0.50` to your private ip. +``` +[req] +req_extensions = v3_req +distinguished_name = req_distinguished_name +[req_distinguished_name] +[ v3_req ] +basicConstraints = CA:FALSE +keyUsage = nonRepudiation, digitalSignature, keyEncipherment +extendedKeyUsage = clientAuth, serverAuth +subjectAltName = @alt_names +[alt_names] +DNS.1 = myrancher.local +IP.1 = 192.168.0.50 +``` +1. Generate private key and certificate file for `myrancher.local`. +``` +openssl genrsa -out tls.key 2048 +openssl req -sha256 -new -key tls.key -out tls.csr -subj "/CN=myrancher.local" -config openssl.cnf +openssl x509 -sha256 -req -in tls.csr -CA cacerts.pem \ + -CAkey cakey.pem -CAcreateserial -out tls.crt \ + -days 3650 -extensions v3_req \ + -extfile openssl.cnf +``` +1. Create `cattle-system` namespace. +``` +kubectl create ns cattle-system +``` +1. Create `tls.sa` secret. +``` +kubectl -n cattle-system create secret generic tls-ca \ + --from-file=cacerts.pem=./cacerts.pem +``` +1. Create `tls-rancher-ingress` secret. +``` +kubectl -n cattle-system create secret tls tls-rancher-ingress \ + --cert=tls.crt \ + --key=tls.key +``` +1. Generate Rancher YAML files. +``` +helm template rancher ./rancher-2.6.4.tgz --output-dir . \ + --no-hooks \ + --namespace cattle-system \ + --set hostname=myrancher.local \ + --set rancherImageTag=v2.6.4 \ + --set rancherImage=myregistry.local:5000/rancher/rancher \ + --set systemDefaultRegistry=myregistry.local:5000 \ + --set useBundledSystemChart=true \ + --set ingress.tls.source=secret \ + --set privateCA=true +``` +1. Install Rancher. +``` +kubectl -n cattle-system apply -R -f ./rancher +``` +1. Add `myrancher.local` to `/etc/hosts`. Remember to change `192.168.0.50` to your private ip. +``` +# vim /etc/hosts +192.168.0.50 myregistry.local myrancher.local +``` + +### Integrate Harvester with Rancher in air-gapped environment + +1. (Harvester VM) Add `myregistry.local` to `/etc/hosts`. Remember to change `192.168.0.50` to your private ip. +``` +# vim /etc/hosts +192.168.0.50 myregistry.local +``` +1. (Harvester VM) Add `registries.yaml` to `/etc/rancher/rke2/`. +``` +# vim /etc/rancher/rke2/registries.yaml +mirrors: + docker.io: + endpoint: + - "https://myregistry.local:5000" +configs: + "myregistry.local:5000": + tls: + insecure_skip_verify: true +``` +1. (Harvester VM) Restart RKE2. +``` +systemctl restart rke2-server.service +``` +1. (Harvester VM) Update `rke2-coredns-rke2-coredns` ConfigMap. Remember to change `192.168.0.50` to your private ip. +``` +# replace data like following +data: + Corefile: ".:53 {\n errors \n health {\n lameduck 5s\n }\n ready + \n kubernetes cluster.local cluster.local in-addr.arpa ip6.arpa {\n pods + insecure\n fallthrough in-addr.arpa ip6.arpa\n ttl 30\n }\n prometheus + \ 0.0.0.0:9153\n hosts /etc/coredns/customdomains.db myrancher.local {\n + \ fallthrough\n }\n forward . /etc/resolv.conf\n cache 30\n loop + \n reload \n loadbalance \n}" + customdomains.db: | + 192.168.0.50 myrancher.local +``` +1. (Harvester VM) Update `rke2-coredns-rke2-coredns` Deployment. +``` +# Add customdomains.db to volumes +- key: customdomains.db + path: customdomains.db +``` +![rke2-dns-customdomains.db](./assets/rke2-dns-customdomains.db.png) +1. Follow [Rancher Integration](./rancher/rancher-integration.md) to import Harvester to Rancher. + ## Working Behind an HTTP Proxy In some environments, the connection to external services, from the servers or VMs, requires an HTTP(S) proxy. diff --git a/docs/assets/rke2-dns-customdomains.db.png b/docs/assets/rke2-dns-customdomains.db.png new file mode 100644 index 00000000000..a8f8bd24554 Binary files /dev/null and b/docs/assets/rke2-dns-customdomains.db.png differ