Add setup rancher and harvester in air-gapped environment

docs/airgap.md

@@ -12,6 +12,287 @@ This section describes how to use Harvester in an air gapped environment. Some u
 
 The Harvester ISO image contains all the packages to make it work in an air gapped environment.
 
+## Working without HTTP Proxy
+
+In this example, we use KVM to provision Harvester and integrate it with Rancher.
+
+### Setup offline Harvester
+
+1. Clone [harvester/ipxe-examples](https://github.com/harvester/ipxe-examples).
+1. Go to `vagrant-pxe-harvester` folder.
+1. Edit [settings.yml](https://github.com/harvester/ipxe-examples/blob/c8267b6270660255bf71149a1fef3a7a914550de/vagrant-pxe-harvester/settings.yml#L44) and set `offline: true`.
+1. Execute [setup_harvester.sh](https://github.com/harvester/ipxe-examples/blob/main/vagrant-pxe-harvester/setup_harvester.sh) script.
+
+### Deploy Rancher on k3s and setup private registry in another VM.
+
+1. Create another KVM with two network interfaces `harvester` and `vagrant-libvirt`. `harvester` network interface is for intranet and `vagrant-libvirt` is for internet. We need `vagrant-libvirt` to download all required resources. We will remove it before we start Rancher. Also, please give this VM at least 300GB to save all required images.
+1. Install [Docker](https://www.docker.com/) and [Helm](https://helm.sh/).
+1. Create `certs` folder.
+```
+mkdir -p certs
+```
+1. Generate private registry certificate files.
+```
+openssl req \
+  -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key \
+  -addext "subjectAltName = DNS:myregistry.local" \
+  -x509 -days 365 -out certs/domain.crt
+```
+1. Move certificate files to `/etc/docker/certs.d`.
+```
+sudo mkdir -p /etc/docker/certs.d/myregistry.local:5000
+sudo cp certs/domain.crt /etc/docker/certs.d/myregistry.local:5000/domain.crt
+```
+1. Start private registry.
+```
+docker run -d \
+  -p 5000:5000 \
+  --restart=always \
+  --name registry \
+  -v "$(pwd)"/certs:/certs \
+  -v "$(pwd)"/registry:/var/lib/registry \
+  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
+  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
+  registry:2
+```
+1. Add `myregistry.local` to `/etc/hosts`. Remember to change `192.168.0.50` to your private ip.
+```
+# vim /etc/hosts
+192.168.0.50 myregistry.local
+```
+1. Create `get-rancher` script.
+```
+# vim get-rancher
+#!/bin/bash
+if [[ $# -eq 0 ]] ; then
+    echo 'This requires you to pass a version for the url like "v2.6.4"'
+    exit 1
+fi
+wget https://github.com/rancher/rancher/releases/download/$1/rancher-images.txt
+wget https://github.com/rancher/rancher/releases/download/$1/rancher-load-images.sh
+wget https://github.com/rancher/rancher/releases/download/$1/rancher-save-images.sh
+chmod +x ./rancher-save-images.sh
+chmod +x ./rancher-load-images.sh
+```
+1. Make `get-rancher` script be excutable.
+```
+chmod +x get-rancher
+```
+1. Download `rancher-images.txt`, `rancher-load-images.sh` and `rancher-save-images.sh`.
+```
+./get-rancher v2.6.4
+```
+1. Add cert-manager images to `rancher-images.txt`.
+```
+helm repo add jetstack https://charts.jetstack.io/
+helm repo update
+helm fetch jetstack/cert-manager --version v1.7.1
+helm template ./cert-manager-v1.7.1.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt
+```
+1. Sort `rancher-images.txt`.
+```
+sort -u rancher-images.txt -o rancher-images.txt
+```
+1. Get images. This step may take 1 to 2 hours. It depends on your network speed.
+```
+./rancher-save-images.sh --image-list ./rancher-images.txt
+```
+1. Load images to local registry.
+```
+./rancher-load-images.sh --image-list ./rancher-images.txt --registry myregistry.local:5000
+```
+1. Create `get-k3s` script.
+```
+# vim get-k3s
+#!/bin/bash
+if [[ $# -eq 0 ]] ; then
+    echo 'This requires you to pass a version for the url like "v1.23.4+k3s1"'
+    exit 1
+fi
+wget https://github.com/k3s-io/k3s/releases/download/$1/k3s-airgap-images-amd64.tar
+wget https://github.com/k3s-io/k3s/releases/download/$1/k3s
+wget https://get.k3s.io/ -O install.sh
+chmod +x ./k3s
+chmod +x ./install.sh
+```
+1. Make `get-k3s` be excutable.
+```
+chmod +x get-k3s
+```
+1. Download `k3s-airgap-images-amd64.tar`, `k3s` and `install.sh`.
+```
+./get-k3s v1.23.4+k3s1
+```
+1. Download Rancher.
+```
+helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
+helm fetch rancher-latest/rancher --version=v2.6.4
+```
+1. Download `cert-manager-crds.yaml`.
+```
+mkdir cert-manager
+curl -L -o cert-manager/cert-manager-crd.yaml https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml
+```
+1. Remove `vagrant-libvirt` network interface. We've downloaded all required resources. We can cut network from this step.
+1. Move `k3s-airgap-images-amd64.tar` to `/var/lib/rancher/k3s/agent/images/`.
+```
+sudo mkdir -p /var/lib/rancher/k3s/agent/images/
+sudo cp k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images/
+```
+1. Create `/etc/rancher/k3s` folder.
+```
+mkdir -p /etc/rancher/k3s
+```
+1. Add `registries.yaml` to `/etc/rancher/k3s`.
+```
+# vim /etc/rancher/k3s/registries.yaml
+mirrors:
+  docker.io:
+    endpoint:
+      - "https://myregistry.local:5000/"
+configs:
+  "myregistry.local:5000":
+    tls:
+      insecure_skip_verify: true
+```
+1. Install k3s.
+```
+INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh
+```
+1. Generate cert-manager YAML files.
+```
+helm template cert-manager ./cert-manager-v1.7.1.tgz --output-dir . \
+    --namespace cert-manager \
+    --set image.repository=myregistry.local:5000/quay.io/jetstack/cert-manager-controller \
+    --set webhook.image.repository=myregistry.local:5000/quay.io/jetstack/cert-manager-webhook \
+    --set cainjector.image.repository=myregistry.local:5000/quay.io/jetstack/cert-manager-cainjector \
+    --set startupapicheck.image.repository=myregistry.local:5000/quay.io/jetstack/cert-manager-ctl
+```
+1. Move `/etc/rancher/k3s/k3s.yaml` to `~/.kube`.
+```
+mkdir ~/.kube
+sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
+sudo chown $USER ~/.kube/config
+export KUBECONFIG=~/.kube/config
+```
+1. Install cert-manager.
+```
+kubectl create namespace cert-manager
+kubectl apply -f cert-manager/cert-manager-crd.yaml
+kubectl apply -R -f ./cert-manager
+```
+1. Create CA private key and certifacate file.
+```
+openssl genrsa -out cakey.pem 2048
+openssl req -x509 -sha256 -new -nodes -key cakey.pem -days 3650 -out cacerts.pem -subj "/CN=cattle-ca"
+```
+1. Create `openssl.cnf`. Remember to change `192.168.0.50` to your private ip.
+```
+[req]
+req_extensions = v3_req
+distinguished_name = req_distinguished_name
+[req_distinguished_name]
+[ v3_req ]
+basicConstraints = CA:FALSE
+keyUsage = nonRepudiation, digitalSignature, keyEncipherment
+extendedKeyUsage = clientAuth, serverAuth
+subjectAltName = @alt_names
+[alt_names]
+DNS.1 = myrancher.local
+IP.1 = 192.168.0.50
+```
+1. Generate private key and certificate file for `myrancher.local`.
+```
+openssl genrsa -out tls.key 2048
+openssl req -sha256 -new -key tls.key -out tls.csr -subj "/CN=myrancher.local" -config openssl.cnf
+openssl x509 -sha256 -req -in tls.csr -CA cacerts.pem \
+    -CAkey cakey.pem -CAcreateserial -out tls.crt \
+    -days 3650 -extensions v3_req \
+    -extfile openssl.cnf
+```
+1. Create `cattle-system` namespace.
+```
+kubectl create ns cattle-system
+```
+1. Create `tls.sa` secret.
+```
+kubectl -n cattle-system create secret generic tls-ca \
+  --from-file=cacerts.pem=./cacerts.pem
+```
+1. Create `tls-rancher-ingress` secret.
+```
+kubectl -n cattle-system create secret tls tls-rancher-ingress \
+  --cert=tls.crt \
+  --key=tls.key
+```
+1. Generate Rancher YAML files.
+```
+helm template rancher ./rancher-2.6.4.tgz --output-dir . \
+    --no-hooks \
+    --namespace cattle-system \
+    --set hostname=myrancher.local \
+    --set rancherImageTag=v2.6.4 \
+    --set rancherImage=myregistry.local:5000/rancher/rancher \
+    --set systemDefaultRegistry=myregistry.local:5000 \
+    --set useBundledSystemChart=true \
+    --set ingress.tls.source=secret \
+    --set privateCA=true
+```
+1. Install Rancher.
+```
+kubectl -n cattle-system apply -R -f ./rancher
+```
+1. Add `myrancher.local` to `/etc/hosts`. Remember to change `192.168.0.50` to your private ip.
+```
+# vim /etc/hosts
+192.168.0.50 myregistry.local myrancher.local
+```
+
+### Integrate Harvester with Rancher in air-gapped environment
+
+1. (Harvester VM) Add `myregistry.local` to `/etc/hosts`. Remember to change `192.168.0.50` to your private ip.
+```
+# vim /etc/hosts
+192.168.0.50 myregistry.local
+```
+1. (Harvester VM) Add `registries.yaml` to `/etc/rancher/rke2/`.
+```
+# vim /etc/rancher/rke2/registries.yaml
+mirrors:
+  docker.io:
+    endpoint:
+      - "https://myregistry.local:5000"
+configs:
+  "myregistry.local:5000":
+    tls:
+      insecure_skip_verify: true
+```
+1. (Harvester VM) Restart RKE2.
+```
+systemctl restart rke2-server.service
+```
+1. (Harvester VM) Update `rke2-coredns-rke2-coredns` ConfigMap. Remember to change `192.168.0.50` to your private ip.
+```
+# replace data like following
+data:
+  Corefile: ".:53 {\n    errors \n    health  {\n        lameduck 5s\n    }\n    ready
+    \n    kubernetes   cluster.local  cluster.local in-addr.arpa ip6.arpa {\n        pods
+    insecure\n        fallthrough in-addr.arpa ip6.arpa\n        ttl 30\n    }\n    prometheus
+    \  0.0.0.0:9153\n   hosts /etc/coredns/customdomains.db myrancher.local {\n
+    \   fallthrough\n    }\n forward   . /etc/resolv.conf\n    cache   30\n    loop
+    \n    reload \n    loadbalance \n}"
+  customdomains.db: |
+    192.168.0.50 myrancher.local
+```
+1. (Harvester VM) Update `rke2-coredns-rke2-coredns` Deployment.
+```
+# Add customdomains.db to volumes
+- key: customdomains.db
+  path: customdomains.db
+```
+![rke2-dns-customdomains.db](./assets/rke2-dns-customdomains.db.png)
+1. Follow [Rancher Integration](./rancher/rancher-integration.md) to import Harvester to Rancher.
+
 ## Working Behind an HTTP Proxy
 
 In some environments, the connection to external services, from the servers or VMs, requires an HTTP(S) proxy.