Updated airgap.md with steps to setup rancher and harvester in air-gapped env

docs/airgap.md

@@ -13,6 +13,369 @@ This section describes how to use Harvester in an air gapped environment. Some u
 
 The Harvester ISO image contains all the packages to make it work in an air gapped environment.
 
+## Working without HTTP Proxy
+
+In this example, we use KVM to provision Harvester and integrate it with Rancher.
+
+### Setup offline Harvester
+
+1. Clone [harvester/ipxe-examples](https://github.com/harvester/ipxe-examples).
+1. Select the `vagrant-pxe-harvester` folder.
+1. Edit [settings.yml](https://github.com/harvester/ipxe-examples/blob/c8267b6270660255bf71149a1fef3a7a914550de/vagrant-pxe-harvester/settings.yml#L44) and set `offline: true`.
+1. Execute the [setup_harvester.sh](https://github.com/harvester/ipxe-examples/blob/main/vagrant-pxe-harvester/setup_harvester.sh) script.
+
+### Deploy Rancher on K3s and setup private registry in another VM
+
+1. Create another KVM with two network interfaces, `harvester` and `vagrant-libvirt`.
+
+    - `harvester` is for intranet and `vagrant-libvirt` is for internet. 
+    - We need `vagrant-libvirt` to download all required resources. We will remove it before we start Rancher. 
+    - Configure this VM with at least 300GB to save all required images.
+
+1. Install [Docker](https://www.docker.com/) and [Helm](https://helm.sh/).
+1. Create a `certs` folder.
+
+    ```
+    mkdir -p certs
+    ```
+
+1. Generate private registry certificate files.
+
+    ```
+    openssl req \
+    -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key \
+    -addext "subjectAltName = DNS:myregistry.local" \
+    -x509 -days 365 -out certs/domain.crt
+    ```
+
+1. Move certificate files to `/etc/docker/certs.d`.
+
+    ```
+    sudo mkdir -p /etc/docker/certs.d/myregistry.local:5000
+    sudo cp certs/domain.crt /etc/docker/certs.d/myregistry.local:5000/domain.crt
+    ```
+
+1. Start the private registry.
+
+    ```
+    docker run -d \
+    -p 5000:5000 \
+    --restart=always \
+    --name registry \
+    -v "$(pwd)"/certs:/certs \
+    -v "$(pwd)"/registry:/var/lib/registry \
+    -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
+    -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
+    registry:2
+    ```
+
+1. Add `myregistry.local` to `/etc/hosts`. Remember to change `192.168.0.50` to your private IP.
+
+    ```
+    # vim /etc/hosts
+    192.168.0.50 myregistry.local
+    ```
+
+1. Create a `get-rancher` script.
+
+    ```
+    # vim get-rancher
+    #!/bin/bash
+    if [[ $# -eq 0 ]] ; then
+        echo 'This requires you to pass a version for the url like "v2.6.4"'
+        exit 1
+    fi
+    wget https://github.com/rancher/rancher/releases/download/$1/rancher-images.txt
+    wget https://github.com/rancher/rancher/releases/download/$1/rancher-load-images.sh
+    wget https://github.com/rancher/rancher/releases/download/$1/rancher-save-images.sh
+    chmod +x ./rancher-save-images.sh
+    chmod +x ./rancher-load-images.sh
+    ```
+
+1. Make `get-rancher` script be excutable.
+
+    ```
+    chmod +x get-rancher
+    ```
+
+1. Download `rancher-images.txt`, `rancher-load-images.sh` and `rancher-save-images.sh`.
+
+    ```
+    ./get-rancher v2.6.4
+    ```
+
+1. Add cert-manager images to `rancher-images.txt`.
+
+    ```
+    helm repo add jetstack https://charts.jetstack.io/
+    helm repo update
+    helm fetch jetstack/cert-manager --version v1.7.1
+    helm template ./cert-manager-v1.7.1.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt
+    ```
+
+1. Sort `rancher-images.txt`.
+
+    ```
+    sort -u rancher-images.txt -o rancher-images.txt
+    ```
+
+1. Get images. This step may take 1 to 2 hours depending on your network speed.
+
+    ```
+    ./rancher-save-images.sh --image-list ./rancher-images.txt
+    ```
+
+1. Load images to local registry.
+
+    ```
+    ./rancher-load-images.sh --image-list ./rancher-images.txt --registry myregistry.local:5000
+    ```
+
+1. Create a `get-k3s` script.
+
+    ```
+    # vim get-k3s
+    #!/bin/bash
+    if [[ $# -eq 0 ]] ; then
+        echo 'This requires you to pass a version for the url like "v1.23.4+k3s1"'
+        exit 1
+    fi
+    wget https://github.com/k3s-io/k3s/releases/download/$1/k3s-airgap-images-amd64.tar
+    wget https://github.com/k3s-io/k3s/releases/download/$1/k3s
+    wget https://get.k3s.io/ -O install.sh
+    chmod +x ./k3s
+    chmod +x ./install.sh
+    ```
+
+1. Make `get-k3s` excuteable.
+
+    ```
+    chmod +x get-k3s
+    ```
+
+1. Download `k3s-airgap-images-amd64.tar`, `k3s` and `install.sh`.
+
+    ```
+    ./get-k3s v1.23.4+k3s1
+    ```
+
+1. Download Rancher.
+
+    ```
+        helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
+        helm fetch rancher-latest/rancher --version=v2.6.4
+    ```
+
+1. Download `cert-manager-crds.yaml`.
+
+    ```
+    mkdir cert-manager
+    curl -L -o cert-manager/cert-manager-crd.yaml https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml
+    ```
+
+1. Remove the `vagrant-libvirt` network interface. Once the required resources download, you can remove the network.
+
+1. Move `k3s-airgap-images-amd64.tar` to `/var/lib/rancher/k3s/agent/images/`.
+
+    ```
+    sudo mkdir -p /var/lib/rancher/k3s/agent/images/
+    sudo cp k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images/
+    ```
+
+1. Create a `/etc/rancher/k3s` folder.
+
+    ```
+    mkdir -p /etc/rancher/k3s
+    ```
+
+1. Add `registries.yaml` to `/etc/rancher/k3s`.
+
+    ```
+    # vim /etc/rancher/k3s/registries.yaml
+    mirrors:
+    docker.io:
+        endpoint:
+        - "https://myregistry.local:5000/"
+    configs:
+    "myregistry.local:5000":
+        tls:
+        insecure_skip_verify: true
+    ```
+
+1. Install K3s.
+
+    ```
+    INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh
+    ```
+
+1. Generate cert-manager YAML files.
+
+    ```
+    helm template cert-manager ./cert-manager-v1.7.1.tgz --output-dir . \
+        --namespace cert-manager \
+        --set image.repository=myregistry.local:5000/quay.io/jetstack/cert-manager-controller \
+        --set webhook.image.repository=myregistry.local:5000/quay.io/jetstack/cert-manager-webhook \
+        --set cainjector.image.repository=myregistry.local:5000/quay.io/jetstack/cert-manager-cainjector \
+        --set startupapicheck.image.repository=myregistry.local:5000/quay.io/jetstack/cert-manager-ctl
+    ```
+
+1. Move `/etc/rancher/k3s/k3s.yaml` to `~/.kube`.
+
+    ```
+    mkdir ~/.kube
+    sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
+    sudo chown $USER ~/.kube/config
+    export KUBECONFIG=~/.kube/config
+    ```
+
+1. Install cert-manager.
+
+    ```
+    kubectl create namespace cert-manager
+    kubectl apply -f cert-manager/cert-manager-crd.yaml
+    kubectl apply -R -f ./cert-manager
+    ```
+
+1. Create a CA private key and a certificate file.
+
+    ```
+    openssl genrsa -out cakey.pem 2048
+    openssl req -x509 -sha256 -new -nodes -key cakey.pem -days 3650 -out cacerts.pem -subj "/CN=cattle-ca"
+    ```
+
+1. Create `openssl.cnf`. Remember to change `192.168.0.50` to your private IP.
+
+    ```
+    [req]
+    req_extensions = v3_req
+    distinguished_name = req_distinguished_name
+    [req_distinguished_name]
+    [ v3_req ]
+    basicConstraints = CA:FALSE
+    keyUsage = nonRepudiation, digitalSignature, keyEncipherment
+    extendedKeyUsage = clientAuth, serverAuth
+    subjectAltName = @alt_names
+    [alt_names]
+    DNS.1 = myrancher.local
+    IP.1 = 192.168.0.50
+    ```
+
+1. Generate s private key and a certificate file for `myrancher.local`.
+
+    ```
+    openssl genrsa -out tls.key 2048
+    openssl req -sha256 -new -key tls.key -out tls.csr -subj "/CN=myrancher.local" -config openssl.cnf
+    openssl x509 -sha256 -req -in tls.csr -CA cacerts.pem \
+        -CAkey cakey.pem -CAcreateserial -out tls.crt \
+        -days 3650 -extensions v3_req \
+        -extfile openssl.cnf
+    ```
+
+1. Create a `cattle-system` namespace.
+
+    ```
+    kubectl create ns cattle-system
+    ```
+
+1. Create a `tls.sa` secret.
+
+    ```
+    kubectl -n cattle-system create secret generic tls-ca \
+    --from-file=cacerts.pem=./cacerts.pem
+    ```
+
+1. Create a `tls-rancher-ingress` secret.
+
+    ```
+    kubectl -n cattle-system create secret tls tls-rancher-ingress \
+    --cert=tls.crt \
+    --key=tls.key
+    ```
+
+1. Generate Rancher YAML files.
+
+    ```
+    helm template rancher ./rancher-2.6.4.tgz --output-dir . \
+        --no-hooks \
+        --namespace cattle-system \
+        --set hostname=myrancher.local \
+        --set rancherImageTag=v2.6.4 \
+        --set rancherImage=myregistry.local:5000/rancher/rancher \
+        --set systemDefaultRegistry=myregistry.local:5000 \
+        --set useBundledSystemChart=true \
+        --set ingress.tls.source=secret \
+        --set privateCA=true
+    ```
+
+1. Install Rancher.
+
+    ```
+    kubectl -n cattle-system apply -R -f ./rancher
+    ```
+
+1. Add `myrancher.local` to `/etc/hosts`. Remember to change `192.168.0.50` to your private IP.
+
+    ```
+    # vim /etc/hosts
+    192.168.0.50 myregistry.local myrancher.local
+    ```
+
+### Integrate Harvester with Rancher in air-gapped environment
+
+1. (Harvester VM) Add `myregistry.local` to `/etc/hosts`. Remember to change `192.168.0.50` to your private IP.
+
+    ```
+    # vim /etc/hosts
+    192.168.0.50 myregistry.local
+    ```
+
+1. (Harvester VM) Add `registries.yaml` to `/etc/rancher/rke2/`.
+
+    ```
+    # vim /etc/rancher/rke2/registries.yaml
+    mirrors:
+    docker.io:
+        endpoint:
+        - "https://myregistry.local:5000"
+    configs:
+    "myregistry.local:5000":
+        tls:
+        insecure_skip_verify: true
+    ```
+
+1. (Harvester VM) Restart RKE2.
+
+    ```
+    systemctl restart rke2-server.service
+    ```
+
+1. (Harvester VM) Update the `rke2-coredns-rke2-coredns` ConfigMap. Remember to change `192.168.0.50` to your private IP.
+
+    ```
+    # replace data like following
+    data:
+    Corefile: ".:53 {\n    errors \n    health  {\n        lameduck 5s\n    }\n    ready
+        \n    kubernetes   cluster.local  cluster.local in-addr.arpa ip6.arpa {\n        pods
+        insecure\n        fallthrough in-addr.arpa ip6.arpa\n        ttl 30\n    }\n    prometheus
+        \  0.0.0.0:9153\n   hosts /etc/coredns/customdomains.db myrancher.local {\n
+        \   fallthrough\n    }\n forward   . /etc/resolv.conf\n    cache   30\n    loop
+        \n    reload \n    loadbalance \n}"
+    customdomains.db: |
+        192.168.0.50 myrancher.local
+    ```
+
+1. (Harvester VM) Update the `rke2-coredns-rke2-coredns` deployment.
+
+    ```
+    # Add customdomains.db to volumes
+    - key: customdomains.db
+    path: customdomains.db
+    ```
+
+    ![rke2-dns-customdomains.db](/img/v1.2/rke2-dns-customdomains.db.png)
+
+1. Follow [Rancher Integration](./rancher/rancher-integration.md) to import Harvester to Rancher.
+
 ## Working Behind an HTTP Proxy
 
 In some environments, the connection to external services, from the servers or VMs, requires an HTTP(S) proxy.
@@ -31,16 +394,16 @@ You can configure the HTTP(S) proxy in the settings page of the Harvester dashbo
 1. Find the `http-proxy` setting, click **⋮ > Edit setting**
 1. Enter the value(s) for `http-proxy`, `https-proxy` and `no-proxy`.
 
-![proxy-setting](/img/v1.2/proxy-setting.png)
+    ![proxy-setting](/img/v1.2/proxy-setting.png)
 
-:::note
+    :::note
 
-Harvester appends necessary addresses to user configured `no-proxy` to ensure the internal traffic works.
-i.e., `localhost,127.0.0.1,0.0.0.0,10.0.0.0/8,longhorn-system,cattle-system,cattle-system.svc,harvester-system,.svc,.cluster.local`. `harvester-system` was added into the list since v1.1.2.
+    Harvester appends necessary addresses to user configured `no-proxy` to ensure the internal traffic works.
+    i.e., `localhost,127.0.0.1,0.0.0.0,10.0.0.0/8,longhorn-system,cattle-system,cattle-system.svc,harvester-system,.svc,.cluster.local`. `harvester-system` was added into the list since v1.1.2.
 
-When the nodes in the cluster do not use a proxy to communicate with each other, the CIDR needs to be added to `http-proxy.noProxy` after the first node is installed successfully. Please refer to [fail to deploy a multi-node cluster](./troubleshooting/harvester.md#fail-to-deploy-a-multi-node-cluster-due-to-incorrect-http-proxy-setting).
+    When the nodes in the cluster do not use a proxy to communicate with each other, the CIDR needs to be added to `http-proxy.noProxy` after the first node is installed successfully. Please refer to [fail to deploy a multi-node cluster](./troubleshooting/harvester.md#fail-to-deploy-a-multi-node-cluster-due-to-incorrect-http-proxy-setting).
 
-:::
+    :::
 
 ## Guest Cluster Images