Support Git over HTTPS with creds from env vars

chart/flux/README.md

@@ -37,11 +37,13 @@ until it can find one.
 ## Installation
 
 We put together a simple [Get Started
-tutorial](https://docs.fluxcd.io/en/latest/tutorials/get-started-helm.html) which takes about 5-10 minutes to follow.
+tutorial](https://docs.fluxcd.io/en/stable/tutorials/get-started-helm.html) which takes about 5-10 minutes to follow.
 You will have a fully working Flux installation deploying workloads to your cluster.
 
 ## Installing Flux using Helm
 
+The [configuration](#configuration) section lists all the parameters that can be configured during installation.
+
 ### Installing the Chart
 
 Add the Flux repo:
@@ -50,28 +52,37 @@ Add the Flux repo:
 helm repo add fluxcd https://charts.fluxcd.io
 ```
 
-#### To install the chart with the release name `flux`
+#### Install the chart with the release name `flux`
 
-Replace `fluxcd/flux-get-started` with your own git repository and run helm install:
+1. Replace `fluxcd/flux-get-started` with your own git repository and run helm install:
 
-```sh
-$ helm install --name flux \
---set [email protected]:fluxcd/flux-get-started \
---namespace flux \
-fluxcd/flux
-```
+   ```sh
+   helm install --name flux \
+   --set [email protected]:fluxcd/flux-get-started \
+   --namespace flux \
+   fluxcd/flux
+   ```
 
-#### To connect Flux to a Weave Cloud instance:
+1. Setup Git deploy
 
-```sh
-helm install --name flux \
---set [email protected]:fluxcd/flux-get-started \
---set token=YOUR_WEAVE_CLOUD_SERVICE_TOKEN \
---namespace flux \
-fluxcd/flux
-```
+   > **Note:** this not required when [using git over HTTPS](#flux-with-git-over-https).
+
+   At startup Flux generates a SSH key and logs the public key. Find the
+   SSH public key by installing [fluxctl](https://docs.fluxcd.io/en/stable/references/fluxctl.html)
+   and running:
 
-#### To install Flux with the Helm operator:
+   ```sh
+   fluxctl identity --k8s-fwd-ns flux
+   ```
+
+   In order to sync your cluster state with GitHub you need to copy the
+   public key and create a deploy key with access on your GitHub
+   repository.  Go to _Settings > Deploy keys_ click on _Add deploy key_,
+   paste the Flux public key and click _Add key_. If you want Flux to
+   have write access to your repo, check _Allow write access_; if you
+   have set `git.readonly=true`, you can leave this box unchecked.
+
+#### Install Flux with the Helm operator
 
 Apply the Helm Release CRD:
 
@@ -82,15 +93,47 @@ kubectl apply -f https://raw.githubusercontent.com/fluxcd/flux/helm-0.10.1/deplo
 Install Flux with Helm:
 
 ```sh
-$ helm install --name flux \
+helm install --name flux \
 --set [email protected]:fluxcd/flux-get-started \
 --set helmOperator.create=true \
 --set helmOperator.createCRD=false \
 --namespace flux \
 fluxcd/flux
 ```
 
-#### To install Flux with a private git host:
+#### Flux with git over HTTPS
+
+By setting the `env.secretName`, all key/value pairs in this secret will
+be defined in the Flux container as environment variables. This can be
+utilized in combination with Kubernetes feature of [using environment
+variables inside of your config](https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#using-environment-variables-inside-of-your-config)
+to securely provide the HTTPS credentials which then can be used in the
+`git.url`.
+
+1. Create a personal access token to be used as the `GIT_AUTHKEY`:
+
+   - [GitHub](https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line)
+   - [GitLab](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#creating-a-personal-access-token)
+   - [BitBucket](https://confluence.atlassian.com/bitbucketserver/personal-access-tokens-939515499.html)
+
+1. Create a secret with your `GIT_AUTHUSER` (the username the token belongs
+   to) and the `GIT_AUTHKEY` you created in the first step:
+
+   ```sh
+   kubectl create secret generic flux-git-auth --from-literal=GIT_AUTHUSER=<username> --from-literal=GIT_AUTHKEY=<token>
+   ```
+
+1. Install Flux:
+
+   ```sh
+   helm install --name flux \
+   --set git.url='https://$(GIT_AUTHUSER):$(GIT_AUTHKEY)@github.com:fluxcd/flux-get-started.git' \
+   --set env.secretName=flux-git-auth \
+   --namespace flux \
+   fluxcd/flux
+   ```
+
+#### Flux with a private git host
 
 When using a private git host, setting the `ssh.known_hosts` variable
 is required for enabling successful key matches because `StrictHostKeyChecking`
@@ -100,73 +143,61 @@ By setting the `ssh.known_hosts` variable, a configmap will be created
 called `flux-ssh-config` which in turn will be mounted into a volume named
 `sshdir` at `/root/.ssh/known_hosts`.
 
-* Get the `ssh.known_hosts` keys by running the following command:
+1. Get the `ssh.known_hosts` keys by running the following command:
 
-```sh
-ssh-keyscan <your_git_host_domain>
-```
+   ```sh
+   ssh-keyscan <your_git_host_domain>
+   ```
 
-To prevent a potential man-in-the-middle attack, one should
-verify the ssh keys acquired through the `ssh-keyscan` match expectations
-using an alternate mechanism.
-
-* Start Flux and Flux helm operator:
-
-  - Using a string for setting `known_hosts`
-
-    ```sh
-    YOUR_GIT_HOST=your_git_host.example.com
-    YOUR_GIT_USER=your_git_user
-    KNOWN_HOSTS='domain ssh-rsa line1
-    domain ecdsa-sha2-line2
-    domain ssh-ed25519 line3'
-
-    helm install \
-    --name flux \
-    --set helmOperator.create=true \
-    --set helmOperator.createCRD=false \
-    --set git.url="git@${YOUR_GIT_HOST}:${YOUR_GIT_USER}/flux-get-started" \
-    --set-string ssh.known_hosts="${KNOWN_HOSTS}" \
-    --namespace flux \
-    chart/flux
-    ```
-
-  - Using a file for setting `known_hosts`
-
-    Copy known_hosts keys into a temporary file `/tmp/flux_known_hosts`
-
-    ```sh
-    YOUR_GIT_HOST=your_git_host.example.com
-    YOUR_GIT_USER=your_git_user
-
-    helm install \
-    --name flux \
-    --set helmOperator.create=true \
-    --set helmOperator.createCRD=false \
-    --set git.url="git@${YOUR_GIT_HOST}:${YOUR_GIT_USER}/flux-get-started" \
-    --set-file ssh.known_hosts=/tmp/flux_known_hosts \
-    --namespace flux \
-    chart/flux
-    ```
+   To prevent a potential man-in-the-middle attack, one should
+   verify the ssh keys acquired through the `ssh-keyscan` match expectations
+   using an alternate mechanism.
 
-The [configuration](#configuration) section lists all the parameters that can be configured during installation.
+1. Install Flux:
+
+   - Using a string for setting `known_hosts`
 
-#### Setup Git deploy
+     ```sh
+     YOUR_GIT_HOST=your_git_host.example.com
+     YOUR_GIT_USER=your_git_user
+     KNOWN_HOSTS='domain ssh-rsa line1
+     domain ecdsa-sha2-line2
+     domain ssh-ed25519 line3'
 
-At startup Flux generates a SSH key and logs the public key.
-Find the SSH public key by installing [fluxctl](https://docs.fluxcd.io/en/latest/references/fluxctl.html) and
-running:
+     helm install \
+     --name flux \
+     --set git.url="git@${YOUR_GIT_HOST}:${YOUR_GIT_USER}/flux-get-started" \
+     --set-string ssh.known_hosts="${KNOWN_HOSTS}" \
+     --namespace flux \
+     chart/flux
+     ```
+
+   - Using a file for setting `known_hosts`
+
+     Copy `known_hosts` keys into a temporary file `/tmp/flux_known_hosts`
+
+     ```sh
+     YOUR_GIT_HOST=your_git_host.example.com
+     YOUR_GIT_USER=your_git_user
+
+     helm install \
+     --name flux \
+     --set git.url="git@${YOUR_GIT_HOST}:${YOUR_GIT_USER}/flux-get-started" \
+     --set-file ssh.known_hosts=/tmp/flux_known_hosts \
+     --namespace flux \
+     chart/flux
+     ```
+
+#### Connect Flux to a Weave Cloud instance
 
 ```sh
-fluxctl identity --k8s-fwd-ns flux
+helm install --name flux \
+--set [email protected]:fluxcd/flux-get-started \
+--set token=YOUR_WEAVE_CLOUD_SERVICE_TOKEN \
+--namespace flux \
+fluxcd/flux
 ```
 
-In order to sync your cluster state with GitHub you need to copy the
-public key and create a deploy key with access on your GitHub
-repository.  Go to _Settings > Deploy keys_ click on _Add deploy key_,
-paste the Flux public key and click _Add key_. If you want Flux to
-have write access to your repo, check _Allow write access_; if you
-have set `git.readonly=true`, you can leave this box unchecked.
 
 ### Uninstalling the Chart
 
@@ -203,6 +234,7 @@ The following tables lists the configurable parameters of the Flux chart and the
 | `dnsConfig`                                       | ``                                                   | Pod DNS config
 | `token`                                           | `None`                                               | Weave Cloud service token
 | `extraEnvs`                                       | `[]`                                                 | Extra environment variables for the Flux pod(s)
+| `env.secretName`                                  | ``                                                   | Name of the secret that contains environment variables which should be defined in the Flux container (using `envFrom`)
 | `rbac.create`                                     | `true`                                               | If `true`, create and use RBAC resources
 | `rbac.pspEnabled`                                 | `false`                                              | If `true`, create and use a restricted pod security policy for Flux pod(s)
 | `serviceAccount.create`                           | `true`                                               | If `true`, create a new service account
@@ -304,7 +336,7 @@ The following tables lists the configurable parameters of the Flux chart and the
 Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example:
 
 ```sh
-$ helm upgrade --install --wait flux \
+helm upgrade --install --wait flux \
 --set [email protected]:stefanprodan/k8s-podinfo \
 --set git.path="deploy/auto-scaling\,deploy/local-storage" \
 --namespace flux \

chart/flux/templates/deployment.yaml

@@ -168,6 +168,11 @@ spec:
           {{- if .Values.extraEnvs }}
 {{ toYaml .Values.extraEnvs | indent 10 }}
           {{- end }}
+          {{- if .Values.env.secretName }}
+          envFrom:
+          - secretRef:
+              name: {{ .Values.env.secretName }}
+          {{- end }}
           args:
           {{- if not .Values.clusterRole.create }}
           - --k8s-allow-namespace={{ .Release.Namespace }}

chart/flux/values.yaml

@@ -176,11 +176,13 @@ git:
   pollInterval: "5m"
   # Duration after which git operations time out
   timeout: "20s"
-  # generate a SSH key named identity: ssh-keygen -q -N "" -f ./identity
-  # create a Kubernetes secret: kubectl -n flux create secret generic flux-ssh --from-file=./identity
-  # delete the private key: rm ./identity
-  # add ./identity.pub as a deployment key with write access in your Git repo
-  # set the secret name (flux-ssh) below
+  # The secret name can be used to supply your own SSH key, instead of
+  # relying on Flux to generate one for you:
+  # 1. Generate a SSH key named identity: ssh-keygen -q -N "" -f ./identity
+  # 2. Create a Kubernetes secret: kubectl -n flux create secret generic flux-ssh --from-file=./identity
+  # 3. Delete the private key: rm ./identity
+  # 4. Add ./identity.pub as a deployment key with write access in your Git repo
+  # 5. Set the secret name (flux-ssh) below
   secretName: ""
   # Global Git configuration See https://git-scm.com/docs/git-config for more details.
   config:
@@ -275,6 +277,12 @@ kube:
 # additionalArgs:
 # - --connect=ws://fluxcloud
 
+# The contents of the secret will be defined as environment variables
+# in the Flux container. Once defined, you can use the variables in your
+# `git.url`: `https://$(GIT_AUTHUSER):$(GIT_AUTHKEY)@github.com/fluxcd/flux-get-started.git`
+env:
+  secretName: ""
+
 # Additional environment variables to set
 extraEnvs: []
 # extraEnvs:

cmd/fluxd/main.go

@@ -6,6 +6,7 @@ import (
 	"io/ioutil"
 	"net/http"
 	_ "net/http/pprof"
+	"net/url"
 	"os"
 	"os/exec"
 	"os/signal"
@@ -319,7 +320,13 @@ func main() {
 		}
 	}
 
-	if *sshKeygenDir == "" {
+	// Used to determine if we need to generate a SSH key and setup a keyring
+	var httpGitURL bool
+	if pURL, err := url.Parse(*gitURL); err == nil {
+		httpGitURL = pURL.Scheme == "http" || pURL.Scheme == "https"
+	}
+
+	if *sshKeygenDir == "" && !httpGitURL {
 		logger.Log("info", fmt.Sprintf("SSH keygen dir (--ssh-keygen-dir) not provided, so using the deploy key volume (--k8s-secret-volume-mount-path=%s); this may cause problems if the deploy key volume is mounted read-only", *k8sSecretVolumeMountPath))
 		*sshKeygenDir = *k8sSecretVolumeMountPath
 	}
@@ -430,7 +437,7 @@ func main() {
 		}
 		clusterVersion = "kubernetes-" + serverVersion.GitVersion
 
-		if *k8sInCluster {
+		if *k8sInCluster && !httpGitURL {
 			namespace, err := ioutil.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/namespace")
 			if err != nil {
 				logger.Log("err", err)
@@ -621,7 +628,7 @@ func main() {
 	}
 
 	logger.Log(
-		"url", *gitURL,
+		"url", gitRemote.SafeURL(),
 		"user", *gitUser,
 		"email", *gitEmail,
 		"signing-key", *gitSigningKey,

deploy/flux-deployment.yaml

@@ -115,6 +115,13 @@ spec:
         #   mountPath: /root/gpg-import
         #   readOnly: true
 
+        # Include this if you want to supply HTTP basic auth credentials for git
+        # via the `GIT_AUTHUSER` and `GIT_AUTHKEY` environment variables using a
+        # secret.
+        # envFrom:
+        # - secretRef:
+        #     name: flux-git-auth
+
         args:
 
         # If you deployed memcached in a different namespace to flux,
@@ -131,9 +138,11 @@ spec:
         - --ssh-keygen-dir=/var/fluxd/keygen
 
         # Replace the following URL to change the Git repository used by Flux.
+        # HTTP basic auth credentials can be supplied using environment variables:
+        # https://$(GIT_AUTHUSER):$(GIT_AUTHKEY)@github.com/user/repository.git
         - [email protected]:fluxcd/flux-get-started
         - --git-branch=master
-        # include this if you want to restrict the manifests considered by flux
+        # Include this if you want to restrict the manifests considered by flux
         # to those under the following relative paths in the git repository
         # - --git-path=subdir1,subdir2
 

docs/guides/index.rst

@@ -8,5 +8,6 @@ Guides
    :maxdepth: 1
 
    provide-own-ssh-key
+   use-git-over-https
    use-private-git-host
    upgrading-to-1.0