Need ability to add to known_hosts

[squaremo]

When using e.g., a private gitlab repo, flux will balk at cloning the repo because the host has an unknown identity (i.e., it's not in ~/.ssh/known_hosts).

Currently we prepopulate ~/.ssh/known_hosts with keys for github, gitlab (public), and bitbucket, using ssh-keyscan, when building the flux image.

A few notes/ideas:

• we certainly do not want to switch host verification off, and have to be careful about automatically adding host keys.
• use an init container to ssh-keyscan the private host; but,
  • is the host key stable, if you're running it yourself? Can you make it stable?
• alternative to an init container: an argument to fluxd giving extra hosts to keyscan on startup.

[lukemarsden]

Why don't we want to turn host key verification off? What are the specific attack vectors we're worried about?

How is using an init container any more secure? It just means that you weren't MITM'd at the beginning, rather than later I guess. At least in Kubernetes, I suspect GitLab's key will probably change if it's upgraded or re-deployed, so you'd need a way to refresh it.

Giving fluxd an argument for this (and threading it through the launch generator etc): Is the hit to usability worth the extra security? Flux is already hard enough to configure (and we've gone to great lengths to make it easier, with some success; this seems like a step backwards; I know – security and convenience are often in tension). Couldn't we make the more-secure option an optional default instead for folks concerned about the attack vectors above?

[lukemarsden]

From discussion with Ilya: probably what we need to do here is expose the "do you want to trust this host?" experience somehow through the Flux CLI and GUI.

[errordeveloper]

Also, current approach suffers from an inability to handle the scenario where either of the hosts (whose keys we trust at build time) re-issue their keys.

[errordeveloper]

I suppose it'd make sense to consider bubbling this question to the UI (that's like what OpenSSH client does). May be we could ask the user at the time they set the repo URL. Just an idea.

In the meantime, I'd suggest that perhaps we could trust hosts that are inside of the cluster, i.e. *.cluster.local.

[lukemarsden]

Relevant answer: https://security.stackexchange.com/questions/39990/is-it-safe-to-disable-ssh-host-key-checking-if-key-based-authentication-is-used/40002#40002

I guess the bad thing that could happen is that Flux connects to an attacking server, and the attacking server tells it what software to run in its production cluster (perhaps referring to the attacker's own docker images hosted on a public registry). That would be bad.

[squaremo]

I think optionally trusting (as in, turning off StrictHostKeyChecking for) *.cluster.local could be a reasonable compromise. That would cope with unstable keys, I think.

[squaremo]

A workaround, until we figure out a general solution, is for people to mount a known_hosts file into the fluxd pod using a ConfigMap. One can use ssh-keyscan to prepare one:

$ ssh-keyscan github.com > known_hosts
$ kubectl -n kube-system create configmap flux-known-hosts --from-file=./known_hosts

then adapt the flux daemon manifest to mount it into the expected place:

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: flux
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: flux
    spec:
      serviceAccount: flux
      volumes:
      - name: git-key
        secret:
          secretName: flux-git-deploy
      # vvv Additional volume
      - name: known-hosts
        configMap: flux-known-hosts
      # ^^^
      containers:
      - name: fluxd
        image: quay.io/weaveworks/flux
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 3030
        volumeMounts:
        - name: git-key
          mountPath: /etc/fluxd/ssh
        # vvv Additional volume mount
        - name: known-hosts
          mountPath: /root/.ssh/known_hosts
          subPath: known_hosts
        # ^^^
        args:
        - [email protected]:squaremo/flux-example
        - --git-set-author

(EDITED: changed the mount path to reflect the actual home directory)

(UPDATED: use a subPath so that it doesn't interfere with /root/.ssh/config from the container image)

[RochesterinNYC]

@squaremo I tried this solution (mounting /home/flux/.ssh/known_hosts) and it isn't working for me.

I notice that there's a /root/.ssh/known_hosts file in the container. When I did the manual git clone test inside the container in #899, it added the GHE entry into that /root/.ssh/known_hosts file. Is the /root/.ssh/known_hosts or /home/flux/.ssh/known_hosts used by the flux daemon during git cloning/pulling?

[RochesterinNYC]

On a potentially unrelated note, when I do mount at /root/.ssh/known_hosts instead of /home/flux/.ssh/known_hosts the pod gets stuck in CrashLoopBackOff with the following log messages:

$ kubectl -n company-system logs flux-647d789b-whq25
ts=2018-01-03T18:58:42.355262534Z caller=main.go:195 component=platform identity=/etc/fluxd/ssh/identity
ts=2018-01-03T18:58:42.355416093Z caller=main.go:196 component=platform identity.pub="ssh-rsa <key>"
ts=2018-01-03T18:58:42.355465194Z caller=main.go:197 component=platform host=https://10.71.240.1:443 version=kubernetes-v1.8.4-gke.0
ts=2018-01-03T18:58:42.35554082Z caller=main.go:209 component=platform kubectl=/usr/local/bin/kubectl
ts=2018-01-03T18:58:42.357019183Z caller=main.go:221 component=platform ping=true
ts=2018-01-03T18:58:42.357103512Z caller=main.go:323 upstream="no upstream URL given"
ts=2018-01-03T18:58:42.359290975Z caller=main.go:341 addr=:3030
ts=2018-01-03T18:58:46.452843584Z caller=main.go:389 working-dir=/tmp/flux-gitclone162993532/repo user="Weave Flux" [email protected] sync-tag=flux-sync notes-ref=flux set-author=false
panic: registry.Warmer fields are nil

goroutine 66 [running]:
github.com/weaveworks/flux/registry.(*Warmer).Loop(0xc4201eac60, 0xc4203df5c0, 0xc420371980, 0xc4202c9540)
	/go/src/github.com/weaveworks/flux/registry/warming.go:39 +0x47b
created by main.main
	/go/src/github.com/weaveworks/flux/cmd/fluxd/main.go:423 +0x3f68

I'm trying to use GCR as the registry, could that be related?

[squaremo]

I notice that there's a /root/.ssh/known_hosts file in the container.

Oh of course! $HOME is /root, rather than /home/flux (which is merely the working directory, where we put the fluxd binary). My bad. Yes, I think you'll want /root/.ssh/known_hosts.

On a potentially unrelated note, when I do mount at /root/.ssh/known_hosts instead of /home/flux/.ssh/known_hosts the pod gets stuck in CrashLoopBackOff

Sorry that's my fault -- the example yamel I gave above omits a bunch of arguments that aren't necessary for an image built from master, but are required for the release images (i.e., 1.1.0). If you just grab the two extra bits and put them in your non-CrashLooping config I think it'll go better.

I'm going to do a release of a new image pretty soon -- lots of stuff works much better now :)

I'm trying to use GCR as the registry, could that be related?

That should be OK, or at least not be an immediate problem. We have done a bit of rewriting, recently, to work with GCR better since it dropped support for older Docker image manifests. It'll be in the new release.

[RochesterinNYC]

When trying to change the mount to /root/.ssh/known_hosts, CrashLoopBackOff and the error above about panic: registry.Warmer fields are nil occurs.

This is the new deployment.yaml and the additional ConfigMap I'm deploying:

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: flux
  namespace: company-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: flux
    spec:
      serviceAccount: flux
      volumes:
      - name: git-key
        secret:
          secretName: flux-git-deploy
      - name: known-hosts
        configMap:
          name: flux-known-hosts
      containers:
      - name: flux
        image: quay.io/weaveworks/flux:1.1.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 3030 # informational
        volumeMounts:
        - name: git-key
          mountPath: /etc/fluxd/ssh
        - name: known-hosts
          mountPath: /root/.ssh
        args:
        - [email protected]:jameswen/kubernetes-monorepo-prototype
        - --git-branch=master
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: flux-known-hosts
  namespace: company-system
data:
  known_hosts: |-
    ghe.company.net ssh-rsa <key>
    ghe.company.net <ecdsa-...> <key>

[squaremo]

the example yamel I gave above omits a bunch of arguments that aren't necessary for an image built from master, but are required for the release images (i.e., 1.1.0). If you just grab the two extra bits and put them in your non-CrashLooping config I think it'll go better.

Version 1.2.0 is released now. If you bump the flux image version to 1.2.0, I think that manifest will work.

      containers:
      - name: flux
        image: quay.io/weaveworks/flux:1.2.0 # <---  change here

[RochesterinNYC]

Version 1.2.0 worked for me. Thanks for the help! 👍
Out of curiosity, what was the PR/fix that fixed the issue?

[squaremo]

#851 removed the need for a lot of the command-line arguments (so omitting them didn't lead to crashloops); the other bit was mounting the SSH known_hosts, which was already possible but I hadn't properly thought through before.

If the git syncing is now working for you, I can add it to our documentation -- great!

[squaremo]

Another approach (which could be automatable, even) would be to add the host key in an initContainer. This does carry some of the risks expressed above: that is, it could be vulnerable to man-in-the-middle attacks.

Yet another approach: do an explicit SSH connection check when starting up, and report the failure as part of the git status, so it has a chance of being remedied.

[jondlm]

FWIW I also ran into this issue and worked around it by creating my own flux docker image. Now that I've read through this issue a better solution would definitely be to mount in my own known_hosts file instead of baking it into the image.

[aboyett]

Using a config-map for known-hosts in the form above did not work since it specifies a volume mountPath of /root/.ssh/, thereby masking the default /root/.ssh/config included in the image. flux would fail to connect to the git server with the following error:

caller=main.go:407 component=git err="git clone: fatal: Could not read from remote repository."

Modifying the config-map to include the contents of the ssh config allows flux 1.2.3 to work correctly.

apiVersion: v1
data:
  config: "Host *\nStrictHostKeyChecking yes\nIdentityFile /etc/fluxd/ssh/identity\nLogLevel error  \n"
  known_hosts: |
    gitlab.exmaple.com ssh-rsa <insert your keys here>
kind: ConfigMap
metadata:
  name: flux-known-hosts

[aboyett]

@squaremo your new subpath volumeMounts spec has a small mistake. Replace this:

        volumeMounts:
        - name: git-key
          mountPath: /etc/fluxd/ssh
        # vvv Additional volume mount
        - name: known-hosts
          mountPath: /root/.ssh/config
          subPath: config

with this:

        volumeMounts:
        - mountPath: /etc/fluxd/ssh
          name: git-key
        - mountPath: /root/.ssh/known_hosts
          name: known-hosts
          subPath: known_hosts

[squaremo]

We now bake a known_hosts into the image at /etc/ssh/ssh_known_hosts, and config at /etc/ssh/ssh_config, so it is safe to mount both of those into /root/.ssh/ from a configmap.

How to do so is documented in https://github.com/weaveworks/flux/blob/master/site/standalone-setup.md#using-a-private-git-host and and demonstrated in the example deployment https://github.com/weaveworks/flux/blob/master/deploy/flux-deployment.yaml.

(EDIT: revised URL to doc)

[gellweiler]

The url for the documentation for the private git host has changed:
https://github.com/weaveworks/flux/blob/1.8.2/site/standalone-setup.md#using-a-private-git-host

[cloudoutloud]

I have an issue where it seems like the flux deployment is recreating the pod every so often and loses the ssh known_host config. Had anyone else experienced this?

Whats the best way to update the flux deployment if it's already deployed. I just outputting it to yaml then ran 'kubectl apply-f'