How to supply a CA file to Kustomize Controller when decrypting using Hashicorp Vault

[abangs-nn]

I have a Kustomization that needs to decrypt files using keys stored in HashiCorp Vault. I am able to supply the vault token through a secret with key "sops.vault-token" however our Vault instance uses a self signed certificate so I'm getting the following error "x509 certificate signed by unknown authority". Is there a way to provide my CA file to the controller? I faced a similar issue when connecting to our Gitlab instance, but there was an option to provide the CA file within the secret for the source controller.

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: environment
  namespace: test
spec:
  interval: 1m
  sourceRef:
    kind: GitRepository
    name: environment-repo
  path: ${path}
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: vault-token

My secret is being created with Terraform

resource "kubernetes_secret" "vault_token" {
  metadata {
    name = "vault-token"
    namespace = "test"
  }
  data = {
    "sops.vault-token" = var.vault_token
  }
  type = "Opaque"
}

GitLab secret example that is working

resource "kubernetes_secret" "repo_access" {
  metadata {
    name = "repo"
    namespace = "test"
  }
  data = {
    "password" = "${var.repo_password}"
    "caFile" =  "${file("${path.module}/path/to/cert.crt")}"
  }
  type = "Opaque"
}

[souleb]

This is not supported at the moment. I would convert this to an issue.

[pjbgf]

@abangs-nn This should be doable by mounting the CA-cert into the source-controller pod. Here's what a patch to your kustomization.yaml might look like:

patches:
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/volumeMounts
        value:
          - name: vault-ca-pemstore
            mountPath: /etc/ssl/certs/vault-ca.pem
            subPath: vault-ca.pem
            readOnly: true
      - op: add
        path: /spec/template/spec/volumes
        value:
          - name: vault-ca-pemstore
            secret:
              secretName: vault-ca-pemstore
    target:
      kind: Deployment
      name: "source-controller"

You just need to create the vault-ca-pemstore secret with your CA root.

---

Update: added patch for readOnlyRootFilesystem

[pjbgf]

The read-only filesystem setting could be disabled by adding an extra patch:

      - op: replace
        path: /spec/template/spec/containers/0/securityContext/readOnlyRootFilesystem
        value: false

If that works well, you could either leave as-is, or alternatively build your own custom container image that bakes the CA file into /etc/ssl/certs/, so you don't need the patches.

At some point custom CA support should be added to the controller as a first class citizen, which would better cater for multi-tenancy scenarios as well.

[abangs-nn]

@pjbgf That worked for me, agreed it would be nice to not have to bundle CA like this in the future. Thanks.

[pjbgf]

@abangs-nn FYI, if you set the volume mount readOnly: true, then readOnlyRootFilesystem no longer needs to be changed. I updated my example above accordingly.

[kylejackp]

Can an example be given how to apply the above patch?
kubectl patch deployment kustomize-controller --patch-file patch.yml -n flux-system

Returns:

Warning: unknown field "patches"
deployment.apps/kustomize-controller patched (no change)

[stefanprodan]

@kylejackp see the docs here https://fluxcd.io/flux/installation/configuration/boostrap-customization/