Azure DevOps: SSH Unable to extract public key from private key (after upgrade from flux 0.13.3 to 0.15.1)

[frbosch]

Describe the bug

I did try to update an existing cluster which is already controlled by fluxcd 0.13.3. I did not change the secret or anything, but the gitrepository sync stopped working.

To Reproduce

I did execute the following:

1. brew upgrade fluxcd/tap/flux
2. flux install --components-extra=image-reflector-controller,image-automation-controller --export > ./flux-system/gotk-components.yaml
3. commit this to git, this folder is synced to an AKS cluster

Now the i get this:

kubectl get gitrepository -nflux-system
... unable to clone 'ssh://[email protected]/v3/<correct path removed>, error: Failed to authenticate SSH session: Unable to extract public key from private key.

After rollback to v0.13.3 it works again.

Additional context

Provisioning of a newly created cluster fails also. With the same error.
I create the git repository with the following command:

flux create source git foundation-git \
  --git-implementation=libgit2 \
  --url=ssh://[email protected]/v3/IOTESC/xxx \
  --branch=main \
  --ssh-key-algorithm=rsa \
  --ssh-rsa-bits=4096 \
  --interval=1m

• Kubernetes version: 1.20.7
• Git provider: Azure Repository using libgit2

Below please provide the output of the following commands:

#flux --version
flux version 0.15.1

#flux check
► checking prerequisites
✔ kubectl 1.19.7 >=1.18.0-0
✔ Kubernetes 1.20.7 >=1.16.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.11.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.12.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.10.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.13.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.15.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.15.0
✔ all checks passed


#kubectl -n <namespace> logs deploy/source-controller
{"level":"error","ts":"2021-06-18T13:00:19.503Z","logger":"controller.gitrepository","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"foundation-git","namespace":"flux-system","error":"unable to clone 'ssh://[email protected]/v3/xxx', error: Failed to authenticate SSH session: Unable to extract public key from private key."}

[hiddeco]

Does the header of your private key start with -----BEGIN OPENSSH PRIVATE KEY----- or with something else?

[frbosch]

It starts with -----BEGIN PRIVATE KEY-----.

[stefanprodan]

@frbosch how did you generated that key?

[frbosch]

With flux cli (command is in the description of this task), not sure what this uses internally. Guess openssl?

openssl version                                                                                                                                                                                     
LibreSSL 2.8.3

[dzmitry-lahoda]

Not sure, it seems I also got issues while upgrading. Previously 2 repos worked, but after redeploy of flux stopped with #1545

[sevenam]

It's working for us in v0.14.2

[Nordes]

I also have the same issue and my private key has the following header -----BEGIN OPENSSH PRIVATE KEY-----.

I looked at the breaking changes and did not see anything about this topic until I upgraded. Is there a quick hack we can do to fix this?

[hiddeco]

I have been digging into this most of the day, and it seems that it is yet another issue with the relatively complex build requirements for libgit2.

With the information I have collected thus far, it seems that when we switched from Alpine to Debian for the source-controller image because of other issues with libgit2, it now no longer is capable of using private keys in the (modern) PKCS#8 and "OpenSSH" formats.

A workaround that is worth giving a try is to try using a key that has a PKCS#1 format, so the "classic" -----BEGIN RSA PRIVATE KEY-----.

[hiddeco]

Identified the root cause of the issue, dating back to 2012. The libssh2 package in Debian bullseye is linked against gcrypt instead of openssl, which results in odd behavior with PKCS* formats:

The Libgcrypt backend in libssh2 contains a hand written
slimmed down ASN.1 parser to read out the RSA key, but it does not
support any of the PKCS* encrypted forms of RSA keys. The OpenSSL
backend in libssh2 uses OpenSSL to read the keys, so it supports
whatever private key formats that OpenSSL supports
— https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=668271

Will try to use the packages from unstable, build libssh2 from source, or submit a package to bullseye-backports as suggested in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=668271#40. Please bear with me till then.

[hiddeco]

We forgot to post an update to this specific issue, but after the pull request that closed this was merged, another release was made to the source-controller.

Updating the controller to v0.15.2 using the latest release of flux should solve this issue.

[frbosch]

We forgot to post an update to this specific issue, but after the pull request that closed this was merged, another release was made to the source-controller.

Updating the controller to v0.15.2 using the latest release of flux should solve this issue.

Release 0.15.3 works for me now. Thanks a lot, great work.