git auth with multiple keys for the same provider fail

[r0bj]

Expected Behavior

Multiple ssh keys to the same host/provider (eg. github) are working.

Actual Behavior

Only first ssh key for github is working, rest is failing.

Steps to Reproduce the Problem

Two different secrets with two different ssh keys for two different github repos:

---
apiVersion: v1
kind: Secret
metadata:
  name: ssh-key-repo1
  annotations:
    tekton.dev/git-0: github.com
type: kubernetes.io/ssh-auth
data:
  ssh-privatekey: <key>

---
apiVersion: v1
kind: Secret
metadata:
  name: ssh-key-repo2
  annotations:
    tekton.dev/git-0: github.com
type: kubernetes.io/ssh-auth
data:
  ssh-privatekey: <key>

ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fetch-repo
secrets:
- name: ssh-key-repo1
- name: ssh-key-repo2

It produces .ssh/config like this:

Host github.com
    HostName github.com
    Port 22
    IdentityFile /tekton/home/.ssh/id_ssh-key-repo1
    IdentityFile /tekton/home/.ssh/id_ssh-key-repo2

For ssh this config is fine but git for some reason respect only first IdentityFile clause and ignores rest. So in this case you can only authenticate to repo1. Authentication to repo2 will fail - github will complain that the repository doesn't match.

So in order to authenticate properly with git and multiple ssh keys, .ssh/config file should look similar to this:

Host github.aaakk.us.kg-ssh-key-repo1
    HostName github.com
    Port 22
    IdentityFile /tekton/home/.ssh/id_ssh-key-repo1

Host github.aaakk.us.kg-ssh-key-repo2
    HostName github.com
    Port 22
    IdentityFile /tekton/home/.ssh/id_ssh-key-repo2

Unfortunately git client then should also use those fake hosts:

git clone [email protected]:example/repo1.git

and

git clone [email protected]:example/repo2.git

For this reason, above workaround seems not quite reasonable since you can have git repo url dynamically fetched from git push webhook and then it would be in a form: [email protected]:example/repo1.git or [email protected]:example/repo2.git

Additional Info

This is particularly painful with github because it doesn't allow to use the same deploy key for different repos so you need to use different keys. And this it how you hit this issue.

• Kubernetes version:

  Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T23:41:24Z", GoVersion:"go1.14", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

• Tekton Pipeline version:

Client version: 0.8.0
Pipeline version: v0.11.1

[r0bj]

More thoughts about it, if it's not possible to create single .ssh/config file that allows git to authenticate to multiple github private repos then it completely changes EventListener use case. Instead of use one EventListener for multiple private repos (single TriggerTemplate with ServiceAccount) we need to use many EventListeners, each for every single private github repo (each requires separate ServiceAccount with TriggerTemplate).
Am I wrong on this?

[r0bj]

Or maybe at least we can make the git-init image (gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init) from catalog https://github.com/tektoncd/catalog/blob/v1beta1/git/git-clone.yaml to work properly and respect multiple IdentityFile clauses in .ssh/config?

[vdemeester]

It is indeed one of the current limitiation of creds-init. There is multiple potential solution for this:

• have one .ssh/config as proposed above
• for each "duplicate provider", generate its own .ssh/config (and what comes with it, the ssh key, …) — this would require the clone task to be able to refer the correct one

More thoughts about it, if it's not possible to create single .ssh/config file that allows git to authenticate to multiple github private repos then it completely changes EventListener use case. Instead of use one EventListener for multiple private repos (single TriggerTemplate with ServiceAccount) we need to use many EventListeners, each for every single private github repo (each requires separate ServiceAccount with TriggerTemplate).
Am I wrong on this?

PipelineRun (or TaskRun) are the one in need for TriggerTemplates. It's not directly at the EventListener level that you specify this. You could have a parametrized serviceAccount in your template so that, depending on the repository the event originates from, you use different serviceAccount. I also think the EventListener, as of today, covers both use cases, it's not really opiniated on if you should use.

The git-init binary (and thus image) doesn't do anything specific related to authentication. It just reads .gitcredentials, .ssh/config and .gitconfig as it uses git and ssh.

/kind feature

[bobcatfish]

Note that @sbwsg is proposing some improvements to creds init in #2343 which I think takes a pretty firm divergence from the annotation based approach we've had so far and makes usage more explicit

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ghost]

/reopen
/remove-lifecycle stale
/remove-lifecycle rotten

[tekton-robot]

@sbwsg: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle stale
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ghost]

This remains an issue and we've had a user in slack recently query us about it. I think longer-term the Credentials UX improvements I'm trying to push in #2343 will allow users to handle this problem but I would like to keep it on the radar until optional workspaces or an equivalent feature are well-supported.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[vdemeester]

/remove-lifecycle stale

[dmitry-mightydevops]

Is my understanding correct that if we have 2 repos, defined via PipelineResources as:

apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: backend-repo
spec:
  type: git
  params:
    - name: url
      value: [email protected]/team/pga-backend.git
    - name: revision
      value: main

---

apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: frontend-repo
spec:
  type: git
  params:
    - name: url
      value: [email protected]:team/pga-frontend.git
    - name: revision
      value: main

then both of these should have just a single and same github deployment key for both repos in github repo settings?

[ghost]

then both of these should have just a single and same github deployment key for both repos in github repo settings?

Are you using these PipelineResources with a single Task? If so then yes you will unfortunately need to use the same deploy key.

If you use them as part of separate Tasks of a Pipeline you can assign each Task its own ServiceAccount + Secret. So the deploy keys can be different.

The more robust approach is to build a Pipeline that uses git-clone Tasks from the catalog. Each can be provided a different service account w/ deploy key. There's more coordination involved (with workspaces, etc) but the result is much more flexibility in the ways you can orchestrate.

[dmitry-mightydevops]

thank you Scott! Makes total sense!

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.