Exec with ConnectGateway + Workload Identity does not work on remote clusters it seems

[esn89]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
I cannot use the Terminal Exec when connecting from a central ArgoCD GKE cluster to other GKE clusters via Workload Identity

To Reproduce
On the central GKE ArgoCD:

# Create a Google service account and set required permissions
gcloud iam service-accounts create argocd-fleet-admin --project $PROJECT_ID

gcloud projects add-iam-policy-binding $PROJECT_ID --member "serviceAccount:argocd-fleet-admin@${PROJECT_ID}.iam.gserviceaccount.com" --role roles/gkehub.gatewayEditor

# Create IAM policy allowing the ArgoCD namespace/KSA to impersonate the previously created GSA
gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:${PROJECT_ID}.svc.id.goog[argocd/argocd-server]" argocd-fleet-admin@$PROJECT_ID.iam.gserviceaccount.com

gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:${PROJECT_ID}.svc.id.goog[argocd/argocd-application-controller]" argocd-fleet-admin@$PROJECT_ID.iam.gserviceaccount.com

Then make sure that this is in your argocd-cm

exec.enabled: "true"

On the worker GKE cluster:

gcloud services enable --project=$PROJECT_ID gkeconnect.googleapis.com gkehub.googleapis.com cloudresourcemanager.googleapis.com iam.googleapis.com ​​connectgateway.googleapis.com

# Grant ArgoCD permissions to access and manage the application-cluster
gcloud projects add-iam-policy-binding $PROJECT_ID --member "serviceAccount:argocd-fleet-admin@${PROJECT_ID}.iam.gserviceaccount.com" --role roles/container.admin

# List application cluster(s) URI used for registration
gcloud container clusters list --uri --project $PROJECT_ID

# Register the application-cluster(s) with the Fleet Project. We use gcloud SDK commands as an example here. If you prefer other tools like Terraform, please refer to this [document](https://cloud.google.com/anthos/fleet-management/docs/register/gke).
gcloud container fleet memberships register $CLUSTER_NAME --gke-uri {URI from step 2} --enable-workload-identity --project $PROJECT_ID

Once that is added deploy an ApplicationSet to it.

Expected behavior

I expect to see a prompt to the terminal to that remote cluster. But instead I only see:

Screenshots

https://imgur.com/a/iccXhrQ

Version

{
    "Version": "v2.7.6+00c914a.dirty",
    "BuildDate": "2023-06-20T20:51:13Z",
    "GitCommit": "00c914a948d9e8ad99be8bd82a368fbdeba12f88",
    "GitTreeState": "dirty",
    "GoVersion": "go1.19.10",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.0.1 2023-03-14T01:32:48Z",
    "HelmVersion": "v3.11.2+g912ebc1",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.19.1"
}

argocd: v2.7.6+00c914a
  BuildDate: 2023-06-20T21:18:20Z
  GitCommit: 00c914a948d9e8ad99be8bd82a368fbdeba12f88
  GitTreeState: clean
  GoVersion: go1.19.10
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.7.6+00c914a.dirty

Logs

2023/08/15 16:32:31 http: response.WriteHeader on hijacked connection from github.com/argoproj/argo-cd/v2/server/application.(*terminalHandler).ServeHTTP (terminal.go:245)
time="2023-08-15T16:35:46Z" level=info msg="terminal session starting" appNamespace=argocd application=$APPLICATIONAME container=main namespace=datatransfer podName=$PODNAME project=dev userName=$MYNAME
2023/08/15 16:35:47 http: response.WriteHeader on hijacked connection from github.com/argoproj/argo-cd/v2/server/application.(*terminalHandler).ServeHTTP (terminal.go:245)
time="2023-08-15T21:56:53Z" level=info msg="terminal session starting" appNamespace=argocd application=$APPLICATIONAME container=$CONTAINER namespace=$NAMESPACE podName=$PODNAME project=dev userName=$MYNAME
2023/08/15 21:56:53 http: response.WriteHeader on hijacked connection from github.com/argoproj/argo-cd/v2/server/application.(*terminalHandler).ServeHTTP (terminal.go:245)

Interestingly enough, in my remote cluster that I connect to, I checked the Kubernetes Service Account and its ClusterRole and it seems to be good (in my view):

SA:

apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2023-06-28T22:24:02Z"
  name: argocd-manager
  namespace: kube-system
  resourceVersion: "135066360"
  uid: d9fafe5b-6785-4c7c-bc4e-93277fd5f2c6
secrets:
- name: argocd-manager-token-gbqw8

ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argocd-manager-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argocd-manager-role
subjects:
- kind: ServiceAccount
  name: argocd-manager
  namespace: kube-system

ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argocd-manager-role
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'

[torfjor]

Connect gateway does not support this. There was a mention of adding support for both port-forward and attach once 1.28 has landed and the HTTP/2 websocket support was in place (The CGW frontend doesn't support SPDY).

EDIT: Actually, it looks like this feature has been pushed to 1.29: kubernetes/enhancements#4006

[esn89]

Thank you for letting me know, @torfjor

Would it be as simply as waiting for 1.29 of K8s? Or will there need to be changes/updates/code changes on the ArgoCD codebase side of things as well?

[torfjor]

I assume that some changes must be done on the Connect Gateway side of things when this lands. No changes required for ArgoCD that I'm aware of.