Argo UI fails to load workflows when running in auth-mode=client

[gurvindersingh]

Summary

Argo UI gives error as 403 forbidden when trying to list workflows for a logged in user when running in client auth mode.

What happened

This is the exception in the logs

time="2021-06-08T13:22:41.269Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = workflows.argoproj.io is forbidden: User "https://server.com:8443/idauth/oidc/endpoint/OP#[email protected]\" cannot list resource "workflows" in API group "argoproj.io" at the cluster scope" grpc.code=PermissionDenied grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2021-06-08T13:22:41Z" grpc.time_ms=3.072 span.kind=server system=grpc

Its correct that this user doesn't have access at cluster scope but user has access to multiple namespaces and UI should select one of those namespace and list workflows from that rather than giving error.

What is expected to happen

Argo UI list the workflows from the allowed namespaces and list all accessible namespaces for the logged in user

Diagnostics

What Kubernetes provider are you using? 1.16.8

What version of Argo Workflows are you running? v3.1.0-rc12

What executor are you running? Docker

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

Can you confirm if the user can enter the namespace in the filter options?

[gurvindersingh]

If a user enter the namespace manually in the filter option, then UI displays the workflows correctly. But UI doesn't populate the namespace list itself which I think it should based on credentials provided by the user.

[alexec]

I think it should based on credentials provided by the user.

I agreed, how to implement - we need a new API called GetNamespaces:

1. List all namespaces. This cannot be done with the user's account. By definition they are limited to some namespaces, then you would not give them access to all namespaces. You could use the argo-server's default account for this.
2. For each namespace, use the kubectl can-i API (SelfSubjectAccessReview) to see if you can list workflows.

This is O(N+1) - so the result should be cached by the browser.

Would you like to submit a PR?

[alexec]

As this bugged me and other users, I've created a PoC PR for you to test. Can you please test the argoproj/argo-server:dev-ns image and tell me what you think?

[gurvindersingh]

I have tested it but the error still persists. Here is the logs from current run

time="2021-06-10T09:11:19.437Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=GetInfo grpc.service=info.InfoService grpc.start_time="2021-06-10T09:11:19Z" grpc.time_ms=0.083 span.kind=server system=grpc
time="2021-06-10T09:11:20.578Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=GetInfo grpc.service=info.InfoService grpc.start_time="2021-06-10T09:11:20Z" grpc.time_ms=0.059 span.kind=server system=grpc
time="2021-06-10T09:11:35.278Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetInfo grpc.service=info.InfoService grpc.start_time="2021-06-10T09:11:35Z" grpc.time_ms=2.222 span.kind=server system=grpc
time="2021-06-10T09:11:35.339Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetVersion grpc.service=info.InfoService grpc.start_time="2021-06-10T09:11:35Z" grpc.time_ms=2.387 span.kind=server system=grpc
time="2021-06-10T09:11:48.206Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = workflowtemplates.argoproj.io is forbidden: User \"https://server.com:8443/idauth/oidc/endpoint/OP#[email protected]\" cannot list resource \"workflowtemplates\" in API group \"argoproj.io\" in the namespace \"null\"" grpc.code=PermissionDenied grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2021-06-10T09:11:48Z" grpc.time_ms=3.668 span.kind=server system=grpc
time="2021-06-10T09:11:48.207Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = cronworkflows.argoproj.io is forbidden: User \"https://server.com:8443/idauth/oidc/endpoint/OP#[email protected]\" cannot list resource \"cronworkflows\" in API group \"argoproj.io\" in the namespace \"null\"" grpc.code=PermissionDenied grpc.method=ListCronWorkflows grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2021-06-10T09:11:48Z" grpc.time_ms=3.314 span.kind=server system=grpc
time="2021-06-10T09:11:48.225Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = workflows.argoproj.io is forbidden: User \"https://server.com:8443/idauth/oidc/endpoint/OP#[email protected]\" cannot list resource \"workflows\" in API group \"argoproj.io\" in the namespace \"null\"" grpc.code=PermissionDenied grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2021-06-10T09:11:48Z" grpc.time_ms=3.68 span.kind=server system=grpc
time="2021-06-10T09:11:50.435Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = workflows.argoproj.io is forbidden: User \"https://server.com:8443/idauth/oidc/endpoint/OP#[email protected]\" cannot list resource \"workflows\" in API group \"argoproj.io\" at the cluster scope" grpc.code=PermissionDenied grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2021-06-10T09:11:50Z" grpc.time_ms=14.881 span.kind=server system=grpc

Between I assume, you meant this image argoproj/argocli:dev-ns to test

[alexec]

Yes. Can you please include a screenshot of the workflow list page options?

[gurvindersingh]

[alexec]

It says “null” in the workflow namespace filter. Please enter a real namespace.

[gurvindersingh]

That is the problem as by default when user logs in, UI selects namespace as null and displays the error. If a user manually enter the namespace where he/she has access to then UI does work fine.

[alexec]

The screenshot you attached is not from the image. But I did enter the wrong image name, it should be argoproj/argocli:dev-ns.

You'll see a new drop-down if have the right version.

[gurvindersingh]

i think the issue is that the UI display the null namespace too quickly without waiting for listNamespaces call to return the result and display in UI. I am running this image argoproj/argocli:dev-ns with imagePullPolicy: Always so should have the correct version running.

If I wait on the workflow listing page for 1 mins or so with error, UI displays the correct list of namespaces where current user access. So I guess UI need to wait for namesapce call to get back before rendering the page

[alexec]

@gurvindersingh can you try argoproj/argocli:dev-undef? This directly addresses what I think is the issue with null.