kfp_server_api.exceptions.ApiException: (500) - failed to authenticate with kubeflow pipeline

[oonisim]

Objective

Please provide the steps how to implement the authentication for Kubeflow pipeline addressed in Add authentication with ServiceAccountToken #5138

Background

The Kubeflow sdk - error in client.list_experiments() refers to Kubeflow sdk - error in client.list_experiments() #6120 github issue through which the author looks fixed the issue.

I received feedback from the developers (see the closed issue). This is one of the current caveats of multi-user mode (see documentation). This usage is now being supported through #5138.

However, I could not figure out what is exactly the cause and how to fix it. It looks Connecting to Kubeflow Pipelines using the SDK client gives the configurations but not sure how exactly I need to do.

Reproduction steps

1. Deployed Minikube on a remote instance and setup kubectl connection.
2. Deployed Kubeflow 1.5.0 by following Install with a single command.
3. Verify the connection and confirm pods are running from the local laptop.

$ kubectl get pods -n kubeflow
NAME                                                         READY   STATUS    RESTARTS   AGE
admission-webhook-deployment-7df7558c67-drdzw                1/1     Running   5          2d18h
cache-deployer-deployment-6f4bcc969-8kpm6                    2/2     Running   15         2d18h
cache-server-575d97c95-k7rv4                                 2/2     Running   10         2d18h
centraldashboard-5dd4f57bbd-gcxn5                            2/2     Running   10         2d18h
jupyter-web-app-deployment-5886974887-8c2cf                  1/1     Running   5          2d18h
katib-controller-58ddb4b856-mzq2l                            1/1     Running   46         2d18h
katib-db-manager-6df878f5b8-c9dhr                            1/1     Running   8          2d18h
katib-mysql-6dcb447c6f-lz5b8                                 1/1     Running   5          2d18h
katib-ui-f787b9d88-8h25n                                     1/1     Running   6          2d18h
kfserving-controller-manager-0                               2/2     Running   50         2d18h
kfserving-models-web-app-7884f597cf-m9n59                    2/2     Running   10         2d18h
kserve-models-web-app-5c64c8d8bb-bpdsb                       2/2     Running   10         2d18h
kubeflow-pipelines-profile-controller-84bcbdb899-669hr       1/1     Running   5          2d18h
metacontroller-0                                             1/1     Running   6          2d18h
metadata-envoy-deployment-7b847ff6c5-d2fjv                   1/1     Running   5          2d18h
metadata-grpc-deployment-6f6f7776c5-2vqp6                    2/2     Running   21         2d18h
metadata-writer-78fc7d5bb8-q8hfq                             2/2     Running   11         2d18h
minio-5b65df66c9-fttpm                                       2/2     Running   10         2d18h
ml-pipeline-75b5c59d7f-k7mm7                                 2/2     Running   59         2d18h
ml-pipeline-persistenceagent-87b6888c4-swv8k                 2/2     Running   10         2d18h
ml-pipeline-scheduledworkflow-665847bb9-4b5vr                2/2     Running   10         2d18h
ml-pipeline-ui-68cc764f66-892rz                              2/2     Running   14         2d18h
ml-pipeline-viewer-crd-68777557fb-6lq88                      2/2     Running   16         2d18h
ml-pipeline-visualizationserver-58ccb76855-qz2rc             2/2     Running   12         2d18h
mysql-f7b9b7dd4-2dpqv                                        2/2     Running   10         2d18h
notebook-controller-deployment-6c5f5d6cfc-mxmzw              2/2     Running   17         2d18h
profiles-deployment-5cdc5dc577-szhjk                         3/3     Running   61         2d18h
tensorboard-controller-controller-manager-5cbddb7fb5-xgq2v   3/3     Running   21         2d18h
tensorboards-web-app-deployment-7c5db448d7-t8xqp             1/1     Running   5          2d18h
training-operator-7b8cc9865d-qr8hm                           1/1     Running   7          2d18h
volumes-web-app-deployment-87484c848-qvsnc                   1/1     Running   5          2d18h
workflow-controller-6bf87db995-snfdn                         2/2     Running   20         2d18h

4. Installed kubeflow SDK in the local laptop.

$ pip list | grep kfp
kfp                              1.8.12
kfp-pipeline-spec                0.1.15
kfp-server-api                   1.8.1

5. Connected to the kubeflow pipeline as per Connecting to Kubeflow Pipelines using the SDK client

kubectl port-forward svc/ml-pipeline-ui 3000:80 --namespace kubeflow\

6. Verified the pipeline ui appears as in the document.

You can verify that port forwarding is working properly by visiting http://localhost:3000 in your browser. If port forwarding is working properly, the Kubeflow Pipelines UI appears.

7. Applied the YAML by kubectl.

apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
  name: access-ml-pipeline
  namespace: "kubeflow"
spec:
  desc: Allow access to Kubeflow Pipelines
  selector:
    matchLabels:
      access-ml-pipeline: "true"
  volumes:
    - name: volume-kf-pipeline-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 7200
              audience: pipelines.kubeflow.org
  volumeMounts:
    - mountPath: /var/run/secrets/kubeflow/pipelines
      name: volume-kf-pipeline-token
      readOnly: true
  env:
    - name: KF_PIPELINES_SA_TOKEN_PATH
      value: /var/run/secrets/kubeflow/pipelines/token

and

apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
  name: access-ml-pipeline
  namespace: "kubeflow-user-example-com"
spec:
  desc: Allow access to Kubeflow Pipelines
  selector:
    matchLabels:
      access-ml-pipeline: "true"
  volumes:
    - name: volume-kf-pipeline-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 7200
              audience: pipelines.kubeflow.org
  volumeMounts:
    - mountPath: /var/run/secrets/kubeflow/pipelines
      name: volume-kf-pipeline-token
      readOnly: true
  env:
    - name: KF_PIPELINES_SA_TOKEN_PATH
      value: /var/run/secrets/kubeflow/pipelines/token

8. Run the code from the local laptop.

import kfp
client = kfp.Client(host='http://localhost:3000', namespace='kubeflow')
print(client.list_experiments(namespace='kubeflow'))

Got the error.

Traceback (most recent call last):
  File "connect_kubeflow_pipeline.py", line 8, in <module>
    print(client.list_experiments(namespace='kubeflow'))
  File "/Users/1245095/venv/ml/lib/python3.7/site-packages/kfp/_client.py", line 540, in list_experiments
    filter=filter)
  File "/Users/1245095/venv/ml/lib/python3.7/site-packages/kfp_server_api/api/experiment_service_api.py", line 567, in list_experiment
    return self.list_experiment_with_http_info(**kwargs)  # noqa: E501
  File "/Users/1245095/venv/ml/lib/python3.7/site-packages/kfp_server_api/api/experiment_service_api.py", line 682, in list_experiment_with_http_info
    collection_formats=collection_formats)
  File "/Users/1245095/venv/ml/lib/python3.7/site-packages/kfp_server_api/api_client.py", line 369, in call_api
    _preload_content, _request_timeout, _host)
  File "/Users/1245095/venv/ml/lib/python3.7/site-packages/kfp_server_api/api_client.py", line 188, in __call_api
    raise e
  File "/Users/1245095/venv/ml/lib/python3.7/site-packages/kfp_server_api/api_client.py", line 185, in __call_api
    _request_timeout=_request_timeout)
  File "/Users/1245095/venv/ml/lib/python3.7/site-packages/kfp_server_api/api_client.py", line 393, in request
    headers=headers)
  File "/Users/1245095/venv/ml/lib/python3.7/site-packages/kfp_server_api/rest.py", line 234, in GET
    query_params=query_params)
  File "/Users/1245095/venv/ml/lib/python3.7/site-packages/kfp_server_api/rest.py", line 224, in request
    raise ApiException(http_resp=r)
kfp_server_api.exceptions.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'X-Powered-By': 'Express', 'content-type': 'application/json', 'date': 'Tue, 24 May 2022 04:58:42 GMT', 'x-envoy-upstream-service-time': '2', 'server': 'envoy', 'connection': 'close', 'transfer-encoding': 'chunked'})
HTTP response body: {"error":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:279\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).canAccessExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:249\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:148\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1286\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1609\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:934\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:279\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:150\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1286\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1609\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:934\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","code":13,"message":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:279\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).canAccessExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:249\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:148\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1286\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1609\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:934\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:279\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:150\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1286\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1609\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:934\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","details":[{"@type":"type.googleapis.com/api.Error","error_message":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:279\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).canAccessExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:249\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:148\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1286\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1609\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:934\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:279\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:150\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1286\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1609\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:934\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","error_details":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:279\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).canAccessExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:249\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:148\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1286\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1609\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:934\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:279\ngithub.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).ListExperiment\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:150\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler.func1\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1089\nmain.apiServerInterceptor\n\t/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30\ngithub.com/kubeflow/pipelines/backend/api/go_client._ExperimentService_ListExperiment_Handler\n\t/go/src/github.com/kubeflow/pipelines/backend/api/go_client/experiment.pb.go:1091\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1286\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1609\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:934\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"}]}

[oonisim]

By further research, found How to do programmatic authentication with Dex? #140. By modifying the code as per the solution provided there, it worked.

Please clarify if this we need to do. If so, please confirm where it is documented?

import requests
import kfp
import kfp.dsl as dsl
from kfp.components import create_component_from_func


# Does not work
#import kfp
#client = kfp.Client(host='http://localhost:3000', namespace='kubeflow')
#print(client.list_experiments(namespace='kubeflow'))


# --------------------------------------------------------------------------------
# https://github.com/kubeflow/kfctl/issues/140#issuecomment-719894529
# How to do programmatic authentication with Dex? #140
# --------------------------------------------------------------------------------
HOST = "http://localhost:8080/"
USERNAME = "[email protected]"
PASSWORD = "12341234"
NAMESPACE = "kubeflow-user-example-com"

session = requests.Session()
response = session.get(HOST)

headers = {
    "Content-Type": "application/x-www-form-urlencoded",
}

data = {"login": USERNAME, "password": PASSWORD}
session.post(response.url, headers=headers, data=data)
session_cookie = session.cookies.get_dict()["authservice_session"]

client = kfp.Client(
    host=f"{HOST}/pipeline",
    cookies=f"authservice_session={session_cookie}",
    namespace=NAMESPACE,
)

print(client.list_pipelines())


def add(a: float, b: float) -> float:
    '''Calculates sum of two arguments'''
    return a + b


add_op = create_component_from_func(
    add, output_component_file='add_component.yaml')


@dsl.pipeline(
    name='Addition pipeline',
    description='An example pipeline that performs addition calculations.'
)
def add_pipeline(
        a='1',
        b='7',
):
    # Passes a pipeline parameter and a constant value to the `add_op` factory
    # function.
    first_add_task = add_op(a, 4)
    # Passes an output reference from `first_add_task` and a pipeline parameter
    # to the `add_op` factory function. For operations with a single return
    # value, the output reference can be accessed as `task.output` or
    # `task.outputs['output_name']`.
    second_add_task = add_op(first_add_task.output, b)

# Specify argument values for your pipeline run.
arguments = {'a': '7', 'b': '8'}

# Create a pipeline run, using the client you initialized in a prior step.
#client.create_run_from_pipeline_func(add_pipeline, arguments=arguments)

kfp.compiler.Compiler().compile(
    pipeline_func=add_pipeline,
    package_path='pipeline.yaml')

[juliusvonkohout]

client = kfp.Client(host='http://localhost:3000', namespace='kubeflow') is wrong. client = kfp.Client() is enough for an in-cluster jupyterlab (jupyter notebook is wrong, a notebook is a file). And especially localhost is nonsense. You want the apiserver from the kubeflow namespace. You should really spend some time alone with the Kubeflow architecture and code/documentation.
Also kubectl port-forward svc/ml-pipeline-ui 3000:80 --namespace kubeflow\ shows that you do not understand the Kubeflow architecture. You are trying to abuse the frontend as apiserver.

[jlyaoyuli]

@oonisim Please confirm that you are able to use Kubeflow dashboard without port forwarding. Then you can create KFP client using the URL endpoint to pipeline.

[rimolive]

Closing this issue. No activity for more than a year.

/close

[google-oss-prow]

@rimolive: Closing this issue.

In response to this:

Closing this issue. No activity for more than a year.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.