CSI volume getting mounted as EmptyDir

[jotaylo]

What steps did you take:

I'm trying to mount a CSI volume in my kubeflow pipeline using the Azure Key Vault Provider for Secrets Store CSI Driver.

I'm using python to create the pipeline and mount the volume:

def use_keyvault_secret_provider(volume_name='secrets-store-inline', secret_provider_class='azure-kvname', secret_volume_mount_path='/app/secrets'):
    def _use_keyvault_secret_provider(task):
        from kubernetes import client as k8s_client
        task = task.add_volume(
            k8s_client.V1Volume(
                name=volume_name,
                csi=k8s_client.V1CSIVolumeSource(
                    driver="secrets-store.csi.k8s.io",
                    read_only=True,
                    volume_attributes={
                        "secretProviderClass" : secret_provider_class
                        },
                    node_publish_secret_ref=k8s_client.V1LocalObjectReference(
                        name="secrets-store-creds"
                        )
                )
            )
        ).add_volume_mount(
                k8s_client.V1VolumeMount(
                    name=volume_name,
                    mount_path=secret_volume_mount_path,
                    read_only=True
                )
            )
        return task
    return _use_keyvault_secret_provider

What happened:

The pipeline starts, but the volume gets mounted to the container as EmptyDir.

What did you expect to happen:

The volume is mounted as a CSI volume.

Environment:

How did you deploy Kubeflow Pipelines (KFP)?
https://github.com/kaizentm/kubemlops/blob/master/setup/kfp/kubeflow-install.sh

KFP version: 1.0.0

KFP SDK version:
kfp 1.0.0
kfp-server-api 1.0.0

Anything else you would like to add:

I tried unzipping the compiled pipeline and submitting it using the argo cli. This worked, the volume was mounted successfully.

This is what the volume looks like in the yaml file:

    volumes:
    - csi:
        driver: secrets-store.csi.k8s.io
        nodePublishSecretRef: {name: secrets-store-creds}
        readOnly: true
        volumeAttributes: {secretProviderClass: azure-kvname}
      name: secrets-store-inline

This is what it looks like in the pod created by the kubeflow pipeline:

Volumes:
  secrets-store-inline:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>

This is what it looks like in the pod created by submitting directly to argo:

Volumes:
  secrets-store-inline:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=azure-kvname

The csi driver logs contain no record of any attempts to create the volume.

/kind bug
/area backend

[Ark-kun]

I tried unzipping the compiled pipeline and submitting it using the argo cli. This worked, the volume was mounted successfully.

This is really strange. The KFP backend does very minimal modifications to the pipeline before handing it to Argo. I wonder could it be caused by using outdated Go clients for Kubernetes and Argo. Maybe the csi field is lost in the process.

This is what the volume looks like in the yaml file:

Can you please check the Workflow yaml submitted to execution?

kubectl get wf ... --output yaml

[jotaylo]

That only the contains the name:

    volumes:
    - name: secrets-store-inline

[rmgogogo]

Is it running in Azure? Help wanted from community.

[jotaylo]

Yes, it's running in AKS.

[Ark-kun]

That only the contains the name:

Then the most likely cause is the one I listed: The outdated Go clients for Kubernetes and Argo lose fields when constructing Workflow object from YAML.

@rmgogogo I really think we should improve the backend build story so that we can upgrade the modules. They're pretty old now.

[Bobgy]

Then this should be resolved by #3770.
Already planned in Q3

[Bobgy]

/assign @jingzhang36

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rmgogogo]

/cc @Bobgy

[Bobgy]

I think this should be fixed by upgrading kubernetes client, partially related to #4553.

[Bobgy]

and argoproj/argo-workflows#4426

[Shaked]

@Bobgy what's missing for this issue to be resolved?

[Bobgy]

@Shaked the argo upstream issue seems to target to be released as part of argo V3 in January: argoproj/argo-workflows#4426 (comment).

We'll upgrade argo to V3 and get this resolved after argo's release.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Shaked]

@Bobgy any update about this issue?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bobgy]

We have upgraded to latest argo, so most likely the issue is fixed. Please let me know if it's not the case.

[vijayreddybo]

Hi @Bobgy, I am still facing this issue with kubeflow 1.4 and AWS EKS 1.21. Any inputs on how to get past this?