no error message: PVC is not created yet

[awesomescot]

Hi. I'm trying to mount an s3 bucket into airflow. But currently the PVC isn't being created and I'm not sure why. The logs aren't giving me much to go on, so here's all the infomation. Any help would be greatly appreciated.

Thanks!

Dataset

kind: Dataset
metadata:
  name: s3-dags-dataset
  namespace: dlf
spec:
  local:
    type: "COS"
    # Secret needs to be deployed as per format on README
    secret-name: airflow-s3-user-creds
    secret-namespace: airflow
    endpoint: "https://s3.amazonaws.com"
    bucket: "airflow-dags-staging"
    readonly: "true"
    region: "us-east-1"

Secret (as terraform code)

resource "kubernetes_secret" "airflow-s3-user-creds" {
  metadata {
    name = "airflow-s3-user-creds"
    namespace = "airflow"
  }

  data = {
    accessKeyID: aws_iam_access_key.airflow_s3_user_staging.id
    secretAccessKey: aws_iam_access_key.airflow_s3_user_staging.secret
  }
}

Logs from dataset-operator

1.7214020186748595e+09	INFO	controller.dataset	Reconciling Dataset	{"reconciler group": "datashim.io", "reconciler kind": "Dataset", "name": "s3-dags-dataset", "namespace": "dlf", "Request.Namespace": "dlf", "Request.Name": "s3-dags-dataset"}
1.7214020186749408e+09	INFO	controller.dataset	PVC is not created yet	{"reconciler group": "datashim.io", "reconciler kind": "Dataset", "name": "s3-dags-dataset", "namespace": "dlf", "Request.Namespace": "dlf", "Request.Name": "s3-dags-dataset"}

Logs from csi-provisioner-s3

I0719 15:12:06.109780       1 reflector.go:381] k8s.io/client-go/informers/factory.go:134: forcing resync
I0719 15:12:06.242355       1 reflector.go:381] sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller/controller.go:869: forcing resync
I0719 15:14:06.979971       1 reflector.go:530] sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller/controller.go:872: Watch close - *v1.StorageClass total 8 items received
I0719 15:14:57.979023       1 reflector.go:530] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PersistentVolumeClaim total 9 items received
I0719 15:15:43.053443       1 reflector.go:530] sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller/controller.go:869: Watch close - *v1.PersistentVolume total 7 items received
I0719 15:16:14.894603       1 reflector.go:530] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StorageClass total 11 items received

Logs from csi-attacher-nfsplugin

I0719 15:16:00.766171       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PersistentVolume total 9 items received
I0719 15:16:00.766435       1 round_trippers.go:435] curl -v -XGET  -H "Authorization: Bearer <masked>" -H "Accept: application/json, */*" -H "User-Agent: csi-attacher/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.31.0.1:443/api/v1/persistentvolumes?allowWatchBookmarks=true&resourceVersion=3407208&timeout=7m46s&timeoutSeconds=466&watch=true'
I0719 15:16:00.769287       1 round_trippers.go:454] GET https://10.31.0.1:443/api/v1/persistentvolumes?allowWatchBookmarks=true&resourceVersion=3407208&timeout=7m46s&timeoutSeconds=466&watch=true 200 OK in 2 milliseconds
I0719 15:16:00.769310       1 round_trippers.go:460] Response Headers:
I0719 15:16:00.769315       1 round_trippers.go:463]     Audit-Id: 1dde8d42-3021-498f-bc66-9c80b3be9c67
I0719 15:16:00.769319       1 round_trippers.go:463]     Cache-Control: no-cache, private
I0719 15:16:00.769322       1 round_trippers.go:463]     Content-Type: application/json
I0719 15:16:00.769325       1 round_trippers.go:463]     X-Kubernetes-Pf-Flowschema-Uid: d12147bc-b3fb-4cd6-a06f-41072f44d647
I0719 15:16:00.769328       1 round_trippers.go:463]     X-Kubernetes-Pf-Prioritylevel-Uid: 5c9b0362-997b-4e40-985a-6c29586c5028
I0719 15:16:00.769331       1 round_trippers.go:463]     Date: Fri, 19 Jul 2024 15:16:00 GMT
I0719 15:17:06.972289       1 reflector.go:381] k8s.io/client-go/informers/factory.go:134: forcing resync
I0719 15:17:06.972327       1 reflector.go:381] k8s.io/client-go/informers/factory.go:134: forcing resync

Anything else that would be useful?

[AlessandroPomponio]

Hi @awesomescot ,

What happens if you try to provision your dataset in the same namespace where the secret is located?

This piece of code makes me think what you're trying to do isn't supported anymore

https://github.com/datashim-io/datashim/blob/master/src%2Fdataset-operator%2Fcontrollers%2Fdatasetinternal_controller.go#L365-L377

As a disclaimer, though, I only looked at this from my phone, I will probably need to have a more in-depth look

[srikumar003]

hi @awesomescot, the key problem is that you are creating the dataset in dlf namespace while the secret is defined in airflow namespace. Datashim requires that the dataset and secret must be in the same namespace.

We used to allow the namespaces to be different but blocked this due to #146. Hope this helps

We should provide a better error description. Will create a new issue for this change

[awesomescot]

Thank you for the responses. That makes sense now.

…

On Mon, Jul 22, 2024 at 3:57 AM Srikumar Venugopal ***@***.***> wrote: hi @awesomescot <https://github.com/awesomescot>, the key problem is that you are creating the dataset in dlf namespace while the secret is defined in airflow namespace. Datashim requires that the dataset and secret must be in the same namespace. We used to allow the namespaces to be different but blocked this due to #146 <#146>. Hope this helps — Reply to this email directly, view it on GitHub <#370 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASLFLGRUGOYS676ZBSCQHGLZNTQSZAVCNFSM6AAAAABLEZDXZKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBSGY3TKNZTGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[AlessandroPomponio]

I'll close this issue @awesomescot. Feel free to re-open it or open a new one if you need anything else.