PVC in pending state when GKE ckuster and CVS instance not in the same region

[tboussar]

Describe the bug
when creating PVC using trident 21.07.1, this one stay in pending state
our CVS instance is on europe-west3 and our GKE cluster on europe-west1

we added following topology in our backend configuration:

        "supportedTopologies": [
          {
            "topology.kubernetes.io/region": "europe-west3",
            "topology.kubernetes.io/zone": "europe-west3-a"
          },
          {
            "topology.kubernetes.io/region": "europe-west3",
            "topology.kubernetes.io/zone": "europe-west3-b"
          },
          {
            "topology.kubernetes.io/region": "europe-west3",
            "topology.kubernetes.io/zone": "europe-west3-c"
          },
          {
            "topology.kubernetes.io/region": "europe-west1",
            "topology.kubernetes.io/zone": "europe-west1-b"
          },
          {
            "topology.kubernetes.io/region": "europe-west1",
            "topology.kubernetes.io/zone": "europe-west1-c"
          },
          {
            "topology.kubernetes.io/region": "europe-west1",
            "topology.kubernetes.io/zone": "europe-west1-d"
          }

trident is installed with helm chart in version 21.07.1
all pods are up but we have following errors:

trident logs:

GRPC error: rpc error: code = Unknown desc = encountered error(s) in creating the volume: [Failed to create volume pvc-ffc6e2c3-0c72-4cff-91a5-58ef25fd8cf4 on storage pool cluster_name_02 from backend cluster_name_02: configured region does not match topology region]

Environment
Provide accurate information about the environment to help us reproduce the issue.

• Trident version: 21.07.1
• Trident installation flags used: [e.g. -d -n trident --use-custom-yaml]
• Container runtime: docker://19.3.2
• Kubernetes version: v1.19.13-gke.1200
• Kubernetes orchestrator: GKE
• OS:Ubuntu 18.04.5 LTS
• NetApp backend types: CVS for GCP

To Reproduce
install trident 21.07.1 on a cluster in a region and use à CVS instance in another

Expected behavior
It should be possible to have kubernetes cluster and CVS not in the same region

Additional context
Add any other context about the problem here.

[balaramesh]

Hello @tboussar

Presently, Trident does not allow the creation of CVS volumes cross-region, i.e., the region in which a CVS volume is provisioned by Trident must match the region of the node(s) it runs on/the volume is consumed. This was introduced with Trident's support for the Zone Redundant Standard-SW CVS service type, and is called out in the Trident docs

[tboussar]

Hello @balaramesh
thank you for the answer
Before the new version of trident it was possbile to have CVS and Kubernetes cluster not in the same region.
Today we have a workload in this case and we want to upgrade trident.
We want to reuse our NFS volumes and not lose data, when doing a migration of CVS (and in fact as I know, CVS is not the same on west1 and west3)

Is there a solution to bypass this limitation?

[balaramesh]

@tboussar can you provide additional context on how/why cross-region traffic is valuable? I understand that you would like to migrate volumes from one region to another. However, I also want you to be mindful of the fact that cross-region traffic can introduce latency and result in n/w ingress costs. I wanted to point this out and make sure this isn't something that goes unnoticed.

[tboussar]

Hello @balaramesh
in our case this is a problem of migration of our actual netapp volume.
There is also an issue between CVS and CVS performance which is only available on Frankfurt.

this is a blocking point for us today, as you know Google is upgrading all gke leaders to kubernetes 1.19
and will surely ask for version 1.20 soon.
our actual trident version 20.07.0 is not compatible with this kubernetes version, we are trying to use 20.07.1 (which should be compatible) but soon these version will also not be compatible, when google will decide to upgrade to kubernetes leader to 1.20.

I think by default user should not use cross region, but should be able to use it.

[tboussar]

Hello @balaramesh
can we have an update about this issue?
to complete, we are using trident operator
https://netapp-trident.readthedocs.io/en/stable-v21.01/kubernetes/deploying/operator-deploy.html

[gnarl]

This issue is fixed with commit f456ef6 and will be included in the Trident 22.01 release.