deal w/ persistent storage issues on EKS

[joel-bluedata]

Probably need to set volumeBindingMode if/when EKS picks up support for K8s 1.12. It's unclear at the moment whether that will be entirely sufficient to make persistent storage work on EKS... maybe it will!

The issue: kubernetes/kubernetes#34583

The solution: https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode

[joel-bluedata]

Actually I misunderstood volumeBindingMode. That's a property on the StorageClass, not on the PVC. So if Amazon sets this, then things should "just work" for us when using 1.12 on EKS, without any KD changes. So, just something we need to test when 1.12 becomes available there.

[joel-bluedata]

Might get 1.12 on EKS soon: aws/containers-roadmap#24

[pawelprazak]

FYI the current (v1.12.6-eks-d69f1b) default storage class does not configure this:

  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"
    name: gp2
  parameters:
    fsType: ext4
    type: gp2
  provisioner: kubernetes.io/aws-ebs
  reclaimPolicy: Delete
  volumeBindingMode: Immediate

[joel-bluedata]

Might be able to edit that resource to add that property now. That's something we should try.

[joel-bluedata]

Assigning this to me to try it out.

[joel-bluedata]

So, we can't edit that field on an existing storage class...it's immutable.

If I create a new storage class identical to the existing one, just with volumeBindingMode set to WaitForFirstConsumer (and with a new storage class name), that works fine. I can either set that as the default class with annotations, or I can set the KD config to explicitly use that class, and everything works.

If Amazon isn't going to set that by default then we need to figure out the best way to address it for KD users. Obviously we can/should document this issue -- maybe create a whole "eks-notes.md" in the docs -- but what else? Should KD reject attempts to use persistent storage if the designated class doesn't have this field setting (but then we have to detect whether we are on Amazon)?

[joel-bluedata]

Plan is to document this with an eks-notes thing. We'll give explicit steps for setting up and using the new storage class.