Adjust update strategy for NNF DaemonSets

[bdevcich]

From @behlendorf:

It took a significant amount of time for it to kill and restart all the pods when I added in the merced filesystem since it ran through it sequentially. Thankfully that's a one time thing, but it seems it will make redeploying slow.

When you add/remove a lustrefilesystem resource, the nnf-dm-manager-controller sees that and then adds/removes a Volume and VolumeMount to the nnf-dm-worker DaemonSet. Kubernetes then handles it from there and restarts the nnf-dm-worker pods on each rabbit to mount/umount that filesystem change. The DaemonSet defines this for the updateStrategy:

 updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate

That maxUnavailable: 1 is what is causing the sequential behavior. We'll need to tweak this.

[bdevcich]

We need to consider the same for lusre-csi-driver. There might be other areas for this as well.

[bdevcich]

@behlendorf, I'm making changes to set this to a sane default for our 3 daemonsets:

• nnf-dm-worker
• lustre-csi-node
• nnf-node-manager

maxUnavailable can be set to a number of nodes/pods or a percentage. Setting it to 100% would cause it to attempt a restart on all the nodes at the same time, 50% would do half, 25% a quarter, etc. Do you have a preference on what percentage (or hard number) should be?

This value will be adjustable for each system.

[behlendorf]

@bdevcich as an initial swag how about 25%. This seems like it may be a reasonable compromise between propagating the changes rapidly and potentially overwhelming the system / container repository / other. Then we can tune on a per system basis, and revisit the default as needed.

[bdevcich]

@bdevcich as an initial swag how about 25%. This seems like it may be a reasonable compromise between propagating the changes rapidly and potentially overwhelming the system / container repository / other. Then we can tune on a per system basis, and revisit the default as needed.

Perfect, that's the percentage that I've been playing around with in my testing.

[bdevcich]

PRs here to default these all to 25%:

• Configure the DS to use a rolling update strategy of 25% HewlettPackard/lustre-csi-driver#62
• Configure the NNF Node Manager DS to use a rolling update strategy of 25% nnf-sos#262
• Add UpdateStrategy to DMM for the dm-worker DS and use 25% nnf-dm#155

For nnf-dm, the NnfDataMovementManager resource is edited rather than the DaemonSet directly. The manager is responsibe for managing the DaemonSet.

[bdevcich]

@behlendorf I am comfortable closing this issue after we implemented this manually today on El Cap. Do you agree?

[behlendorf]

Yup, things are looking much better after these changes.