CORTX-30684: change retain policy to Delete for local-path storage

[keithpine]

Description

Change the reclaimPolicy of the local path StorageClass to Delete, from Retain. Note that Delete is the default policy for dynamic provisioner storage class that doesn't specify any policy, and also the default for the Local Path Provisioner.

Using a Retain policy means the cluster administrator needs to manually clean-up the backing data for k8s volumes, after a user deletes their PVCs and PVs. In development usage the local node partitions can easily run out of space without active management, causing deployments to fail. In comparison, if the policy is Delete, the provisioner will automatically remove the backing data on the nodes when the PVCs are deleted (and PVs are unbound). This is the behavior that makes sense as a default for users of the deployment scripts.

It is no longer necessary to manually remove PVs since the provisioner does this itself, in fact doing this at the same time causes the provisioner to be unable to removing the node data.

Breaking change

It's useful for users to be aware of this change to the default behavior. Running the destroy script will now result in all data being removed from the cluster nodes (although, that is really the intent).

The destroy-cortx-cloud.sh script no longer manually removes 3rd party PVs. This means running the script against a cluster where the PV reclaim policies are Retain, the PVs will not be removed. Users should manually delete the PVs and data on the cluster nodes.

Type of change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds new functionality)
• Breaking change (bug fix or new feature that breaks existing functionality)
• Third-party dependency update
• Documentation additions or improvements
• Code quality improvements to existing code or test additions/updates

Applicable issues

• This change address an issue: CORTX-30684
• This change is related to an issue: CORTX-31536

In debugging 31356 I wanted to ensure that no left over Pod data was influencing repeat runs. Switching to the Delete reclaim policy makes sure the left-over data is automatically cleaned up.

How was this tested?

Deploy and destroy. I see all node data removed after the destroy.

Saved files into S3 bucket, confirmed after stopping and starting the cluster (stop/start scripts) the data is still accessible.

Additional information

If users prefer to keep the Retain policy, PVs can be modified after they are created. https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/

Note that future plans include allowing the use of any StorageClass, not just our specific local-path provisioner. In that case, users are free to define the StorageClass any way they prefer, and clean-up PVs and data as required.

Checklist

• The change is tested and works locally.
• New or changed settings in the solution YAML are documented clearly in the README.md file.
• All commits are signed off and are in agreement with the CORTX Community DCO and CLA policy.

If this change addresses a CORTX Jira issue:

• The title of the PR starts with the issue ID (e.g. CORTX-XXXXX:)

[keithpine]

I did a loop of deploy/destroy. I've noticed that some of the data directories are still left over. In the local-path-provisioner log, I see some messages like:

E0523 20:42:58.647690       1 controller.go:991] volume "pvc-94f74a9f-f308-473a-bd88-705ca7f64e5a" in work queue no longer exists

pvc-94f74a9f-f308-473a-bd88-705ca7f64e5a was the PV which corresponds to a left-over data directory.

In the destroy script, we manually delete the PVs:

Removing pvc-94f74a9f-f308-473a-bd88-705ca7f64e5a
persistentvolume "pvc-94f74a9f-f308-473a-bd88-705ca7f64e5a" deleted

My guess is that deleting the volume manually is confusing the provisioner, and it doesn't handle a volume being removed while it's in its work queue.

[osowski]

based on everything else we've seen and have in plan, this lgtm.

Needs re-review

[osowski]

I did a loop of deploy/destroy. I've noticed that some of the data directories are still left over. In the local-path-provisioner log, I see some messages like:

E0523 20:42:58.647690       1 controller.go:991] volume "pvc-94f74a9f-f308-473a-bd88-705ca7f64e5a" in work queue no longer exists

pvc-94f74a9f-f308-473a-bd88-705ca7f64e5a was the PV which corresponds to a left-over data directory.

In the destroy script, we manually delete the PVs:

Removing pvc-94f74a9f-f308-473a-bd88-705ca7f64e5a
persistentvolume "pvc-94f74a9f-f308-473a-bd88-705ca7f64e5a" deleted

My guess is that deleting the volume manually is confusing the provisioner, and it doesn't handle a volume being removed while it's in its work queue.

As we're moving towards handling as much k8s functionality as possible, this makes sense to me to remove the manual removal of PVs when we have the underlying StorageClass and dynamic provisioner taking care of some of that. It will help with separation of roles, as long as we're able to still run atomic deploy/destroy loops.