From 52bd01f9cb0162e953cf7b32cd9d9de940969b5b Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 08:40:35 -0300 Subject: [PATCH 01/49] Create _index.html --- content/docs/prerequisites/_index.html | 1 + 1 file changed, 1 insertion(+) create mode 100644 content/docs/prerequisites/_index.html diff --git a/content/docs/prerequisites/_index.html b/content/docs/prerequisites/_index.html new file mode 100644 index 0000000000..8b13789179 --- /dev/null +++ b/content/docs/prerequisites/_index.html @@ -0,0 +1 @@ + From c2bc1a16f96e2791faf38c66518eedd35737f650 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 11:19:46 -0300 Subject: [PATCH 02/49] Added CSM prerequisites --- content/docs/prerequisites/_index.html | 33 ++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/content/docs/prerequisites/_index.html b/content/docs/prerequisites/_index.html index 8b13789179..b6e670f01a 100644 --- a/content/docs/prerequisites/_index.html +++ b/content/docs/prerequisites/_index.html @@ -1 +1,34 @@ +--- +title: "Prerequisites" +linkTitle: "Prerequisites" +description: Prerequisites for CSM +weight: 3 +--- +## Supported Container Orchestrator Platforms + +{{}} +| Platform | Version | +|---------------|:----------------:| +| Kubernetes | 1.27, 1.28, 1.29 | +| Red Hat OpenShift | 4.13, 4.14 | +| Mirantis Kubernetes Engine | 3.7.x | +| Google Anthos | 1.15 | +| VMware Tanzu | ?? | +| Rancher Kubernetes Engine | 1.4.x| +| Amazon Elastic Kubernetes Service
Anywhere | ?? | +| OS dependencies | iscsi-initiator-utils
multipathd or powerpath
nvme-cli
nfs-utils | [SDC](https://www.dell.com/support/home/en-us/product-support/product/scaleio/drivers) | iscsi-initiator-utils
multipathd
nfs-utils | nfs-utils | iscsi-initiator-utils
multipathd
nvme-cli
nfs-utils | +{{
}} + +## Supported Storage Platforms + +{{}} +| Platform | Version | OS Dependencies | +|---------------|:----------------:|:------------------------:| +| PowerMax | PowerMax 2500/8500 PowerMaxOS 10 (6079) , PowerMaxOS 10.0.1 (6079) , PowerMaxOS 10.1 (6079), PowerMax 2000/8000 - 5978.711.xxx, 5978.479.xxx, Unisphere 10.0,10.0.1,10.1 | iscsi-initiator-utils
multipathd or powerpath
nvme-cli
nfs-utils | +| PowerFlex | 3.6.x, 4.0.x, 4.5.x | [SDC](https://www.dell.com/support/home/en-us/product-support/product/scaleio/drivers)| +| Unity XT | 5.1.x, 5.2.x, 5.3.0 | iscsi-initiator-utils
multipathd
nfs-utils | +| PowerScale | OneFS 9.3, 9.4, 9.5.0.x (x >= 5) | nfs-utils | +| PowerStore | 3.0, 3.2, 3.5 | iscsi-initiator-utils
multipathd
nvme-cli
nfs-utils | +| Rancher Kubernetes Engine | 1.4.x| +{{
}} From 8db1cee1e5a301c4a51ba8f7ebec6c41e950fafe Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 11:20:51 -0300 Subject: [PATCH 03/49] Update _index.html --- content/docs/prerequisites/_index.html | 1 - 1 file changed, 1 deletion(-) diff --git a/content/docs/prerequisites/_index.html b/content/docs/prerequisites/_index.html index b6e670f01a..788d9ad7b2 100644 --- a/content/docs/prerequisites/_index.html +++ b/content/docs/prerequisites/_index.html @@ -17,7 +17,6 @@ | VMware Tanzu | ?? | | Rancher Kubernetes Engine | 1.4.x| | Amazon Elastic Kubernetes Service
Anywhere | ?? | -| OS dependencies | iscsi-initiator-utils
multipathd or powerpath
nvme-cli
nfs-utils | [SDC](https://www.dell.com/support/home/en-us/product-support/product/scaleio/drivers) | iscsi-initiator-utils
multipathd
nfs-utils | nfs-utils | iscsi-initiator-utils
multipathd
nvme-cli
nfs-utils | {{}} ## Supported Storage Platforms From dbb12af820b88cd859b0b045e149d2f5cb97bcda Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 13:35:20 -0300 Subject: [PATCH 04/49] Added notes --- content/docs/prerequisites/_index.html | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/content/docs/prerequisites/_index.html b/content/docs/prerequisites/_index.html index 788d9ad7b2..56a25e746a 100644 --- a/content/docs/prerequisites/_index.html +++ b/content/docs/prerequisites/_index.html @@ -19,6 +19,10 @@ | Amazon Elastic Kubernetes Service
Anywhere | ?? | {{}} +> Notes: +> * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. +> * Any orchestrator platform or version that's not mentioned here must be self-certified using Cert-CSI in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. + ## Supported Storage Platforms {{}} @@ -31,3 +35,6 @@ | PowerStore | 3.0, 3.2, 3.5 | iscsi-initiator-utils
multipathd
nvme-cli
nfs-utils | | Rancher Kubernetes Engine | 1.4.x| {{
}} + +> Notes: +> * The required OS dependencies are only for the protocol needed (e.g. if NVMe isn't the storage access protocol then nvme-cli is not required). From 8117d1cad416e99a31ea4d5d21158a628f2a1953 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 13:39:41 -0300 Subject: [PATCH 05/49] Added cert-csi doc link --- content/docs/prerequisites/_index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/prerequisites/_index.html b/content/docs/prerequisites/_index.html index 56a25e746a..9da4600fdd 100644 --- a/content/docs/prerequisites/_index.html +++ b/content/docs/prerequisites/_index.html @@ -21,7 +21,7 @@ > Notes: > * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. -> * Any orchestrator platform or version that's not mentioned here must be self-certified using Cert-CSI in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. +> * Any orchestrator platform or version that's not mentioned here must be self-certified using [Cert-CSI](https://dell.github.io/csm-docs/docs/csidriver/installation/test/certcsi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. ## Supported Storage Platforms From fc9188ddf167b687031082ca9f9c7afcc01cacaf Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 13:40:48 -0300 Subject: [PATCH 06/49] Removed support matrix from csi drivers section --- content/docs/csidriver/_index.md | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/content/docs/csidriver/_index.md b/content/docs/csidriver/_index.md index 8a4c09fafd..17946d9188 100644 --- a/content/docs/csidriver/_index.md +++ b/content/docs/csidriver/_index.md @@ -12,25 +12,6 @@ The CSI Drivers by Dell implement an interface between [CSI](https://kubernetes- ## Features and capabilities -### Supported Container Orchestrator Platforms - -{{}} -| | PowerMax | PowerFlex | Unity XT | PowerScale | PowerStore | -|---------------|:----------------:|:-------------------:|:----------------:|:-----------------:|:----------------:| -| Kubernetes | 1.27, 1.28, 1.29 | 1.27, 1.28, 1.29 | 1.27, 1.28, 1.29 | 1.27, 1.28, 1.29 | 1.27, 1.28, 1.29 | -| Red Hat OpenShift | 4.13, 4.14 | 4.13, 4.14 | 4.13, 4.14 | 4.13, 4.14 | 4.13, 4.14 | -| Mirantis Kubernetes Engine | 3.7.x | 3.7.x | 3.7.x | 3.7.x | 3.7.x | -| Google Anthos | 1.15 | 1.15 | no | 1.15 | 1.15 | -| VMware Tanzu | no | no | NFS | NFS | NFS,iSCSI | -| Rancher Kubernetes Engine | 1.4.x| 1.4.x | 1.4.x | 1.4.x | 1.4.x | -| Amazon Elastic Kubernetes Service
Anywhere | yes | yes | yes | yes | yes | -| OS dependencies | iscsi-initiator-utils
multipathd or powerpath
nvme-cli
nfs-utils | [SDC](https://www.dell.com/support/home/en-us/product-support/product/scaleio/drivers) | iscsi-initiator-utils
multipathd
nfs-utils | nfs-utils | iscsi-initiator-utils
multipathd
nvme-cli
nfs-utils | -{{
}} - -> Notes: -> * The required OS dependencies are only for the protocol needed (e.g. if NVMe isn't the storage access protocol then nvme-cli is not required). -> * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. - ### CSI Driver Capabilities {{}} | Features | PowerMax | PowerFlex | Unity XT | PowerScale | PowerStore | From 5a349b8f8c95f54d14bae21c7f9914b4782b26a5 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 13:41:28 -0300 Subject: [PATCH 07/49] Removed support matrix --- content/docs/applicationmobility/_index.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/content/docs/applicationmobility/_index.md b/content/docs/applicationmobility/_index.md index 930c02792a..b7174489fa 100644 --- a/content/docs/applicationmobility/_index.md +++ b/content/docs/applicationmobility/_index.md @@ -28,11 +28,3 @@ After a backup has been created, it can be restored on the same Kubernetes clust |-|-| | Restic | Persistent Volume data will be stored in the provided object store bucket | {{
}} - -## Supported Container Orchestrator Platforms -{{}} -| COP/OS | Supported Versions | -|-|-| -| Kubernetes | 1.26, 1.27. 1.28 | -| Red Hat OpenShift | 4.13, 4.14 | -{{
}} From f143886b85c0af599e1c5c5adec6f2a2225150d1 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 13:45:55 -0300 Subject: [PATCH 08/49] Removed orchestrator and platform support matrix --- content/docs/authorization/_index.md | 30 ++-------------------------- 1 file changed, 2 insertions(+), 28 deletions(-) diff --git a/content/docs/authorization/_index.md b/content/docs/authorization/_index.md index 4f9c019f3b..04dc1f89d4 100644 --- a/content/docs/authorization/_index.md +++ b/content/docs/authorization/_index.md @@ -29,38 +29,12 @@ The following diagram shows a high-level overview of CSM for Authorization with **NOTE:** PowerScale OneFS implements its own form of Role-Based Access Control (RBAC). CSM for Authorization does not enforce any role-based restrictions for PowerScale. To configure RBAC for PowerScale, refer to the PowerScale OneFS [documentation](https://www.dell.com/support/home/en-us/product-support/product/isilon-onefs/docs). -## Supported Operating Systems/Container Orchestrator Platforms -{{}} -| COP/OS | Supported Versions | -|-|-| -| Kubernetes | 1.26, 1.27, 1.28 | -{{
}} - -## Supported Storage Platforms - -{{}} -| | PowerMax | PowerFlex | PowerScale | -|---------------|:----------------:|:-------------------:|:----------------:| -| Storage Array |PowerMax 2500/8500 PowerMaxOS 10 (6079) , PowerMaxOS 10.0.1 (6079) , PowerMaxOS 10.1 (6079)
PowerMax 2000/8000 - 5978.711.xxx, 5978.479.xxx,
Unisphere 10.0, 10.0.1, 10.1| 3.6.x, 4.0.x, 4.5 | OneFS 9.5.0.x (x >= 5) | -{{
}} - -## Supported CSI Drivers - -CSM for Authorization supports the following CSI drivers and versions. -{{}} -| Storage Array | CSI Driver | Supported Versions | -| ------------- | ---------- | ------------------ | -| CSI Driver for Dell PowerFlex | [csi-powerflex](https://github.com/dell/csi-powerflex) | v2.0 + | -| CSI Driver for Dell PowerMax | [csi-powermax](https://github.com/dell/csi-powermax) | v2.0 + | -| CSI Driver for Dell PowerScale | [csi-powerscale](https://github.com/dell/csi-powerscale) | v2.0 + | -{{
}} +## Authorization Components Support Matrix +CSM for Authorization consists of 2 components - The authorization sidecar, bundled with the driver, communicates with the Authorization proxy server to validate access to Storage platforms. The authorization sidecar is backward compatible with older Authorization proxy server versions. However, it is highly recommended to have the Authorization proxy server and sidecar installed from the same release of CSM. **NOTE:** If the deployed CSI driver has a number of controller pods equal to the number of schedulable nodes in your cluster, CSM for Authorization may not be able to inject properly into the driver's controller pod. To resolve this, please refer to our [troubleshooting guide](./troubleshooting) on the topic. -## Authorization Components Support Matrix -CSM for Authorization consists of 2 components - The authorization sidecar, bundled with the driver, communicates with the Authorization proxy server to validate access to Storage platforms. The authorization sidecar is backward compatible with older Authorization proxy server versions. However, it is highly recommended to have the Authorization proxy server and sidecar installed from the same release of CSM. - ## Roles and Responsibilities The CSM for Authorization CLI can be executed in the context of the following roles: From eb69f6a42311584e86f1af7df0b1de227b94da8e Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 13:53:57 -0300 Subject: [PATCH 09/49] Added ObjectScale and note about Auth support --- content/docs/prerequisites/_index.html | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/docs/prerequisites/_index.html b/content/docs/prerequisites/_index.html index 9da4600fdd..59edc3f8e4 100644 --- a/content/docs/prerequisites/_index.html +++ b/content/docs/prerequisites/_index.html @@ -33,8 +33,9 @@ | Unity XT | 5.1.x, 5.2.x, 5.3.0 | iscsi-initiator-utils
multipathd
nfs-utils | | PowerScale | OneFS 9.3, 9.4, 9.5.0.x (x >= 5) | nfs-utils | | PowerStore | 3.0, 3.2, 3.5 | iscsi-initiator-utils
multipathd
nvme-cli
nfs-utils | -| Rancher Kubernetes Engine | 1.4.x| +| ObjectScale | 1.2.x | - | {{}} > Notes: > * The required OS dependencies are only for the protocol needed (e.g. if NVMe isn't the storage access protocol then nvme-cli is not required). +> * [CSM Authorization](https://dell.github.io/csm-docs/docs/authorization/) only supports Kubernetes and the following Storage Platforms: PowerMax, PowerFlex, and PowerScale From a16630ce8a516676931bb976cdf1407a8cb4e710 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 13:56:06 -0300 Subject: [PATCH 10/49] Removed support matrix --- content/docs/observability/_index.md | 30 ---------------------------- 1 file changed, 30 deletions(-) diff --git a/content/docs/observability/_index.md b/content/docs/observability/_index.md index 1daf0bee98..d7ff9b4920 100644 --- a/content/docs/observability/_index.md +++ b/content/docs/observability/_index.md @@ -43,36 +43,6 @@ CSM for Observability provides the following capabilities: | Grafana Dashboards for displaying metrics and topology data | yes | yes | no | yes | yes | {{}} -## Supported Operating Systems/Container Orchestrator Platforms - -{{}} -| COP/OS | Supported Versions | -|-|-| -| Kubernetes | 1.27, 1.28, 1.29 | -| Red Hat OpenShift | 4.13, 4.14 | -| Rancher Kubernetes Engine | yes | -{{
}} - -## Supported Storage Platforms - -{{}} -| | PowerFlex | PowerStore | PowerScale | PowerMax | -|---------------|:-------------------:|:----------------:|:----------------:|:----------------:| -| Storage Array | 3.6.x, 4.0.x, 4.5 | 3.0, 3.2, 3.5, 3.6 | OneFS 9.3, 9.4, 9.5.0.x (x >= 5) | PowerMax 2000/8000
PowerMax 2500/8500 PowerMaxOS 10 (6079) , PowerMaxOS 10.0.1 (6079) , PowerMaxOS 10.1 (6079)
PowerMax 2000/8000 - 5978.711.xxx, 5978.479.xxx
Unisphere 10.0, 10.0.1, 10.1 | -{{
}} - -## Supported CSI Drivers - -CSM for Observability supports the following CSI drivers and versions. -{{}} -| Storage Array | CSI Driver | Supported Versions | -| ------------- | ---------- | ------------------ | -| CSI Driver for Dell PowerFlex | [csi-powerflex](https://github.com/dell/csi-powerflex) | v2.0 + | -| CSI Driver for Dell PowerStore | [csi-powerstore](https://github.com/dell/csi-powerstore) | v2.0 + | -| CSI Driver for Dell PowerScale | [csi-powerscale](https://github.com/dell/csi-powerscale) | v2.0 + | -| CSI Driver for Dell PowerMax | [csi-powermax](https://github.com/dell/csi-powermax) | v2.5 + | -{{
}} - ## Topology Data CSM for Observability provides Kubernetes administrators with the topology data related to containerized storage. This topology data is visualized using Grafana: From 30a6092d988990316560f3c98f1d4422f264426b Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 13:57:34 -0300 Subject: [PATCH 11/49] Added Observability note --- content/docs/prerequisites/_index.html | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/docs/prerequisites/_index.html b/content/docs/prerequisites/_index.html index 59edc3f8e4..7a9230e3ff 100644 --- a/content/docs/prerequisites/_index.html +++ b/content/docs/prerequisites/_index.html @@ -38,4 +38,5 @@ > Notes: > * The required OS dependencies are only for the protocol needed (e.g. if NVMe isn't the storage access protocol then nvme-cli is not required). -> * [CSM Authorization](https://dell.github.io/csm-docs/docs/authorization/) only supports Kubernetes and the following Storage Platforms: PowerMax, PowerFlex, and PowerScale +> * [CSM Authorization](https://dell.github.io/csm-docs/docs/authorization/) only supports Kubernetes and the following Storage Platforms: PowerMax, PowerFlex, and PowerScale. +> * [CSM Observability](https://dell.github.io/csm-docs/docs/observability/) does not support the Unity XT Storage Platform From dad13be0c5d3453e7977e51fceb128d8fc5dab85 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 13:59:30 -0300 Subject: [PATCH 12/49] Removed support matrix --- content/docs/replication/_index.md | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/content/docs/replication/_index.md b/content/docs/replication/_index.md index be893e5fad..4156622663 100644 --- a/content/docs/replication/_index.md +++ b/content/docs/replication/_index.md @@ -32,25 +32,6 @@ CSM for Replication provides the following capabilities: | Provides a command line utility - [repctl](tools) for configuring & managing replication related resources across multiple clusters | yes | yes | yes | yes | no | {{}} -## Supported Operating Systems/Container Orchestrator Platforms - -{{}} -| COP/OS | PowerMax | PowerStore | PowerScale | PowerFlex | -| ----------------- | ---------------- | ---------------- | ---------------- | ---------------- | -| Kubernetes | 1.26, 1.27, 1.28 | 1.26, 1.27, 1.28 | 1.26, 1.27, 1.28 | 1.26, 1.27, 1.28 | -| Red Hat OpenShift | 4.13, 4.14 | 4.13, 4.14 | 4.13, 4.14 | 4.13, 4.14 | -{{
}} - -## Supported Storage Platforms - -{{}} -| | PowerMax | PowerStore | PowerScale | PowerFlex | -| ------------- | :------------------------------------------------------------------------------------------------------: | :------------------------------------------: | :------------------------------------------: | :-------: | -| Storage Array | PowerMax 2500/8500 PowerMaxOS 10 (6079) , PowerMaxOS 10.0.1 (6079) , PowerMaxOS 10.1 (6079)
PowerMax 2000/8000 - 5978.711.xxx, 5978.479.xxx
Unisphere 10.0,10.0.1,10.1 | 3.0, 3.2, 3.5, 3.6 | OneFS 9.3, 9.4, 9.5.0.x (x >=5) | 3.6.x, 4.0.x, 4.5 | -{{
}} - ->Note: File Replication for PowerMax is currently not supported - ## Details As on the storage arrays, all replication related Kubernetes entities are required/created in pairs - From 00ea9e5e031bbf9aa6271d42580b812552e14c4f Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 14:00:41 -0300 Subject: [PATCH 13/49] Added replication note --- content/docs/prerequisites/_index.html | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/docs/prerequisites/_index.html b/content/docs/prerequisites/_index.html index 7a9230e3ff..d1a3c6b71a 100644 --- a/content/docs/prerequisites/_index.html +++ b/content/docs/prerequisites/_index.html @@ -39,4 +39,5 @@ > Notes: > * The required OS dependencies are only for the protocol needed (e.g. if NVMe isn't the storage access protocol then nvme-cli is not required). > * [CSM Authorization](https://dell.github.io/csm-docs/docs/authorization/) only supports Kubernetes and the following Storage Platforms: PowerMax, PowerFlex, and PowerScale. -> * [CSM Observability](https://dell.github.io/csm-docs/docs/observability/) does not support the Unity XT Storage Platform +> * [CSM Observability](https://dell.github.io/csm-docs/docs/observability/) does not support the Unity XT Storage Platform. +> * [CSM Replication](https://dell.github.io/csm-docs/docs/replication/) does not support the Unity XT Storage Platform or File Replication for PowerMax. From dadb3b5fdeaa91b2d000aa2a3fef7f7be4fd06ed Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 14:39:40 -0300 Subject: [PATCH 14/49] Create cert-csi page --- content/docs/cert-csi/_index.html | 843 ++++++++++++++++++++++++++++++ 1 file changed, 843 insertions(+) create mode 100644 content/docs/cert-csi/_index.html diff --git a/content/docs/cert-csi/_index.html b/content/docs/cert-csi/_index.html new file mode 100644 index 0000000000..44a66a6efc --- /dev/null +++ b/content/docs/cert-csi/_index.html @@ -0,0 +1,843 @@ +--- +title: Cert-CSI +linktitle: Cert-CSI +description: Tool to validate Dell CSI Drivers +--- + +Cert-CSI is a tool to validate Dell CSI Drivers. It contains various test suites to validate the drivers. + +## Installation + +There are three methods of installing `cert-csi`. + +1. [Download the executable from the latest GitHub release](#download-release-linux). +2. [Pull the container image from DockerHub](#pull-the-container-image). +3. [Build the exectuable or container image locally](#building-locally). + +> The exectuable from the GitHub Release only supports Linux. For non-Linux users, you must build the `cert-csi` executable [locally](#building-locally). + +### Download Release (Linux) + +1. Download the latest release of the cert-csi zip file. + +```bash +curl -LO https://github.com/dell/cert-csi/releases/download/v1.3.1/cert-csi-v1.3.1.zip +``` + +2. Unzip the file. + +``` bash +unzip cert-csi-v1.3.1.zip +chmod +x ./cert-csi-v1.3.1 +``` + +3. Install cert-csi-v1.3.1 as cert-csi. + +```bash +sudo install -o root -g root -m 0755 cert-csi-v1.3.1 /usr/local/bin/cert-csi +``` + +If you do not have root access on the target system, you can still install cert-csi to the ~/.local/bin directory: + +```bash +chmod +x cert-csi-v1.3.1 +mkdir -p ~/.local/bin +mv ./cert-csi-v1.3.1 ~/.local/bin/cert-csi +# and then append (or prepend) ~/.local/bin to $PATH +``` + +### Pull The Container Image + + {{< tabs name="pulling-cert-csi-image" >}} + {{% tab name="Docker" %}} + + ```bash + docker pull dellemc/cert-csi:v1.3.1 + ``` + + {{% /tab %}} + {{% tab name="Podman" %}} + + ```bash + podman pull dellemc/cert-csi:v1.3.1 + ``` + + {{% /tab %}} + {{< /tabs >}} + +### Building Locally +#### Prerequisites +- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +- [Go](https://go.dev/doc/install) (If buidling the executable) +- Podman or Docker (If building the container image) + +1. Clone the repository + +```bash +git clone -b "v1.3.1" https://github.com/dell/cert-csi.git && cd cert-csi +``` + +2. Build cert-csi + +{{< tabs name="build-cert-csi" >}} +{{% tab name="Executable" %}} + +```bash + make build # the cert-csi executable will be in the working directory + chmod +x ./cert-csi # if building on *nix machine +``` + +{{% /tab %}} +{{% tab name="Container Image" %}} + +```bash + # uses podman if available, otherwise uses docker. The resulting image is tagged cert-csi:latest + make docker +``` + +{{% /tab %}} +{{< /tabs >}} + +### Optional + +If you want to collect csi-driver resource usage metrics, then please provide the namespace where it can be found and install the metric-server using this command (kubectl is required): + +```bash +make install-ms +``` + +## Running Cert-CSI + +{{< tabs name="running-cert-csi" >}} +{{% tab name="Executable" %}} +```bash + cert-csi --help +``` +{{% /tab %}} +{{% tab name="Docker" %}} +```bash + docker run --rm -it -v ~/.kube/config:/root/.kube/config dellemc/cert-csi:v1.3.1 --help +``` +{{% /tab %}} +{{% tab name="Podman" %}} +```bash + podman run --rm -it -v ~/.kube/config:/root/.kube/config dellemc/cert-csi:v1.3.1 --help +``` + +{{% /tab %}} +{{< /tabs >}} + +> The following sections showing how to execute the various test suites use the executable for brevity. For executions requiring special behavior, such as mounting file arguments into the container image, it will be noted for the relevant command. + +> Log files are located in the `logs` directory in the working directory of cert-csi.\ +> Report files are located in the default `$HOME/.cert-csi/reports` directory.\ +> Database (SQLite) file for test suites is `.db` in the working directory of cert-csi.\ +> Database (SQLite) file for functional test suites is `cert-csi-functional.db` in the working directory of cert-csi. + +> NOTE: If using the container image, these files will be inside the container. If you are interested in these files, it is recommended to use the exectuable. + +## Run All Test Suites + +You can use cert-csi to launch a test run against multiple storage classes to check if the driver adheres to advertised capabilities. + +### Preparing Config + +To run the test suites you need to provide `.yaml` config with storage classes and their capabilities. You can use `example-certify-config.yaml` as an example. + +Template: +```yaml +storageClasses: + - name: # storage-class-name (ex. powerstore) + minSize: # minimal size for your sc (ex. 1Gi) + rawBlock: # is Raw Block supported (true or false) + expansion: # is volume expansion supported (true or false) + clone: # is volume cloning supported (true or false) + snapshot: # is volume snapshotting supported (true or false) + RWX: # is ReadWriteMany volume access mode supported for non RawBlock volumes (true or false) + volumeHealth: # set this to enable the execution of the VolumeHealthMetricsSuite (true or false) + # Make sure to enable healthMonitor for the driver's controller and node pods before running this suite. It is recommended to use a smaller interval time for this sidecar and pass the required arguments. + VGS: # set this to enable the execution of the VolumeGroupSnapSuite (true or false) + # Additionally, make sure to provide the necessary required arguments such as volumeSnapshotClass, vgs-volume-label, and any others as needed. + RWOP: # set this to enable the execution of the MultiAttachSuite with the AccessMode set to ReadWriteOncePod (true or false) + ephemeral: # if exists, then run EphemeralVolumeSuite. See the Ephemeral Volumes suite section for example Volume Attributes + driver: # driver name for EphemeralVolumeSuite (e.g., csi-vxflexos.dellemc.com) + fstype: # fstype for EphemeralVolumeSuite + volumeAttributes: # volume attrs for EphemeralVolumeSuite. + attr1: # volume attr for EphemeralVolumeSuite + attr2: # volume attr for EphemeralVolumeSuite + capacityTracking: + driverNamespace: # namepsace where driver is installed + pollInterval: # duration to poll capacity (e.g., 2m) +``` + +Driver specific examples: + + {{< tabs name="cerity-config-examples" >}} + {{% tab name="CSI PowerFlex" %}} + +```yaml +storageClasses: + - name: vxflexos + minSize: 8Gi + rawBlock: true + expansion: true + clone: true + snapshot: true + RWX: false + ephemeral: + driver: csi-powerstore.dellemc.com + fstype: ext4 + volumeAttributes: + volumeName: "my-ephemeral-vol" + size: "8Gi" + storagepool: "sample" + systemID: "sample" + - name: vxflexos-nfs + minSize: 8Gi + rawBlock: false + expansion: true + clone: true + snapshot: true + RWX: true + RWOP: true + ephemeral: + driver: csi-vxflexos.dellemc.com + fstype: "nfs" + volumeAttributes: + volumeName: "my-ephemeral-vol" + size: "8Gi" + storagepool: "sample" + systemID: "sample" + capacityTracking: + driverNamespace: powerstore + pollInterval: 2m +``` + + {{% /tab %}} + {{% tab name="CSI PowerScale" %}} + +```yaml +storageClasses: + - name: isilon + minSize: 8Gi + rawBlock: false + expansion: true + clone: true + snapshot: true + RWX: false + ephemeral: + driver: csi-isilon.dellemc.com + fstype: nfs + volumeAttributes: + size: "10Gi" + ClusterName: "sample" + AccessZone: "sample" + IsiPath: "/ifs/data/sample" + IsiVolumePathPermissions: "0777" + AzServiceIP: "192.168.2.1" +``` + + {{% tab name="CSI PowerMax" %}} + +```yaml +storageClasses: + - name: powermax-iscsi + minSize: 5Gi + rawBlock: true + expansion: true + clone: true + snapshot: true + capacityTracking: + driverNamespace: powerstore + pollInterval: 2m + - name: powermax-nfs + minSize: 5Gi + rawBlock: false + expansion: true + clone: true + snapshot: true + RWX: true + RWOP: true + capacityTracking: + driverNamespace: powerstore + pollInterval: 2m +``` + + {{% /tab %}} + + {{% /tab %}} + {{% tab name="CSI PowerStore" %}} + +```yaml +storageClasses: + - name: powerstore + minSize: 5Gi + rawBlock: true + expansion: true + clone: true + snapshot: true + RWX: false + ephemeral: + driver: csi-powerstore.dellemc.com + fstype: ext4 + volumeAttributes: + arrayID: "arrayid" + protocol: iSCSI + size: 5Gi + - name: powerstore-nfs + minSize: 5Gi + rawBlock: false + expansion: true + clone: true + snapshot: true + RWX: true + RWOP: true + ephemeral: + driver: csi-powerstore.dellemc.com + fstype: "nfs" + volumeAttributes: + arrayID: "arrayid" + protocol: NFS + size: 5Gi + nasName: "nas-server" + capacityTracking: + driverNamespace: powerstore + pollInterval: 2m +``` + + {{% /tab %}} + {{% tab name="CSI Unity" %}} + +```yaml +storageClasses: + - name: unity-iscsi + minSize: 3Gi + rawBlock: true + expansion: true + clone: false + snapshot: true + RWX: false + ephemeral: + driver: csi-unity.dellemc.com + fstype: ext4 + volumeAttributes: + arrayId: "array-id" + storagePool: pool-name + protocol: NFS + size: 5Gi + - name: unity-nfs + minSize: 3Gi + rawBlock: false + expansion: true + clone: false + snapshot: true + RWX: true + RWOP: true + ephemeral: + driver: csi-unity.dellemc.com + fstype: "nfs" + volumeAttributes: + arrayId: "array-id" + storagePool: pool-name + protocol: NFS + size: 5Gi + nasServer: "nas-server" + nasName: "nas-name" + capacityTracking: + driverNamespace: unity + pollInterval: 2m +``` + + {{% /tab %}} + {{< /tabs >}} + +### Launching Test Run +1. Executes the [VolumeIO](#volume-io) suite. +2. Executes the [Scaling](#scalability) suite. +3. If `storageClasses.clone` is `true`, executes the [Volume Cloning](#volume-cloning) suite. +4. If `storageClasses.expansion` is `true`, executes the [Volume Expansion](#volume-expansion) suite. +5. If `storageClasses.expansion` is `true` and `storageClasses.rawBlock` is `true`, executes the [Volume Expansion](#volume-expansion) suite with raw block volumes. +6. If `storageClasses.snapshot` is `true`, exeuctes the [Snapshot](#snapshots) suite and the [Replication](#replication) suite. +7. If `storageClasses.rawBlock` is `true`, executes the [Multi-Attach Volume](#multi-attach-volume) suite with raw block volumes. +8. If `storageClasses.rwx` is `true`, executes the [Multi-Attach Volume](#multi-attach-volume) suite. (Storgae Class must be NFS.) +9. If `storageClasses.volumeHealth` is `true`, executes the [Volume Health Metrics](#volume-health-metrics) suite. +10. If `storageClasses.rwop` is `true`, executes the [Multi-Attach Volume](#multi-attach-volume) suite with the volume access mode `ReadWriteOncePod`. +11. If `storageClasses.ephemeral` exists, executes the [Ephemeral Volumes](#ephemeral-volumes) suite. +12. If `storageClasses.vgs` is `true`, executes the [Volume Group Snapshot]() suite. +13. If `storageClasses.capacityTracking` exists, exeuctes the [Storage Class Capacity Tracking](#storage-capacity-tracking) suite. + +> NOTE: For testing/debugging purposes, it can be useful to use the `--no-cleanup` so resources do not get deleted. + +> NOTE: If you are using CSI PowerScale with [SmartQuotas](../../features/powerscale/#usage-of-smartquotas-to-limit-storage-consumption) disabled, the `Volume Expansion` suite is expected to timeout due to the way PowerScale provisions storage. Set `storageClasses.expansion` to `false` to skip this suite. + +```bash +cert-csi certify --cert-config --vsc +``` + +Withold the `--vsc` argument if Snapshot capabilities are disabled. + +```bash +cert-csi certify --cert-config +Optional Params: + --vsc: volume snapshot class, required if you specified snapshot capability +``` + +Run `cert-csi certify -h` for more options. + +If you are using the container image, the `cert-config` file must be mounted into the container. Assuming your `cert-config` file is `/home/user/example-certify-config.yaml`, here are examples of how to exeucte this suite with the container image. + +{{< tabs name="running-container-certify" >}} +{{% tab name="Docker" %}} +```bash + docker run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/example-certify-config.yaml:/example-certify-config.yaml dellemc/cert-csi:v1.3.1 certify --cert-config /example-certify-config.yaml --vsc +``` +{{% /tab %}} +{{% tab name="Podman" %}} +```bash + podman run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/example-certify-config.yaml:/example-certify-config.yaml dellemc/cert-csi:v1.3.1 certify --cert-config /example-certify-config.yaml --vsc +``` + +{{% /tab %}} +{{< /tabs >}} + +### Running Invidual Test Suites + +> NOTE: For testing/debugging purposes, it can useful to use the `--no-cleanup` flag so resources do not get deleted. + +#### Volume I/O +1. Creates the namespace `volumeio-test-*` where resources will be created. +2. Creates Persistent Volume Claims. +3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claims to be bound to Persistent Volumes. +4. For each Persistent Volume Claim, executes the following workflow concurrently: + 1. Creates a Pod to consume the Persistent Volume Claim. + 2. Writes data to the volume and verifies the checksum of the data. + 3. Deletes the Pod. + 4. Waits for the associated Volume Attachment to be deleted. + +```bash +cert-csi test vio --sc +``` + +Run `cert-csi test vio -h` for more options. + +#### Scalability +1. Creates the namespace `scale-test-*` where resources will be created. +2. Creates a StatefulSet. +3. Scales up the StatefulSet. +4. Scales down the StatefulSet to zero. + +```bash +cert-csi test scaling --sc +``` + +Run `cert-csi test scaling -h` for more options. + +#### Snapshots +1. Creates the namespace `snap-test-*` where resources will be created. +2. Creates Persistent Volume Claim. +3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claim to be bound to Persistent Volumes. +4. Create Pod to consume the Persistent Volume Claim. +5. Writes data to the volume. +6. Deletes the Pod. +7. Creates a Volume Snapshot from the Persistent Volume Claim. +8. Waits for the Volume Snapshot to be Ready. +9. Creates a new Persistent Volume Claim from the Volume Snapshot. +10. Creates a new Pod to consume the new Persistent Volume Claim. +11. Verifies the checksum of the data. + +```bash +cert-csi test snap --sc --vsc +``` + +Run `cert-csi test snap -h` for more options. + +#### Volume Group Snapshots +1. Creates the namespace `vgs-snap-test-*` where resources will be created. +2. Creates Persistent Volume Claims. +3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claim to be bound to Persistent Volumes. +4. Create Pods to consume the Persistent Volume Claims. +5. Creates Volume Group Snapshot. +6. Waits for Volume Group Snapshot state to be COMPLETE. + +> Note: Volume Group Snapshots are only supported by CSI PowerFlex and CSI PowerStore. + +#### Multi-Attach Volume +1. Creates the namespace `mas-test-*` where resources will be created. +2. Creates Persistent Volume Claim. +3. Creates Pod to consume the Persistent Volume Claim. +4. Waits for Pod to be in the Ready state. +5. Creates additional Pods to consume the same Persistent Volume Claim. +6. Watis for Pods to be in the Ready state. +7. Writes data to the volumes on the Pods and verifies checksum of the data. + +```bash +cert-csi test multi-attach-vol --sc +``` + +> The storage class must be an NFS storage class. Otherwise, raw block volumes must be used. + +```bash +cert-csi test multi-attach-vol --sc --block +``` + +Run `cert-csi test multi-attach-vol -h` for more options. + +#### Replication +1. Creates the namespace `replication-suite-*` where resources will be created. +2. Creates Persistent Volume Claims. +3. Create Pods to consume the Persistent Volume Claims. +4. Waits for Pods to be in the Ready state. +5. Creates a Volume Snapshot from each Persistent Volume Claim. +6. Waits for the Volume Snapshots to be Ready. +7. Creates Persistent Volume Claims from the Volume Snapshots. +8. Creates Pods to consume the Persistent Volume Claims. +9. Waits for Pods to be in the Ready state. +10. Verifies the replication group name on ersistent Volume Claims. + +```bash +cert-csi test replication --sc --vsc +``` + +Run `cert-csi test replication -h` for more options. + +#### Volume Cloning +1. Creates the namespace `clonevolume-suite-*` where resources will be created. +2. Creates Persistent Volume Claims. +3. Create Pods to consume the Persistent Volume Claims. +4. Waits for Pods to be in the Ready state. +5. Creates Persistent Volume Claims with the source volume being from the volumes in step 2. +6. Create Pods to consume the Persistent Volume Claims. +7. Waits for Pods to be in the Ready state. + +```bash +cert-csi test clone-volume --sc +``` + +Run `cert-csi test clone-volume -h` for more options. + +#### Volume Expansion +1. Creates the namespace `volume-expansion-suite-*` where resources will be created. +2. Creates Persistent Volume Claims. +3. Create Pods to consume the Persistent Volume Claims. +4. Waits for Pods to be in the Ready state. +5. Expands the size in the Persistent Volume Claims. +6. Verifies that the volumes mounted to the Pods were expanded. + +> Raw block volumes cannot be verified since there is no filesystem. + +> If you are using CSI PowerScale with [SmartQuotas](../../features/powerscale/#usage-of-smartquotas-to-limit-storage-consumption) disabled, the `Volume Expansion` suite is expected to timeout due to the way PowerScale provisions storage. + +```bash +cert-csi test expansion --sc +``` + +Run `cert-csi test expansion -h` for more options. + +#### Blocksnap suite +1. Creates the namespace `block-snap-test-*` where resources will be created. +2. Creates Persistent Volume Claim. +3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claim to be bound to Persistent Volumes. +4. Creates Pod to consume the Persistent Volume Claim. +5. Writes data to the volume. +5. Creates a Volume Snapshot from the Persistent Volume Claim. +6. Waits for the Volume Snapshot to be Ready. +7. Create a Persistent Volume Claim with raw block volume mode from the Volume Snapshot. +8. Creates Pod to consume the Persistent Volume Claim. +9. Mounts the raw block volume and verifes the checksum of the data. + +```bash +cert-csi test blocksnap --sc --vsc +``` + +Run `cert-csi test blocksnap -h` for more options. + +#### Volume Health Metrics +1. Creates the namespace `volume-health-metrics-*` where resources will be created. +2. Creates Persistent Volume Claim. +3. Creates Pod to consume the Persistent Volume Claim. +4. Waits for Pod to be in the Ready state. +4. Veries that ControllerGetVolume and NodeGetVolumeStats are being executed in the controller and node pods, respectively. + +```bash +cert-csi test volumehealthmetrics --sc --driver-ns +``` + +Run `cert-csi test volumehealthmetrics -h` for more options. + +> Note: Make sure to enable healthMonitor for the driver's controller and node pods before running this suite. It is recommended to use a smaller interval time for this sidecar. + +#### Ephemeral Volumes +1. Creates namespace `functional-test` where resources will be created. +2. Creates Pods with one ephemeral inline volume each. +3. Waits for Pods to be in the Ready state. +4. Writes data to the volume on each Pod. +5. Verifies the checksum of the data. + +```bash +cert-csi test ephemeral-volume --driver --attr ephemeral-config.properties +``` + +Run `cert-csi test ephemeral-volume -h` for more options. + +> `--driver` is the name of a CSI Driver from the output of `kubectl get csidriver` (e.g, csi-vxflexos.dellemc.com). +> This suite does not delete resources on success. + +If you are using the container image, the `attr` file must be mounted into the container. Assuming your `attr` file is `/home/user/ephemeral-config.properties`, here are examples of how to exeucte this suite with the container image. + +{{< tabs name="running-container-ephemeral-volume" >}} +{{% tab name="Docker" %}} +```bash + docker run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/ephemeral-config.properties:/ephemeral-config.properties dellemc/cert-csi:v1.3.1 test ephemeral-volume --driver --attr /ephemeral-config.properties +``` +{{% /tab %}} +{{% tab name="Podman" %}} +```bash + podman run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/ephemeral-config.properties:/ephemeral-config.properties dellemc/cert-csi:v1.3.1 test ephemeral-volume --driver --attr /ephemeral-config.properties +``` + +{{% /tab %}} +{{< /tabs >}} + +Sample ephemeral-config.properties (key/value pair) + {{< tabs name="volume-attributes-examples" >}} + {{% tab name="CSI PowerFlex" %}} + + ```yaml + volumeName: "my-ephemeral-vol" + size: "10Gi" + storagepool: "sample" + systemID: "sample" + ``` + + {{% /tab %}} + {{% tab name="CSI PowerScale" %}} + + ```yaml + size: "10Gi" + ClusterName: "sample" + AccessZone: "sample" + IsiPath: "/ifs/data/sample" + IsiVolumePathPermissions: "0777" + AzServiceIP: "192.168.2.1" + ``` + + {{% /tab %}} + {{% tab name="CSI PowerStore" %}} + + ```yaml + size: "10Gi" + arrayID: "sample" + nasName: "sample" + nfsAcls: "0777" + ``` + + {{% /tab %}} + {{% tab name="CSI Unity" %}} + + ```yaml + size: "10Gi" + arrayID: "sample" + protocol: iSCSI + thinProvisioned: "true" + isDataReductionEnabled: "false" + tieringPolicy: "1" + storagePool: pool_2 + nasName: "sample" + ``` + + {{% /tab %}} + {{< /tabs >}} + +#### Storage Capacity Tracking +1. Creates namespace `functional-test` where resources will be created. +2. Creates a duplicate of the provided storge class using prefix `capacity-tracking`. +3. Waits for the associated CSIStorageCapacity object to be created. +4. Deletes the duplicate storge class. +5. Waits for the associated CSIStorageCapacity to be deleted. +6. Sets the capacity of the CSIStorageCapacity of the provided storage class to zero. +7. Creates Pod with a volume using the provided storage class. +8. Verifies that the Pod is in the Pending state. +9. Waits for storage capacity to be polled by the driver. +10. Waits for Pod to be Running. + +> Storage class must use volume binding mode `WaitForFirstConsumer`.\ +> This suite does not delete resources on success. + +```bash +cert-csi functional-test capacity-tracking --sc --drns +``` + +Run `cert-csi test capacity-tracking -h` for more options. + +### Running Longevity mode + +```bash +cert-csi test --sc --longevity +``` +### Use configurable container images + +To use custom images for creating containers pass an image config YAML file as an argument. The YAML file should have linux(test) and postgres images name with their corresponding image URL. For example + +Example: +```yaml +images: + - test: "docker.io/centos:centos7" # change this to your url + postgres: "docker.io/bitnami/postgresql:11.8.0-debian-10-r72" # change this to your url +``` +To use this feature, run cert-csi with the option `--image-config /path/to/config.yaml` along with any other arguments. + + +## Kubernetes End-To-End Tests +All Kubernetes end to end tests require that you provide the driver config based on the storage class you want to test and the version of the kubernetes you want to test against. These are the mandatory parameters that you can provide in command like.. +```bash + --driver-config and --version "v1.25.0" + ``` + +### Running kubernetes end-to-end tests +To run kubernetes end-to-end tests, run the command: +```bash + +cert-csi k8s-e2e --config --driver-config --focus --timeout --version < version of k8s Ex: "v1.25.0"> --skip-tests --skip +``` + +### Kubernetes end-to-end reporting + +- All the reports generated by kubernetes end-to-end tests will be under `$HOME/reports` directory by default if user doesn't mention the report path. +- Kubernetes end to end tests Execution log file will be placed under `$HOME/reports/execution_[storage class name].log` +- Cert-CSI logs will be present in the execution directory `info.log` , `error.log` + +### Test config files format +- #### [driver-config](https://github.com/dell/cert-csi/blob/main/pkg/utils/testdata/config-nfs.yaml) +- #### [ignore-tests](https://github.com/dell/cert-csi/blob/main/pkg/utils/ignore.yaml) + +### Example Commands +- ```bash + + cert-csi k8s-e2e --config "/root/.kube/config" --driver-config "/root/e2e_config/config-nfs.yaml" --focus "External.Storage.*" --timeout "2h" --version "v1.25.0" --skip-tests "/root/e2e_config/ignore.yaml" + ``` +- ```bash + + ./cert-csi k8s-e2e --config "/root/.kube/config" --driver-config "/root/e2e_config/config-iscsi.yaml" --focus "External.Storage.*" --timeout "2h" --version "v1.25.0" --focus-file "capacity.go" + ``` + +### Interacting with DB + +#### Generating report from runs without running tests + +To generate test report from the database, run the command: +```bash +cert-csi --db report --testrun --html --txt +Report types: +--html: performance html report +--txt: performance txt report +--xml: junit compatible xml report, contains basic run infomation +--tabular: tidy html report with basic run information +``` + +To generate tabular report from the database, run the command: +```bash +cert-csi -db ./cert-csi-functional.db functional-report -tabular +``` + +To generate XML report from the database, run the command: +```bash +cert-csi -db ./cert-csi-functional.db functional-report -xml +``` + +#### Customizing report folder + +To specify test report folder path, use --path option as follows: +```bash + +cert-csi --db report --testrun --path +Options: +--path: path to folder where reports will be created (if not specified ~/.cert-csi/ will be used) +``` + +#### Generating report from multiple databases and test runs + +To generate report from multiple databases, run the command: +```bash +cert-csi report --tr : --tr ... --tabular --xml +Supported report types: +--xml +--tabular +``` + +#### Listing all known test runs + +To list all test runs, run the command: +```bash +cert-csi --db list test-runs +``` + +### Other options + +#### Customizing report folder + +To specify test report folder path, use --path option as follows: +```bash +cert-csi --path +Commands: + test + certify + report +``` + +#### Running with enabled driver resource usage metrics + +To run tests with driver resource usage metrics enabled, run the command: +```bash +cert-csi test --sc <...> --ns +``` + +#### Running custom hooks from program + +To run tests with custom hooks, run the command: +```bash + +cert-csi test --sc <...> --sh ./hooks/start.sh --rh ./hooks/ready.sh --fh ./hooks/finish.sh +``` + +## Screenshots + +### Running provisioning test + +![img1](../img/unifiedTest.png) + +You can interrupt the application by sending an interruption signal (for example pressing Ctrl + C). +It will stop polling and try to cleanup resources. + +![img2](../img/interruptTest.png) + +### Running scaling test + +![img3](../img/scaling.PNG) + +### Listing available test runs + +![img4](../img/listRuns.png) + +### Running longevity mode + +![img5](../img/longevity.png) + +### Multi DB Tabular report example + +![img6](../img/multiDBTabularReport.png) + +Text report example + +![img7](../img/textReport.png) + +Tabular Report example + +![img9](../img/tabularReport.png) + +### HTML report example + +![img8](../img/HTMLReport.png) + +### Resource usage example chart + +![img9](../img/resourceUsage.png) From e8c03919085343cb68e01a1d50dbaa11f9e5d2ed Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 14:40:30 -0300 Subject: [PATCH 15/49] Delete content/docs/csidriver/test/certcsi.md --- content/docs/csidriver/test/certcsi.md | 843 ------------------------- 1 file changed, 843 deletions(-) delete mode 100644 content/docs/csidriver/test/certcsi.md diff --git a/content/docs/csidriver/test/certcsi.md b/content/docs/csidriver/test/certcsi.md deleted file mode 100644 index 44a66a6efc..0000000000 --- a/content/docs/csidriver/test/certcsi.md +++ /dev/null @@ -1,843 +0,0 @@ ---- -title: Cert-CSI -linktitle: Cert-CSI -description: Tool to validate Dell CSI Drivers ---- - -Cert-CSI is a tool to validate Dell CSI Drivers. It contains various test suites to validate the drivers. - -## Installation - -There are three methods of installing `cert-csi`. - -1. [Download the executable from the latest GitHub release](#download-release-linux). -2. [Pull the container image from DockerHub](#pull-the-container-image). -3. [Build the exectuable or container image locally](#building-locally). - -> The exectuable from the GitHub Release only supports Linux. For non-Linux users, you must build the `cert-csi` executable [locally](#building-locally). - -### Download Release (Linux) - -1. Download the latest release of the cert-csi zip file. - -```bash -curl -LO https://github.com/dell/cert-csi/releases/download/v1.3.1/cert-csi-v1.3.1.zip -``` - -2. Unzip the file. - -``` bash -unzip cert-csi-v1.3.1.zip -chmod +x ./cert-csi-v1.3.1 -``` - -3. Install cert-csi-v1.3.1 as cert-csi. - -```bash -sudo install -o root -g root -m 0755 cert-csi-v1.3.1 /usr/local/bin/cert-csi -``` - -If you do not have root access on the target system, you can still install cert-csi to the ~/.local/bin directory: - -```bash -chmod +x cert-csi-v1.3.1 -mkdir -p ~/.local/bin -mv ./cert-csi-v1.3.1 ~/.local/bin/cert-csi -# and then append (or prepend) ~/.local/bin to $PATH -``` - -### Pull The Container Image - - {{< tabs name="pulling-cert-csi-image" >}} - {{% tab name="Docker" %}} - - ```bash - docker pull dellemc/cert-csi:v1.3.1 - ``` - - {{% /tab %}} - {{% tab name="Podman" %}} - - ```bash - podman pull dellemc/cert-csi:v1.3.1 - ``` - - {{% /tab %}} - {{< /tabs >}} - -### Building Locally -#### Prerequisites -- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) -- [Go](https://go.dev/doc/install) (If buidling the executable) -- Podman or Docker (If building the container image) - -1. Clone the repository - -```bash -git clone -b "v1.3.1" https://github.com/dell/cert-csi.git && cd cert-csi -``` - -2. Build cert-csi - -{{< tabs name="build-cert-csi" >}} -{{% tab name="Executable" %}} - -```bash - make build # the cert-csi executable will be in the working directory - chmod +x ./cert-csi # if building on *nix machine -``` - -{{% /tab %}} -{{% tab name="Container Image" %}} - -```bash - # uses podman if available, otherwise uses docker. The resulting image is tagged cert-csi:latest - make docker -``` - -{{% /tab %}} -{{< /tabs >}} - -### Optional - -If you want to collect csi-driver resource usage metrics, then please provide the namespace where it can be found and install the metric-server using this command (kubectl is required): - -```bash -make install-ms -``` - -## Running Cert-CSI - -{{< tabs name="running-cert-csi" >}} -{{% tab name="Executable" %}} -```bash - cert-csi --help -``` -{{% /tab %}} -{{% tab name="Docker" %}} -```bash - docker run --rm -it -v ~/.kube/config:/root/.kube/config dellemc/cert-csi:v1.3.1 --help -``` -{{% /tab %}} -{{% tab name="Podman" %}} -```bash - podman run --rm -it -v ~/.kube/config:/root/.kube/config dellemc/cert-csi:v1.3.1 --help -``` - -{{% /tab %}} -{{< /tabs >}} - -> The following sections showing how to execute the various test suites use the executable for brevity. For executions requiring special behavior, such as mounting file arguments into the container image, it will be noted for the relevant command. - -> Log files are located in the `logs` directory in the working directory of cert-csi.\ -> Report files are located in the default `$HOME/.cert-csi/reports` directory.\ -> Database (SQLite) file for test suites is `.db` in the working directory of cert-csi.\ -> Database (SQLite) file for functional test suites is `cert-csi-functional.db` in the working directory of cert-csi. - -> NOTE: If using the container image, these files will be inside the container. If you are interested in these files, it is recommended to use the exectuable. - -## Run All Test Suites - -You can use cert-csi to launch a test run against multiple storage classes to check if the driver adheres to advertised capabilities. - -### Preparing Config - -To run the test suites you need to provide `.yaml` config with storage classes and their capabilities. You can use `example-certify-config.yaml` as an example. - -Template: -```yaml -storageClasses: - - name: # storage-class-name (ex. powerstore) - minSize: # minimal size for your sc (ex. 1Gi) - rawBlock: # is Raw Block supported (true or false) - expansion: # is volume expansion supported (true or false) - clone: # is volume cloning supported (true or false) - snapshot: # is volume snapshotting supported (true or false) - RWX: # is ReadWriteMany volume access mode supported for non RawBlock volumes (true or false) - volumeHealth: # set this to enable the execution of the VolumeHealthMetricsSuite (true or false) - # Make sure to enable healthMonitor for the driver's controller and node pods before running this suite. It is recommended to use a smaller interval time for this sidecar and pass the required arguments. - VGS: # set this to enable the execution of the VolumeGroupSnapSuite (true or false) - # Additionally, make sure to provide the necessary required arguments such as volumeSnapshotClass, vgs-volume-label, and any others as needed. - RWOP: # set this to enable the execution of the MultiAttachSuite with the AccessMode set to ReadWriteOncePod (true or false) - ephemeral: # if exists, then run EphemeralVolumeSuite. See the Ephemeral Volumes suite section for example Volume Attributes - driver: # driver name for EphemeralVolumeSuite (e.g., csi-vxflexos.dellemc.com) - fstype: # fstype for EphemeralVolumeSuite - volumeAttributes: # volume attrs for EphemeralVolumeSuite. - attr1: # volume attr for EphemeralVolumeSuite - attr2: # volume attr for EphemeralVolumeSuite - capacityTracking: - driverNamespace: # namepsace where driver is installed - pollInterval: # duration to poll capacity (e.g., 2m) -``` - -Driver specific examples: - - {{< tabs name="cerity-config-examples" >}} - {{% tab name="CSI PowerFlex" %}} - -```yaml -storageClasses: - - name: vxflexos - minSize: 8Gi - rawBlock: true - expansion: true - clone: true - snapshot: true - RWX: false - ephemeral: - driver: csi-powerstore.dellemc.com - fstype: ext4 - volumeAttributes: - volumeName: "my-ephemeral-vol" - size: "8Gi" - storagepool: "sample" - systemID: "sample" - - name: vxflexos-nfs - minSize: 8Gi - rawBlock: false - expansion: true - clone: true - snapshot: true - RWX: true - RWOP: true - ephemeral: - driver: csi-vxflexos.dellemc.com - fstype: "nfs" - volumeAttributes: - volumeName: "my-ephemeral-vol" - size: "8Gi" - storagepool: "sample" - systemID: "sample" - capacityTracking: - driverNamespace: powerstore - pollInterval: 2m -``` - - {{% /tab %}} - {{% tab name="CSI PowerScale" %}} - -```yaml -storageClasses: - - name: isilon - minSize: 8Gi - rawBlock: false - expansion: true - clone: true - snapshot: true - RWX: false - ephemeral: - driver: csi-isilon.dellemc.com - fstype: nfs - volumeAttributes: - size: "10Gi" - ClusterName: "sample" - AccessZone: "sample" - IsiPath: "/ifs/data/sample" - IsiVolumePathPermissions: "0777" - AzServiceIP: "192.168.2.1" -``` - - {{% tab name="CSI PowerMax" %}} - -```yaml -storageClasses: - - name: powermax-iscsi - minSize: 5Gi - rawBlock: true - expansion: true - clone: true - snapshot: true - capacityTracking: - driverNamespace: powerstore - pollInterval: 2m - - name: powermax-nfs - minSize: 5Gi - rawBlock: false - expansion: true - clone: true - snapshot: true - RWX: true - RWOP: true - capacityTracking: - driverNamespace: powerstore - pollInterval: 2m -``` - - {{% /tab %}} - - {{% /tab %}} - {{% tab name="CSI PowerStore" %}} - -```yaml -storageClasses: - - name: powerstore - minSize: 5Gi - rawBlock: true - expansion: true - clone: true - snapshot: true - RWX: false - ephemeral: - driver: csi-powerstore.dellemc.com - fstype: ext4 - volumeAttributes: - arrayID: "arrayid" - protocol: iSCSI - size: 5Gi - - name: powerstore-nfs - minSize: 5Gi - rawBlock: false - expansion: true - clone: true - snapshot: true - RWX: true - RWOP: true - ephemeral: - driver: csi-powerstore.dellemc.com - fstype: "nfs" - volumeAttributes: - arrayID: "arrayid" - protocol: NFS - size: 5Gi - nasName: "nas-server" - capacityTracking: - driverNamespace: powerstore - pollInterval: 2m -``` - - {{% /tab %}} - {{% tab name="CSI Unity" %}} - -```yaml -storageClasses: - - name: unity-iscsi - minSize: 3Gi - rawBlock: true - expansion: true - clone: false - snapshot: true - RWX: false - ephemeral: - driver: csi-unity.dellemc.com - fstype: ext4 - volumeAttributes: - arrayId: "array-id" - storagePool: pool-name - protocol: NFS - size: 5Gi - - name: unity-nfs - minSize: 3Gi - rawBlock: false - expansion: true - clone: false - snapshot: true - RWX: true - RWOP: true - ephemeral: - driver: csi-unity.dellemc.com - fstype: "nfs" - volumeAttributes: - arrayId: "array-id" - storagePool: pool-name - protocol: NFS - size: 5Gi - nasServer: "nas-server" - nasName: "nas-name" - capacityTracking: - driverNamespace: unity - pollInterval: 2m -``` - - {{% /tab %}} - {{< /tabs >}} - -### Launching Test Run -1. Executes the [VolumeIO](#volume-io) suite. -2. Executes the [Scaling](#scalability) suite. -3. If `storageClasses.clone` is `true`, executes the [Volume Cloning](#volume-cloning) suite. -4. If `storageClasses.expansion` is `true`, executes the [Volume Expansion](#volume-expansion) suite. -5. If `storageClasses.expansion` is `true` and `storageClasses.rawBlock` is `true`, executes the [Volume Expansion](#volume-expansion) suite with raw block volumes. -6. If `storageClasses.snapshot` is `true`, exeuctes the [Snapshot](#snapshots) suite and the [Replication](#replication) suite. -7. If `storageClasses.rawBlock` is `true`, executes the [Multi-Attach Volume](#multi-attach-volume) suite with raw block volumes. -8. If `storageClasses.rwx` is `true`, executes the [Multi-Attach Volume](#multi-attach-volume) suite. (Storgae Class must be NFS.) -9. If `storageClasses.volumeHealth` is `true`, executes the [Volume Health Metrics](#volume-health-metrics) suite. -10. If `storageClasses.rwop` is `true`, executes the [Multi-Attach Volume](#multi-attach-volume) suite with the volume access mode `ReadWriteOncePod`. -11. If `storageClasses.ephemeral` exists, executes the [Ephemeral Volumes](#ephemeral-volumes) suite. -12. If `storageClasses.vgs` is `true`, executes the [Volume Group Snapshot]() suite. -13. If `storageClasses.capacityTracking` exists, exeuctes the [Storage Class Capacity Tracking](#storage-capacity-tracking) suite. - -> NOTE: For testing/debugging purposes, it can be useful to use the `--no-cleanup` so resources do not get deleted. - -> NOTE: If you are using CSI PowerScale with [SmartQuotas](../../features/powerscale/#usage-of-smartquotas-to-limit-storage-consumption) disabled, the `Volume Expansion` suite is expected to timeout due to the way PowerScale provisions storage. Set `storageClasses.expansion` to `false` to skip this suite. - -```bash -cert-csi certify --cert-config --vsc -``` - -Withold the `--vsc` argument if Snapshot capabilities are disabled. - -```bash -cert-csi certify --cert-config -Optional Params: - --vsc: volume snapshot class, required if you specified snapshot capability -``` - -Run `cert-csi certify -h` for more options. - -If you are using the container image, the `cert-config` file must be mounted into the container. Assuming your `cert-config` file is `/home/user/example-certify-config.yaml`, here are examples of how to exeucte this suite with the container image. - -{{< tabs name="running-container-certify" >}} -{{% tab name="Docker" %}} -```bash - docker run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/example-certify-config.yaml:/example-certify-config.yaml dellemc/cert-csi:v1.3.1 certify --cert-config /example-certify-config.yaml --vsc -``` -{{% /tab %}} -{{% tab name="Podman" %}} -```bash - podman run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/example-certify-config.yaml:/example-certify-config.yaml dellemc/cert-csi:v1.3.1 certify --cert-config /example-certify-config.yaml --vsc -``` - -{{% /tab %}} -{{< /tabs >}} - -### Running Invidual Test Suites - -> NOTE: For testing/debugging purposes, it can useful to use the `--no-cleanup` flag so resources do not get deleted. - -#### Volume I/O -1. Creates the namespace `volumeio-test-*` where resources will be created. -2. Creates Persistent Volume Claims. -3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claims to be bound to Persistent Volumes. -4. For each Persistent Volume Claim, executes the following workflow concurrently: - 1. Creates a Pod to consume the Persistent Volume Claim. - 2. Writes data to the volume and verifies the checksum of the data. - 3. Deletes the Pod. - 4. Waits for the associated Volume Attachment to be deleted. - -```bash -cert-csi test vio --sc -``` - -Run `cert-csi test vio -h` for more options. - -#### Scalability -1. Creates the namespace `scale-test-*` where resources will be created. -2. Creates a StatefulSet. -3. Scales up the StatefulSet. -4. Scales down the StatefulSet to zero. - -```bash -cert-csi test scaling --sc -``` - -Run `cert-csi test scaling -h` for more options. - -#### Snapshots -1. Creates the namespace `snap-test-*` where resources will be created. -2. Creates Persistent Volume Claim. -3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claim to be bound to Persistent Volumes. -4. Create Pod to consume the Persistent Volume Claim. -5. Writes data to the volume. -6. Deletes the Pod. -7. Creates a Volume Snapshot from the Persistent Volume Claim. -8. Waits for the Volume Snapshot to be Ready. -9. Creates a new Persistent Volume Claim from the Volume Snapshot. -10. Creates a new Pod to consume the new Persistent Volume Claim. -11. Verifies the checksum of the data. - -```bash -cert-csi test snap --sc --vsc -``` - -Run `cert-csi test snap -h` for more options. - -#### Volume Group Snapshots -1. Creates the namespace `vgs-snap-test-*` where resources will be created. -2. Creates Persistent Volume Claims. -3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claim to be bound to Persistent Volumes. -4. Create Pods to consume the Persistent Volume Claims. -5. Creates Volume Group Snapshot. -6. Waits for Volume Group Snapshot state to be COMPLETE. - -> Note: Volume Group Snapshots are only supported by CSI PowerFlex and CSI PowerStore. - -#### Multi-Attach Volume -1. Creates the namespace `mas-test-*` where resources will be created. -2. Creates Persistent Volume Claim. -3. Creates Pod to consume the Persistent Volume Claim. -4. Waits for Pod to be in the Ready state. -5. Creates additional Pods to consume the same Persistent Volume Claim. -6. Watis for Pods to be in the Ready state. -7. Writes data to the volumes on the Pods and verifies checksum of the data. - -```bash -cert-csi test multi-attach-vol --sc -``` - -> The storage class must be an NFS storage class. Otherwise, raw block volumes must be used. - -```bash -cert-csi test multi-attach-vol --sc --block -``` - -Run `cert-csi test multi-attach-vol -h` for more options. - -#### Replication -1. Creates the namespace `replication-suite-*` where resources will be created. -2. Creates Persistent Volume Claims. -3. Create Pods to consume the Persistent Volume Claims. -4. Waits for Pods to be in the Ready state. -5. Creates a Volume Snapshot from each Persistent Volume Claim. -6. Waits for the Volume Snapshots to be Ready. -7. Creates Persistent Volume Claims from the Volume Snapshots. -8. Creates Pods to consume the Persistent Volume Claims. -9. Waits for Pods to be in the Ready state. -10. Verifies the replication group name on ersistent Volume Claims. - -```bash -cert-csi test replication --sc --vsc -``` - -Run `cert-csi test replication -h` for more options. - -#### Volume Cloning -1. Creates the namespace `clonevolume-suite-*` where resources will be created. -2. Creates Persistent Volume Claims. -3. Create Pods to consume the Persistent Volume Claims. -4. Waits for Pods to be in the Ready state. -5. Creates Persistent Volume Claims with the source volume being from the volumes in step 2. -6. Create Pods to consume the Persistent Volume Claims. -7. Waits for Pods to be in the Ready state. - -```bash -cert-csi test clone-volume --sc -``` - -Run `cert-csi test clone-volume -h` for more options. - -#### Volume Expansion -1. Creates the namespace `volume-expansion-suite-*` where resources will be created. -2. Creates Persistent Volume Claims. -3. Create Pods to consume the Persistent Volume Claims. -4. Waits for Pods to be in the Ready state. -5. Expands the size in the Persistent Volume Claims. -6. Verifies that the volumes mounted to the Pods were expanded. - -> Raw block volumes cannot be verified since there is no filesystem. - -> If you are using CSI PowerScale with [SmartQuotas](../../features/powerscale/#usage-of-smartquotas-to-limit-storage-consumption) disabled, the `Volume Expansion` suite is expected to timeout due to the way PowerScale provisions storage. - -```bash -cert-csi test expansion --sc -``` - -Run `cert-csi test expansion -h` for more options. - -#### Blocksnap suite -1. Creates the namespace `block-snap-test-*` where resources will be created. -2. Creates Persistent Volume Claim. -3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claim to be bound to Persistent Volumes. -4. Creates Pod to consume the Persistent Volume Claim. -5. Writes data to the volume. -5. Creates a Volume Snapshot from the Persistent Volume Claim. -6. Waits for the Volume Snapshot to be Ready. -7. Create a Persistent Volume Claim with raw block volume mode from the Volume Snapshot. -8. Creates Pod to consume the Persistent Volume Claim. -9. Mounts the raw block volume and verifes the checksum of the data. - -```bash -cert-csi test blocksnap --sc --vsc -``` - -Run `cert-csi test blocksnap -h` for more options. - -#### Volume Health Metrics -1. Creates the namespace `volume-health-metrics-*` where resources will be created. -2. Creates Persistent Volume Claim. -3. Creates Pod to consume the Persistent Volume Claim. -4. Waits for Pod to be in the Ready state. -4. Veries that ControllerGetVolume and NodeGetVolumeStats are being executed in the controller and node pods, respectively. - -```bash -cert-csi test volumehealthmetrics --sc --driver-ns -``` - -Run `cert-csi test volumehealthmetrics -h` for more options. - -> Note: Make sure to enable healthMonitor for the driver's controller and node pods before running this suite. It is recommended to use a smaller interval time for this sidecar. - -#### Ephemeral Volumes -1. Creates namespace `functional-test` where resources will be created. -2. Creates Pods with one ephemeral inline volume each. -3. Waits for Pods to be in the Ready state. -4. Writes data to the volume on each Pod. -5. Verifies the checksum of the data. - -```bash -cert-csi test ephemeral-volume --driver --attr ephemeral-config.properties -``` - -Run `cert-csi test ephemeral-volume -h` for more options. - -> `--driver` is the name of a CSI Driver from the output of `kubectl get csidriver` (e.g, csi-vxflexos.dellemc.com). -> This suite does not delete resources on success. - -If you are using the container image, the `attr` file must be mounted into the container. Assuming your `attr` file is `/home/user/ephemeral-config.properties`, here are examples of how to exeucte this suite with the container image. - -{{< tabs name="running-container-ephemeral-volume" >}} -{{% tab name="Docker" %}} -```bash - docker run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/ephemeral-config.properties:/ephemeral-config.properties dellemc/cert-csi:v1.3.1 test ephemeral-volume --driver --attr /ephemeral-config.properties -``` -{{% /tab %}} -{{% tab name="Podman" %}} -```bash - podman run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/ephemeral-config.properties:/ephemeral-config.properties dellemc/cert-csi:v1.3.1 test ephemeral-volume --driver --attr /ephemeral-config.properties -``` - -{{% /tab %}} -{{< /tabs >}} - -Sample ephemeral-config.properties (key/value pair) - {{< tabs name="volume-attributes-examples" >}} - {{% tab name="CSI PowerFlex" %}} - - ```yaml - volumeName: "my-ephemeral-vol" - size: "10Gi" - storagepool: "sample" - systemID: "sample" - ``` - - {{% /tab %}} - {{% tab name="CSI PowerScale" %}} - - ```yaml - size: "10Gi" - ClusterName: "sample" - AccessZone: "sample" - IsiPath: "/ifs/data/sample" - IsiVolumePathPermissions: "0777" - AzServiceIP: "192.168.2.1" - ``` - - {{% /tab %}} - {{% tab name="CSI PowerStore" %}} - - ```yaml - size: "10Gi" - arrayID: "sample" - nasName: "sample" - nfsAcls: "0777" - ``` - - {{% /tab %}} - {{% tab name="CSI Unity" %}} - - ```yaml - size: "10Gi" - arrayID: "sample" - protocol: iSCSI - thinProvisioned: "true" - isDataReductionEnabled: "false" - tieringPolicy: "1" - storagePool: pool_2 - nasName: "sample" - ``` - - {{% /tab %}} - {{< /tabs >}} - -#### Storage Capacity Tracking -1. Creates namespace `functional-test` where resources will be created. -2. Creates a duplicate of the provided storge class using prefix `capacity-tracking`. -3. Waits for the associated CSIStorageCapacity object to be created. -4. Deletes the duplicate storge class. -5. Waits for the associated CSIStorageCapacity to be deleted. -6. Sets the capacity of the CSIStorageCapacity of the provided storage class to zero. -7. Creates Pod with a volume using the provided storage class. -8. Verifies that the Pod is in the Pending state. -9. Waits for storage capacity to be polled by the driver. -10. Waits for Pod to be Running. - -> Storage class must use volume binding mode `WaitForFirstConsumer`.\ -> This suite does not delete resources on success. - -```bash -cert-csi functional-test capacity-tracking --sc --drns -``` - -Run `cert-csi test capacity-tracking -h` for more options. - -### Running Longevity mode - -```bash -cert-csi test --sc --longevity -``` -### Use configurable container images - -To use custom images for creating containers pass an image config YAML file as an argument. The YAML file should have linux(test) and postgres images name with their corresponding image URL. For example - -Example: -```yaml -images: - - test: "docker.io/centos:centos7" # change this to your url - postgres: "docker.io/bitnami/postgresql:11.8.0-debian-10-r72" # change this to your url -``` -To use this feature, run cert-csi with the option `--image-config /path/to/config.yaml` along with any other arguments. - - -## Kubernetes End-To-End Tests -All Kubernetes end to end tests require that you provide the driver config based on the storage class you want to test and the version of the kubernetes you want to test against. These are the mandatory parameters that you can provide in command like.. -```bash - --driver-config and --version "v1.25.0" - ``` - -### Running kubernetes end-to-end tests -To run kubernetes end-to-end tests, run the command: -```bash - -cert-csi k8s-e2e --config --driver-config --focus --timeout --version < version of k8s Ex: "v1.25.0"> --skip-tests --skip -``` - -### Kubernetes end-to-end reporting - -- All the reports generated by kubernetes end-to-end tests will be under `$HOME/reports` directory by default if user doesn't mention the report path. -- Kubernetes end to end tests Execution log file will be placed under `$HOME/reports/execution_[storage class name].log` -- Cert-CSI logs will be present in the execution directory `info.log` , `error.log` - -### Test config files format -- #### [driver-config](https://github.com/dell/cert-csi/blob/main/pkg/utils/testdata/config-nfs.yaml) -- #### [ignore-tests](https://github.com/dell/cert-csi/blob/main/pkg/utils/ignore.yaml) - -### Example Commands -- ```bash - - cert-csi k8s-e2e --config "/root/.kube/config" --driver-config "/root/e2e_config/config-nfs.yaml" --focus "External.Storage.*" --timeout "2h" --version "v1.25.0" --skip-tests "/root/e2e_config/ignore.yaml" - ``` -- ```bash - - ./cert-csi k8s-e2e --config "/root/.kube/config" --driver-config "/root/e2e_config/config-iscsi.yaml" --focus "External.Storage.*" --timeout "2h" --version "v1.25.0" --focus-file "capacity.go" - ``` - -### Interacting with DB - -#### Generating report from runs without running tests - -To generate test report from the database, run the command: -```bash -cert-csi --db report --testrun --html --txt -Report types: ---html: performance html report ---txt: performance txt report ---xml: junit compatible xml report, contains basic run infomation ---tabular: tidy html report with basic run information -``` - -To generate tabular report from the database, run the command: -```bash -cert-csi -db ./cert-csi-functional.db functional-report -tabular -``` - -To generate XML report from the database, run the command: -```bash -cert-csi -db ./cert-csi-functional.db functional-report -xml -``` - -#### Customizing report folder - -To specify test report folder path, use --path option as follows: -```bash - -cert-csi --db report --testrun --path -Options: ---path: path to folder where reports will be created (if not specified ~/.cert-csi/ will be used) -``` - -#### Generating report from multiple databases and test runs - -To generate report from multiple databases, run the command: -```bash -cert-csi report --tr : --tr ... --tabular --xml -Supported report types: ---xml ---tabular -``` - -#### Listing all known test runs - -To list all test runs, run the command: -```bash -cert-csi --db list test-runs -``` - -### Other options - -#### Customizing report folder - -To specify test report folder path, use --path option as follows: -```bash -cert-csi --path -Commands: - test - certify - report -``` - -#### Running with enabled driver resource usage metrics - -To run tests with driver resource usage metrics enabled, run the command: -```bash -cert-csi test --sc <...> --ns -``` - -#### Running custom hooks from program - -To run tests with custom hooks, run the command: -```bash - -cert-csi test --sc <...> --sh ./hooks/start.sh --rh ./hooks/ready.sh --fh ./hooks/finish.sh -``` - -## Screenshots - -### Running provisioning test - -![img1](../img/unifiedTest.png) - -You can interrupt the application by sending an interruption signal (for example pressing Ctrl + C). -It will stop polling and try to cleanup resources. - -![img2](../img/interruptTest.png) - -### Running scaling test - -![img3](../img/scaling.PNG) - -### Listing available test runs - -![img4](../img/listRuns.png) - -### Running longevity mode - -![img5](../img/longevity.png) - -### Multi DB Tabular report example - -![img6](../img/multiDBTabularReport.png) - -Text report example - -![img7](../img/textReport.png) - -Tabular Report example - -![img9](../img/tabularReport.png) - -### HTML report example - -![img8](../img/HTMLReport.png) - -### Resource usage example chart - -![img9](../img/resourceUsage.png) From 59bd4b93600567866b067c840a98112b754f07bb Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 14:44:23 -0300 Subject: [PATCH 16/49] Created community qualified page --- content/docs/cert-csi/qualified.md | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 content/docs/cert-csi/qualified.md diff --git a/content/docs/cert-csi/qualified.md b/content/docs/cert-csi/qualified.md new file mode 100644 index 0000000000..f52eb834f3 --- /dev/null +++ b/content/docs/cert-csi/qualified.md @@ -0,0 +1,10 @@ +--- +title: Community Qualified Configuration +linktitle: Community Qualified Configuration +description: Community Qualified Configuration +--- + +{{}} +| cert-csi results | OS | CO | Storage Platform | Protocol | CSM | +|--------------------------------------------------------|:----------:|:----------------:|:-----------------------:|:---------:|:----------:| +| [Ticket 1079](https://github.com/dell/csm/issues/1079) | Debian 10 | K3s v1.24.7+k3s1 | Unity VSA 5.3.1.0.5.008 | iSCSI | CSI v1.8.0 | From 67d6d1a3f1515d1df8278f9426553187b3bab2a2 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 14:44:54 -0300 Subject: [PATCH 17/49] Removed community qualified table --- content/docs/csidriver/_index.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/content/docs/csidriver/_index.md b/content/docs/csidriver/_index.md index 17946d9188..bbdd747ffc 100644 --- a/content/docs/csidriver/_index.md +++ b/content/docs/csidriver/_index.md @@ -56,11 +56,3 @@ The CSI Drivers by Dell implement an interface between [CSI](https://kubernetes- | Platform-specific configurable settings | Service Level selection
iSCSI CHAP | - | Host IO Limit
Tiering Policy
NFS Host IO size
Snapshot Retention duration | Access Zone
NFS version (3 or 4);Configurable Export IPs | iSCSI CHAP | | Auto RDM(vSphere) | Yes(over FC) | N/A | N/A | N/A | N/A | {{
}} - -### Community Qualified Platforms -{{}} -| cert-csi results | OS | CO | Storage Platform | Protocol | CSM | -|--------------------------------------------------------|:----------:|:----------------:|:-----------------------:|:---------:|:----------:| -| [Ticket 1079](https://github.com/dell/csm/issues/1079) | Debian 10 | K3s v1.24.7+k3s1 | Unity VSA 5.3.1.0.5.008 | iSCSI | CSI v1.8.0 | - -{{
}} From 060d852f3544bc5becb3208403009911061aef95 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 14:51:54 -0300 Subject: [PATCH 18/49] Update qualified.md --- content/docs/cert-csi/qualified.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/docs/cert-csi/qualified.md b/content/docs/cert-csi/qualified.md index f52eb834f3..fd4798725a 100644 --- a/content/docs/cert-csi/qualified.md +++ b/content/docs/cert-csi/qualified.md @@ -8,3 +8,4 @@ description: Community Qualified Configuration | cert-csi results | OS | CO | Storage Platform | Protocol | CSM | |--------------------------------------------------------|:----------:|:----------------:|:-----------------------:|:---------:|:----------:| | [Ticket 1079](https://github.com/dell/csm/issues/1079) | Debian 10 | K3s v1.24.7+k3s1 | Unity VSA 5.3.1.0.5.008 | iSCSI | CSI v1.8.0 | +{{}} From 5504f3942b674ae81506d4212a6b7af0f746f86c Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 16:31:10 -0300 Subject: [PATCH 19/49] Create _index.md --- content/docs/prerequisites/_index.md | 43 ++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 content/docs/prerequisites/_index.md diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md new file mode 100644 index 0000000000..d1a3c6b71a --- /dev/null +++ b/content/docs/prerequisites/_index.md @@ -0,0 +1,43 @@ +--- +title: "Prerequisites" +linkTitle: "Prerequisites" +description: Prerequisites for CSM +weight: 3 +--- + +## Supported Container Orchestrator Platforms + +{{}} +| Platform | Version | +|---------------|:----------------:| +| Kubernetes | 1.27, 1.28, 1.29 | +| Red Hat OpenShift | 4.13, 4.14 | +| Mirantis Kubernetes Engine | 3.7.x | +| Google Anthos | 1.15 | +| VMware Tanzu | ?? | +| Rancher Kubernetes Engine | 1.4.x| +| Amazon Elastic Kubernetes Service
Anywhere | ?? | +{{
}} + +> Notes: +> * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. +> * Any orchestrator platform or version that's not mentioned here must be self-certified using [Cert-CSI](https://dell.github.io/csm-docs/docs/csidriver/installation/test/certcsi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. + +## Supported Storage Platforms + +{{}} +| Platform | Version | OS Dependencies | +|---------------|:----------------:|:------------------------:| +| PowerMax | PowerMax 2500/8500 PowerMaxOS 10 (6079) , PowerMaxOS 10.0.1 (6079) , PowerMaxOS 10.1 (6079), PowerMax 2000/8000 - 5978.711.xxx, 5978.479.xxx, Unisphere 10.0,10.0.1,10.1 | iscsi-initiator-utils
multipathd or powerpath
nvme-cli
nfs-utils | +| PowerFlex | 3.6.x, 4.0.x, 4.5.x | [SDC](https://www.dell.com/support/home/en-us/product-support/product/scaleio/drivers)| +| Unity XT | 5.1.x, 5.2.x, 5.3.0 | iscsi-initiator-utils
multipathd
nfs-utils | +| PowerScale | OneFS 9.3, 9.4, 9.5.0.x (x >= 5) | nfs-utils | +| PowerStore | 3.0, 3.2, 3.5 | iscsi-initiator-utils
multipathd
nvme-cli
nfs-utils | +| ObjectScale | 1.2.x | - | +{{
}} + +> Notes: +> * The required OS dependencies are only for the protocol needed (e.g. if NVMe isn't the storage access protocol then nvme-cli is not required). +> * [CSM Authorization](https://dell.github.io/csm-docs/docs/authorization/) only supports Kubernetes and the following Storage Platforms: PowerMax, PowerFlex, and PowerScale. +> * [CSM Observability](https://dell.github.io/csm-docs/docs/observability/) does not support the Unity XT Storage Platform. +> * [CSM Replication](https://dell.github.io/csm-docs/docs/replication/) does not support the Unity XT Storage Platform or File Replication for PowerMax. From f62122241a175619cf1304e07a65de7bdabf330b Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 16:31:24 -0300 Subject: [PATCH 20/49] Delete content/docs/prerequisites/_index.html --- content/docs/prerequisites/_index.html | 43 -------------------------- 1 file changed, 43 deletions(-) delete mode 100644 content/docs/prerequisites/_index.html diff --git a/content/docs/prerequisites/_index.html b/content/docs/prerequisites/_index.html deleted file mode 100644 index d1a3c6b71a..0000000000 --- a/content/docs/prerequisites/_index.html +++ /dev/null @@ -1,43 +0,0 @@ ---- -title: "Prerequisites" -linkTitle: "Prerequisites" -description: Prerequisites for CSM -weight: 3 ---- - -## Supported Container Orchestrator Platforms - -{{}} -| Platform | Version | -|---------------|:----------------:| -| Kubernetes | 1.27, 1.28, 1.29 | -| Red Hat OpenShift | 4.13, 4.14 | -| Mirantis Kubernetes Engine | 3.7.x | -| Google Anthos | 1.15 | -| VMware Tanzu | ?? | -| Rancher Kubernetes Engine | 1.4.x| -| Amazon Elastic Kubernetes Service
Anywhere | ?? | -{{
}} - -> Notes: -> * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. -> * Any orchestrator platform or version that's not mentioned here must be self-certified using [Cert-CSI](https://dell.github.io/csm-docs/docs/csidriver/installation/test/certcsi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. - -## Supported Storage Platforms - -{{}} -| Platform | Version | OS Dependencies | -|---------------|:----------------:|:------------------------:| -| PowerMax | PowerMax 2500/8500 PowerMaxOS 10 (6079) , PowerMaxOS 10.0.1 (6079) , PowerMaxOS 10.1 (6079), PowerMax 2000/8000 - 5978.711.xxx, 5978.479.xxx, Unisphere 10.0,10.0.1,10.1 | iscsi-initiator-utils
multipathd or powerpath
nvme-cli
nfs-utils | -| PowerFlex | 3.6.x, 4.0.x, 4.5.x | [SDC](https://www.dell.com/support/home/en-us/product-support/product/scaleio/drivers)| -| Unity XT | 5.1.x, 5.2.x, 5.3.0 | iscsi-initiator-utils
multipathd
nfs-utils | -| PowerScale | OneFS 9.3, 9.4, 9.5.0.x (x >= 5) | nfs-utils | -| PowerStore | 3.0, 3.2, 3.5 | iscsi-initiator-utils
multipathd
nvme-cli
nfs-utils | -| ObjectScale | 1.2.x | - | -{{
}} - -> Notes: -> * The required OS dependencies are only for the protocol needed (e.g. if NVMe isn't the storage access protocol then nvme-cli is not required). -> * [CSM Authorization](https://dell.github.io/csm-docs/docs/authorization/) only supports Kubernetes and the following Storage Platforms: PowerMax, PowerFlex, and PowerScale. -> * [CSM Observability](https://dell.github.io/csm-docs/docs/observability/) does not support the Unity XT Storage Platform. -> * [CSM Replication](https://dell.github.io/csm-docs/docs/replication/) does not support the Unity XT Storage Platform or File Replication for PowerMax. From a05bc5a40b3854c385da009ac29ef8e34cc861a3 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 16:32:55 -0300 Subject: [PATCH 21/49] Update _index.md --- content/docs/prerequisites/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index d1a3c6b71a..ce6443eee1 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -28,7 +28,7 @@ weight: 3 {{}} | Platform | Version | OS Dependencies | |---------------|:----------------:|:------------------------:| -| PowerMax | PowerMax 2500/8500 PowerMaxOS 10 (6079) , PowerMaxOS 10.0.1 (6079) , PowerMaxOS 10.1 (6079), PowerMax 2000/8000 - 5978.711.xxx, 5978.479.xxx, Unisphere 10.0,10.0.1,10.1 | iscsi-initiator-utils
multipathd or powerpath
nvme-cli
nfs-utils | +| PowerMax | PowerMax 2500/8500 PowerMaxOS 10 (6079)
PowerMaxOS 10.0.1 (6079)
PowerMaxOS 10.1 (6079)
PowerMax 2000/8000 - 5978.711.xxx
5978.479.xxx
Unisphere 10.0,10.0.1,10.1 | iscsi-initiator-utils
multipathd or powerpath
nvme-cli
nfs-utils | | PowerFlex | 3.6.x, 4.0.x, 4.5.x | [SDC](https://www.dell.com/support/home/en-us/product-support/product/scaleio/drivers)| | Unity XT | 5.1.x, 5.2.x, 5.3.0 | iscsi-initiator-utils
multipathd
nfs-utils | | PowerScale | OneFS 9.3, 9.4, 9.5.0.x (x >= 5) | nfs-utils | From 1c75f8664fe4a8b2c5a91748aff15bb0b45d1612 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 16:34:22 -0300 Subject: [PATCH 22/49] Create _index.md --- content/docs/cert-csi/_index.md | 843 ++++++++++++++++++++++++++++++++ 1 file changed, 843 insertions(+) create mode 100644 content/docs/cert-csi/_index.md diff --git a/content/docs/cert-csi/_index.md b/content/docs/cert-csi/_index.md new file mode 100644 index 0000000000..44a66a6efc --- /dev/null +++ b/content/docs/cert-csi/_index.md @@ -0,0 +1,843 @@ +--- +title: Cert-CSI +linktitle: Cert-CSI +description: Tool to validate Dell CSI Drivers +--- + +Cert-CSI is a tool to validate Dell CSI Drivers. It contains various test suites to validate the drivers. + +## Installation + +There are three methods of installing `cert-csi`. + +1. [Download the executable from the latest GitHub release](#download-release-linux). +2. [Pull the container image from DockerHub](#pull-the-container-image). +3. [Build the exectuable or container image locally](#building-locally). + +> The exectuable from the GitHub Release only supports Linux. For non-Linux users, you must build the `cert-csi` executable [locally](#building-locally). + +### Download Release (Linux) + +1. Download the latest release of the cert-csi zip file. + +```bash +curl -LO https://github.com/dell/cert-csi/releases/download/v1.3.1/cert-csi-v1.3.1.zip +``` + +2. Unzip the file. + +``` bash +unzip cert-csi-v1.3.1.zip +chmod +x ./cert-csi-v1.3.1 +``` + +3. Install cert-csi-v1.3.1 as cert-csi. + +```bash +sudo install -o root -g root -m 0755 cert-csi-v1.3.1 /usr/local/bin/cert-csi +``` + +If you do not have root access on the target system, you can still install cert-csi to the ~/.local/bin directory: + +```bash +chmod +x cert-csi-v1.3.1 +mkdir -p ~/.local/bin +mv ./cert-csi-v1.3.1 ~/.local/bin/cert-csi +# and then append (or prepend) ~/.local/bin to $PATH +``` + +### Pull The Container Image + + {{< tabs name="pulling-cert-csi-image" >}} + {{% tab name="Docker" %}} + + ```bash + docker pull dellemc/cert-csi:v1.3.1 + ``` + + {{% /tab %}} + {{% tab name="Podman" %}} + + ```bash + podman pull dellemc/cert-csi:v1.3.1 + ``` + + {{% /tab %}} + {{< /tabs >}} + +### Building Locally +#### Prerequisites +- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +- [Go](https://go.dev/doc/install) (If buidling the executable) +- Podman or Docker (If building the container image) + +1. Clone the repository + +```bash +git clone -b "v1.3.1" https://github.com/dell/cert-csi.git && cd cert-csi +``` + +2. Build cert-csi + +{{< tabs name="build-cert-csi" >}} +{{% tab name="Executable" %}} + +```bash + make build # the cert-csi executable will be in the working directory + chmod +x ./cert-csi # if building on *nix machine +``` + +{{% /tab %}} +{{% tab name="Container Image" %}} + +```bash + # uses podman if available, otherwise uses docker. The resulting image is tagged cert-csi:latest + make docker +``` + +{{% /tab %}} +{{< /tabs >}} + +### Optional + +If you want to collect csi-driver resource usage metrics, then please provide the namespace where it can be found and install the metric-server using this command (kubectl is required): + +```bash +make install-ms +``` + +## Running Cert-CSI + +{{< tabs name="running-cert-csi" >}} +{{% tab name="Executable" %}} +```bash + cert-csi --help +``` +{{% /tab %}} +{{% tab name="Docker" %}} +```bash + docker run --rm -it -v ~/.kube/config:/root/.kube/config dellemc/cert-csi:v1.3.1 --help +``` +{{% /tab %}} +{{% tab name="Podman" %}} +```bash + podman run --rm -it -v ~/.kube/config:/root/.kube/config dellemc/cert-csi:v1.3.1 --help +``` + +{{% /tab %}} +{{< /tabs >}} + +> The following sections showing how to execute the various test suites use the executable for brevity. For executions requiring special behavior, such as mounting file arguments into the container image, it will be noted for the relevant command. + +> Log files are located in the `logs` directory in the working directory of cert-csi.\ +> Report files are located in the default `$HOME/.cert-csi/reports` directory.\ +> Database (SQLite) file for test suites is `.db` in the working directory of cert-csi.\ +> Database (SQLite) file for functional test suites is `cert-csi-functional.db` in the working directory of cert-csi. + +> NOTE: If using the container image, these files will be inside the container. If you are interested in these files, it is recommended to use the exectuable. + +## Run All Test Suites + +You can use cert-csi to launch a test run against multiple storage classes to check if the driver adheres to advertised capabilities. + +### Preparing Config + +To run the test suites you need to provide `.yaml` config with storage classes and their capabilities. You can use `example-certify-config.yaml` as an example. + +Template: +```yaml +storageClasses: + - name: # storage-class-name (ex. powerstore) + minSize: # minimal size for your sc (ex. 1Gi) + rawBlock: # is Raw Block supported (true or false) + expansion: # is volume expansion supported (true or false) + clone: # is volume cloning supported (true or false) + snapshot: # is volume snapshotting supported (true or false) + RWX: # is ReadWriteMany volume access mode supported for non RawBlock volumes (true or false) + volumeHealth: # set this to enable the execution of the VolumeHealthMetricsSuite (true or false) + # Make sure to enable healthMonitor for the driver's controller and node pods before running this suite. It is recommended to use a smaller interval time for this sidecar and pass the required arguments. + VGS: # set this to enable the execution of the VolumeGroupSnapSuite (true or false) + # Additionally, make sure to provide the necessary required arguments such as volumeSnapshotClass, vgs-volume-label, and any others as needed. + RWOP: # set this to enable the execution of the MultiAttachSuite with the AccessMode set to ReadWriteOncePod (true or false) + ephemeral: # if exists, then run EphemeralVolumeSuite. See the Ephemeral Volumes suite section for example Volume Attributes + driver: # driver name for EphemeralVolumeSuite (e.g., csi-vxflexos.dellemc.com) + fstype: # fstype for EphemeralVolumeSuite + volumeAttributes: # volume attrs for EphemeralVolumeSuite. + attr1: # volume attr for EphemeralVolumeSuite + attr2: # volume attr for EphemeralVolumeSuite + capacityTracking: + driverNamespace: # namepsace where driver is installed + pollInterval: # duration to poll capacity (e.g., 2m) +``` + +Driver specific examples: + + {{< tabs name="cerity-config-examples" >}} + {{% tab name="CSI PowerFlex" %}} + +```yaml +storageClasses: + - name: vxflexos + minSize: 8Gi + rawBlock: true + expansion: true + clone: true + snapshot: true + RWX: false + ephemeral: + driver: csi-powerstore.dellemc.com + fstype: ext4 + volumeAttributes: + volumeName: "my-ephemeral-vol" + size: "8Gi" + storagepool: "sample" + systemID: "sample" + - name: vxflexos-nfs + minSize: 8Gi + rawBlock: false + expansion: true + clone: true + snapshot: true + RWX: true + RWOP: true + ephemeral: + driver: csi-vxflexos.dellemc.com + fstype: "nfs" + volumeAttributes: + volumeName: "my-ephemeral-vol" + size: "8Gi" + storagepool: "sample" + systemID: "sample" + capacityTracking: + driverNamespace: powerstore + pollInterval: 2m +``` + + {{% /tab %}} + {{% tab name="CSI PowerScale" %}} + +```yaml +storageClasses: + - name: isilon + minSize: 8Gi + rawBlock: false + expansion: true + clone: true + snapshot: true + RWX: false + ephemeral: + driver: csi-isilon.dellemc.com + fstype: nfs + volumeAttributes: + size: "10Gi" + ClusterName: "sample" + AccessZone: "sample" + IsiPath: "/ifs/data/sample" + IsiVolumePathPermissions: "0777" + AzServiceIP: "192.168.2.1" +``` + + {{% tab name="CSI PowerMax" %}} + +```yaml +storageClasses: + - name: powermax-iscsi + minSize: 5Gi + rawBlock: true + expansion: true + clone: true + snapshot: true + capacityTracking: + driverNamespace: powerstore + pollInterval: 2m + - name: powermax-nfs + minSize: 5Gi + rawBlock: false + expansion: true + clone: true + snapshot: true + RWX: true + RWOP: true + capacityTracking: + driverNamespace: powerstore + pollInterval: 2m +``` + + {{% /tab %}} + + {{% /tab %}} + {{% tab name="CSI PowerStore" %}} + +```yaml +storageClasses: + - name: powerstore + minSize: 5Gi + rawBlock: true + expansion: true + clone: true + snapshot: true + RWX: false + ephemeral: + driver: csi-powerstore.dellemc.com + fstype: ext4 + volumeAttributes: + arrayID: "arrayid" + protocol: iSCSI + size: 5Gi + - name: powerstore-nfs + minSize: 5Gi + rawBlock: false + expansion: true + clone: true + snapshot: true + RWX: true + RWOP: true + ephemeral: + driver: csi-powerstore.dellemc.com + fstype: "nfs" + volumeAttributes: + arrayID: "arrayid" + protocol: NFS + size: 5Gi + nasName: "nas-server" + capacityTracking: + driverNamespace: powerstore + pollInterval: 2m +``` + + {{% /tab %}} + {{% tab name="CSI Unity" %}} + +```yaml +storageClasses: + - name: unity-iscsi + minSize: 3Gi + rawBlock: true + expansion: true + clone: false + snapshot: true + RWX: false + ephemeral: + driver: csi-unity.dellemc.com + fstype: ext4 + volumeAttributes: + arrayId: "array-id" + storagePool: pool-name + protocol: NFS + size: 5Gi + - name: unity-nfs + minSize: 3Gi + rawBlock: false + expansion: true + clone: false + snapshot: true + RWX: true + RWOP: true + ephemeral: + driver: csi-unity.dellemc.com + fstype: "nfs" + volumeAttributes: + arrayId: "array-id" + storagePool: pool-name + protocol: NFS + size: 5Gi + nasServer: "nas-server" + nasName: "nas-name" + capacityTracking: + driverNamespace: unity + pollInterval: 2m +``` + + {{% /tab %}} + {{< /tabs >}} + +### Launching Test Run +1. Executes the [VolumeIO](#volume-io) suite. +2. Executes the [Scaling](#scalability) suite. +3. If `storageClasses.clone` is `true`, executes the [Volume Cloning](#volume-cloning) suite. +4. If `storageClasses.expansion` is `true`, executes the [Volume Expansion](#volume-expansion) suite. +5. If `storageClasses.expansion` is `true` and `storageClasses.rawBlock` is `true`, executes the [Volume Expansion](#volume-expansion) suite with raw block volumes. +6. If `storageClasses.snapshot` is `true`, exeuctes the [Snapshot](#snapshots) suite and the [Replication](#replication) suite. +7. If `storageClasses.rawBlock` is `true`, executes the [Multi-Attach Volume](#multi-attach-volume) suite with raw block volumes. +8. If `storageClasses.rwx` is `true`, executes the [Multi-Attach Volume](#multi-attach-volume) suite. (Storgae Class must be NFS.) +9. If `storageClasses.volumeHealth` is `true`, executes the [Volume Health Metrics](#volume-health-metrics) suite. +10. If `storageClasses.rwop` is `true`, executes the [Multi-Attach Volume](#multi-attach-volume) suite with the volume access mode `ReadWriteOncePod`. +11. If `storageClasses.ephemeral` exists, executes the [Ephemeral Volumes](#ephemeral-volumes) suite. +12. If `storageClasses.vgs` is `true`, executes the [Volume Group Snapshot]() suite. +13. If `storageClasses.capacityTracking` exists, exeuctes the [Storage Class Capacity Tracking](#storage-capacity-tracking) suite. + +> NOTE: For testing/debugging purposes, it can be useful to use the `--no-cleanup` so resources do not get deleted. + +> NOTE: If you are using CSI PowerScale with [SmartQuotas](../../features/powerscale/#usage-of-smartquotas-to-limit-storage-consumption) disabled, the `Volume Expansion` suite is expected to timeout due to the way PowerScale provisions storage. Set `storageClasses.expansion` to `false` to skip this suite. + +```bash +cert-csi certify --cert-config --vsc +``` + +Withold the `--vsc` argument if Snapshot capabilities are disabled. + +```bash +cert-csi certify --cert-config +Optional Params: + --vsc: volume snapshot class, required if you specified snapshot capability +``` + +Run `cert-csi certify -h` for more options. + +If you are using the container image, the `cert-config` file must be mounted into the container. Assuming your `cert-config` file is `/home/user/example-certify-config.yaml`, here are examples of how to exeucte this suite with the container image. + +{{< tabs name="running-container-certify" >}} +{{% tab name="Docker" %}} +```bash + docker run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/example-certify-config.yaml:/example-certify-config.yaml dellemc/cert-csi:v1.3.1 certify --cert-config /example-certify-config.yaml --vsc +``` +{{% /tab %}} +{{% tab name="Podman" %}} +```bash + podman run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/example-certify-config.yaml:/example-certify-config.yaml dellemc/cert-csi:v1.3.1 certify --cert-config /example-certify-config.yaml --vsc +``` + +{{% /tab %}} +{{< /tabs >}} + +### Running Invidual Test Suites + +> NOTE: For testing/debugging purposes, it can useful to use the `--no-cleanup` flag so resources do not get deleted. + +#### Volume I/O +1. Creates the namespace `volumeio-test-*` where resources will be created. +2. Creates Persistent Volume Claims. +3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claims to be bound to Persistent Volumes. +4. For each Persistent Volume Claim, executes the following workflow concurrently: + 1. Creates a Pod to consume the Persistent Volume Claim. + 2. Writes data to the volume and verifies the checksum of the data. + 3. Deletes the Pod. + 4. Waits for the associated Volume Attachment to be deleted. + +```bash +cert-csi test vio --sc +``` + +Run `cert-csi test vio -h` for more options. + +#### Scalability +1. Creates the namespace `scale-test-*` where resources will be created. +2. Creates a StatefulSet. +3. Scales up the StatefulSet. +4. Scales down the StatefulSet to zero. + +```bash +cert-csi test scaling --sc +``` + +Run `cert-csi test scaling -h` for more options. + +#### Snapshots +1. Creates the namespace `snap-test-*` where resources will be created. +2. Creates Persistent Volume Claim. +3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claim to be bound to Persistent Volumes. +4. Create Pod to consume the Persistent Volume Claim. +5. Writes data to the volume. +6. Deletes the Pod. +7. Creates a Volume Snapshot from the Persistent Volume Claim. +8. Waits for the Volume Snapshot to be Ready. +9. Creates a new Persistent Volume Claim from the Volume Snapshot. +10. Creates a new Pod to consume the new Persistent Volume Claim. +11. Verifies the checksum of the data. + +```bash +cert-csi test snap --sc --vsc +``` + +Run `cert-csi test snap -h` for more options. + +#### Volume Group Snapshots +1. Creates the namespace `vgs-snap-test-*` where resources will be created. +2. Creates Persistent Volume Claims. +3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claim to be bound to Persistent Volumes. +4. Create Pods to consume the Persistent Volume Claims. +5. Creates Volume Group Snapshot. +6. Waits for Volume Group Snapshot state to be COMPLETE. + +> Note: Volume Group Snapshots are only supported by CSI PowerFlex and CSI PowerStore. + +#### Multi-Attach Volume +1. Creates the namespace `mas-test-*` where resources will be created. +2. Creates Persistent Volume Claim. +3. Creates Pod to consume the Persistent Volume Claim. +4. Waits for Pod to be in the Ready state. +5. Creates additional Pods to consume the same Persistent Volume Claim. +6. Watis for Pods to be in the Ready state. +7. Writes data to the volumes on the Pods and verifies checksum of the data. + +```bash +cert-csi test multi-attach-vol --sc +``` + +> The storage class must be an NFS storage class. Otherwise, raw block volumes must be used. + +```bash +cert-csi test multi-attach-vol --sc --block +``` + +Run `cert-csi test multi-attach-vol -h` for more options. + +#### Replication +1. Creates the namespace `replication-suite-*` where resources will be created. +2. Creates Persistent Volume Claims. +3. Create Pods to consume the Persistent Volume Claims. +4. Waits for Pods to be in the Ready state. +5. Creates a Volume Snapshot from each Persistent Volume Claim. +6. Waits for the Volume Snapshots to be Ready. +7. Creates Persistent Volume Claims from the Volume Snapshots. +8. Creates Pods to consume the Persistent Volume Claims. +9. Waits for Pods to be in the Ready state. +10. Verifies the replication group name on ersistent Volume Claims. + +```bash +cert-csi test replication --sc --vsc +``` + +Run `cert-csi test replication -h` for more options. + +#### Volume Cloning +1. Creates the namespace `clonevolume-suite-*` where resources will be created. +2. Creates Persistent Volume Claims. +3. Create Pods to consume the Persistent Volume Claims. +4. Waits for Pods to be in the Ready state. +5. Creates Persistent Volume Claims with the source volume being from the volumes in step 2. +6. Create Pods to consume the Persistent Volume Claims. +7. Waits for Pods to be in the Ready state. + +```bash +cert-csi test clone-volume --sc +``` + +Run `cert-csi test clone-volume -h` for more options. + +#### Volume Expansion +1. Creates the namespace `volume-expansion-suite-*` where resources will be created. +2. Creates Persistent Volume Claims. +3. Create Pods to consume the Persistent Volume Claims. +4. Waits for Pods to be in the Ready state. +5. Expands the size in the Persistent Volume Claims. +6. Verifies that the volumes mounted to the Pods were expanded. + +> Raw block volumes cannot be verified since there is no filesystem. + +> If you are using CSI PowerScale with [SmartQuotas](../../features/powerscale/#usage-of-smartquotas-to-limit-storage-consumption) disabled, the `Volume Expansion` suite is expected to timeout due to the way PowerScale provisions storage. + +```bash +cert-csi test expansion --sc +``` + +Run `cert-csi test expansion -h` for more options. + +#### Blocksnap suite +1. Creates the namespace `block-snap-test-*` where resources will be created. +2. Creates Persistent Volume Claim. +3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claim to be bound to Persistent Volumes. +4. Creates Pod to consume the Persistent Volume Claim. +5. Writes data to the volume. +5. Creates a Volume Snapshot from the Persistent Volume Claim. +6. Waits for the Volume Snapshot to be Ready. +7. Create a Persistent Volume Claim with raw block volume mode from the Volume Snapshot. +8. Creates Pod to consume the Persistent Volume Claim. +9. Mounts the raw block volume and verifes the checksum of the data. + +```bash +cert-csi test blocksnap --sc --vsc +``` + +Run `cert-csi test blocksnap -h` for more options. + +#### Volume Health Metrics +1. Creates the namespace `volume-health-metrics-*` where resources will be created. +2. Creates Persistent Volume Claim. +3. Creates Pod to consume the Persistent Volume Claim. +4. Waits for Pod to be in the Ready state. +4. Veries that ControllerGetVolume and NodeGetVolumeStats are being executed in the controller and node pods, respectively. + +```bash +cert-csi test volumehealthmetrics --sc --driver-ns +``` + +Run `cert-csi test volumehealthmetrics -h` for more options. + +> Note: Make sure to enable healthMonitor for the driver's controller and node pods before running this suite. It is recommended to use a smaller interval time for this sidecar. + +#### Ephemeral Volumes +1. Creates namespace `functional-test` where resources will be created. +2. Creates Pods with one ephemeral inline volume each. +3. Waits for Pods to be in the Ready state. +4. Writes data to the volume on each Pod. +5. Verifies the checksum of the data. + +```bash +cert-csi test ephemeral-volume --driver --attr ephemeral-config.properties +``` + +Run `cert-csi test ephemeral-volume -h` for more options. + +> `--driver` is the name of a CSI Driver from the output of `kubectl get csidriver` (e.g, csi-vxflexos.dellemc.com). +> This suite does not delete resources on success. + +If you are using the container image, the `attr` file must be mounted into the container. Assuming your `attr` file is `/home/user/ephemeral-config.properties`, here are examples of how to exeucte this suite with the container image. + +{{< tabs name="running-container-ephemeral-volume" >}} +{{% tab name="Docker" %}} +```bash + docker run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/ephemeral-config.properties:/ephemeral-config.properties dellemc/cert-csi:v1.3.1 test ephemeral-volume --driver --attr /ephemeral-config.properties +``` +{{% /tab %}} +{{% tab name="Podman" %}} +```bash + podman run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/ephemeral-config.properties:/ephemeral-config.properties dellemc/cert-csi:v1.3.1 test ephemeral-volume --driver --attr /ephemeral-config.properties +``` + +{{% /tab %}} +{{< /tabs >}} + +Sample ephemeral-config.properties (key/value pair) + {{< tabs name="volume-attributes-examples" >}} + {{% tab name="CSI PowerFlex" %}} + + ```yaml + volumeName: "my-ephemeral-vol" + size: "10Gi" + storagepool: "sample" + systemID: "sample" + ``` + + {{% /tab %}} + {{% tab name="CSI PowerScale" %}} + + ```yaml + size: "10Gi" + ClusterName: "sample" + AccessZone: "sample" + IsiPath: "/ifs/data/sample" + IsiVolumePathPermissions: "0777" + AzServiceIP: "192.168.2.1" + ``` + + {{% /tab %}} + {{% tab name="CSI PowerStore" %}} + + ```yaml + size: "10Gi" + arrayID: "sample" + nasName: "sample" + nfsAcls: "0777" + ``` + + {{% /tab %}} + {{% tab name="CSI Unity" %}} + + ```yaml + size: "10Gi" + arrayID: "sample" + protocol: iSCSI + thinProvisioned: "true" + isDataReductionEnabled: "false" + tieringPolicy: "1" + storagePool: pool_2 + nasName: "sample" + ``` + + {{% /tab %}} + {{< /tabs >}} + +#### Storage Capacity Tracking +1. Creates namespace `functional-test` where resources will be created. +2. Creates a duplicate of the provided storge class using prefix `capacity-tracking`. +3. Waits for the associated CSIStorageCapacity object to be created. +4. Deletes the duplicate storge class. +5. Waits for the associated CSIStorageCapacity to be deleted. +6. Sets the capacity of the CSIStorageCapacity of the provided storage class to zero. +7. Creates Pod with a volume using the provided storage class. +8. Verifies that the Pod is in the Pending state. +9. Waits for storage capacity to be polled by the driver. +10. Waits for Pod to be Running. + +> Storage class must use volume binding mode `WaitForFirstConsumer`.\ +> This suite does not delete resources on success. + +```bash +cert-csi functional-test capacity-tracking --sc --drns +``` + +Run `cert-csi test capacity-tracking -h` for more options. + +### Running Longevity mode + +```bash +cert-csi test --sc --longevity +``` +### Use configurable container images + +To use custom images for creating containers pass an image config YAML file as an argument. The YAML file should have linux(test) and postgres images name with their corresponding image URL. For example + +Example: +```yaml +images: + - test: "docker.io/centos:centos7" # change this to your url + postgres: "docker.io/bitnami/postgresql:11.8.0-debian-10-r72" # change this to your url +``` +To use this feature, run cert-csi with the option `--image-config /path/to/config.yaml` along with any other arguments. + + +## Kubernetes End-To-End Tests +All Kubernetes end to end tests require that you provide the driver config based on the storage class you want to test and the version of the kubernetes you want to test against. These are the mandatory parameters that you can provide in command like.. +```bash + --driver-config and --version "v1.25.0" + ``` + +### Running kubernetes end-to-end tests +To run kubernetes end-to-end tests, run the command: +```bash + +cert-csi k8s-e2e --config --driver-config --focus --timeout --version < version of k8s Ex: "v1.25.0"> --skip-tests --skip +``` + +### Kubernetes end-to-end reporting + +- All the reports generated by kubernetes end-to-end tests will be under `$HOME/reports` directory by default if user doesn't mention the report path. +- Kubernetes end to end tests Execution log file will be placed under `$HOME/reports/execution_[storage class name].log` +- Cert-CSI logs will be present in the execution directory `info.log` , `error.log` + +### Test config files format +- #### [driver-config](https://github.com/dell/cert-csi/blob/main/pkg/utils/testdata/config-nfs.yaml) +- #### [ignore-tests](https://github.com/dell/cert-csi/blob/main/pkg/utils/ignore.yaml) + +### Example Commands +- ```bash + + cert-csi k8s-e2e --config "/root/.kube/config" --driver-config "/root/e2e_config/config-nfs.yaml" --focus "External.Storage.*" --timeout "2h" --version "v1.25.0" --skip-tests "/root/e2e_config/ignore.yaml" + ``` +- ```bash + + ./cert-csi k8s-e2e --config "/root/.kube/config" --driver-config "/root/e2e_config/config-iscsi.yaml" --focus "External.Storage.*" --timeout "2h" --version "v1.25.0" --focus-file "capacity.go" + ``` + +### Interacting with DB + +#### Generating report from runs without running tests + +To generate test report from the database, run the command: +```bash +cert-csi --db report --testrun --html --txt +Report types: +--html: performance html report +--txt: performance txt report +--xml: junit compatible xml report, contains basic run infomation +--tabular: tidy html report with basic run information +``` + +To generate tabular report from the database, run the command: +```bash +cert-csi -db ./cert-csi-functional.db functional-report -tabular +``` + +To generate XML report from the database, run the command: +```bash +cert-csi -db ./cert-csi-functional.db functional-report -xml +``` + +#### Customizing report folder + +To specify test report folder path, use --path option as follows: +```bash + +cert-csi --db report --testrun --path +Options: +--path: path to folder where reports will be created (if not specified ~/.cert-csi/ will be used) +``` + +#### Generating report from multiple databases and test runs + +To generate report from multiple databases, run the command: +```bash +cert-csi report --tr : --tr ... --tabular --xml +Supported report types: +--xml +--tabular +``` + +#### Listing all known test runs + +To list all test runs, run the command: +```bash +cert-csi --db list test-runs +``` + +### Other options + +#### Customizing report folder + +To specify test report folder path, use --path option as follows: +```bash +cert-csi --path +Commands: + test + certify + report +``` + +#### Running with enabled driver resource usage metrics + +To run tests with driver resource usage metrics enabled, run the command: +```bash +cert-csi test --sc <...> --ns +``` + +#### Running custom hooks from program + +To run tests with custom hooks, run the command: +```bash + +cert-csi test --sc <...> --sh ./hooks/start.sh --rh ./hooks/ready.sh --fh ./hooks/finish.sh +``` + +## Screenshots + +### Running provisioning test + +![img1](../img/unifiedTest.png) + +You can interrupt the application by sending an interruption signal (for example pressing Ctrl + C). +It will stop polling and try to cleanup resources. + +![img2](../img/interruptTest.png) + +### Running scaling test + +![img3](../img/scaling.PNG) + +### Listing available test runs + +![img4](../img/listRuns.png) + +### Running longevity mode + +![img5](../img/longevity.png) + +### Multi DB Tabular report example + +![img6](../img/multiDBTabularReport.png) + +Text report example + +![img7](../img/textReport.png) + +Tabular Report example + +![img9](../img/tabularReport.png) + +### HTML report example + +![img8](../img/HTMLReport.png) + +### Resource usage example chart + +![img9](../img/resourceUsage.png) From c2e1002d7b24591d3ff684542cfba01169ef493a Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 16:34:30 -0300 Subject: [PATCH 23/49] Delete content/docs/cert-csi/_index.html --- content/docs/cert-csi/_index.html | 843 ------------------------------ 1 file changed, 843 deletions(-) delete mode 100644 content/docs/cert-csi/_index.html diff --git a/content/docs/cert-csi/_index.html b/content/docs/cert-csi/_index.html deleted file mode 100644 index 44a66a6efc..0000000000 --- a/content/docs/cert-csi/_index.html +++ /dev/null @@ -1,843 +0,0 @@ ---- -title: Cert-CSI -linktitle: Cert-CSI -description: Tool to validate Dell CSI Drivers ---- - -Cert-CSI is a tool to validate Dell CSI Drivers. It contains various test suites to validate the drivers. - -## Installation - -There are three methods of installing `cert-csi`. - -1. [Download the executable from the latest GitHub release](#download-release-linux). -2. [Pull the container image from DockerHub](#pull-the-container-image). -3. [Build the exectuable or container image locally](#building-locally). - -> The exectuable from the GitHub Release only supports Linux. For non-Linux users, you must build the `cert-csi` executable [locally](#building-locally). - -### Download Release (Linux) - -1. Download the latest release of the cert-csi zip file. - -```bash -curl -LO https://github.com/dell/cert-csi/releases/download/v1.3.1/cert-csi-v1.3.1.zip -``` - -2. Unzip the file. - -``` bash -unzip cert-csi-v1.3.1.zip -chmod +x ./cert-csi-v1.3.1 -``` - -3. Install cert-csi-v1.3.1 as cert-csi. - -```bash -sudo install -o root -g root -m 0755 cert-csi-v1.3.1 /usr/local/bin/cert-csi -``` - -If you do not have root access on the target system, you can still install cert-csi to the ~/.local/bin directory: - -```bash -chmod +x cert-csi-v1.3.1 -mkdir -p ~/.local/bin -mv ./cert-csi-v1.3.1 ~/.local/bin/cert-csi -# and then append (or prepend) ~/.local/bin to $PATH -``` - -### Pull The Container Image - - {{< tabs name="pulling-cert-csi-image" >}} - {{% tab name="Docker" %}} - - ```bash - docker pull dellemc/cert-csi:v1.3.1 - ``` - - {{% /tab %}} - {{% tab name="Podman" %}} - - ```bash - podman pull dellemc/cert-csi:v1.3.1 - ``` - - {{% /tab %}} - {{< /tabs >}} - -### Building Locally -#### Prerequisites -- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) -- [Go](https://go.dev/doc/install) (If buidling the executable) -- Podman or Docker (If building the container image) - -1. Clone the repository - -```bash -git clone -b "v1.3.1" https://github.com/dell/cert-csi.git && cd cert-csi -``` - -2. Build cert-csi - -{{< tabs name="build-cert-csi" >}} -{{% tab name="Executable" %}} - -```bash - make build # the cert-csi executable will be in the working directory - chmod +x ./cert-csi # if building on *nix machine -``` - -{{% /tab %}} -{{% tab name="Container Image" %}} - -```bash - # uses podman if available, otherwise uses docker. The resulting image is tagged cert-csi:latest - make docker -``` - -{{% /tab %}} -{{< /tabs >}} - -### Optional - -If you want to collect csi-driver resource usage metrics, then please provide the namespace where it can be found and install the metric-server using this command (kubectl is required): - -```bash -make install-ms -``` - -## Running Cert-CSI - -{{< tabs name="running-cert-csi" >}} -{{% tab name="Executable" %}} -```bash - cert-csi --help -``` -{{% /tab %}} -{{% tab name="Docker" %}} -```bash - docker run --rm -it -v ~/.kube/config:/root/.kube/config dellemc/cert-csi:v1.3.1 --help -``` -{{% /tab %}} -{{% tab name="Podman" %}} -```bash - podman run --rm -it -v ~/.kube/config:/root/.kube/config dellemc/cert-csi:v1.3.1 --help -``` - -{{% /tab %}} -{{< /tabs >}} - -> The following sections showing how to execute the various test suites use the executable for brevity. For executions requiring special behavior, such as mounting file arguments into the container image, it will be noted for the relevant command. - -> Log files are located in the `logs` directory in the working directory of cert-csi.\ -> Report files are located in the default `$HOME/.cert-csi/reports` directory.\ -> Database (SQLite) file for test suites is `.db` in the working directory of cert-csi.\ -> Database (SQLite) file for functional test suites is `cert-csi-functional.db` in the working directory of cert-csi. - -> NOTE: If using the container image, these files will be inside the container. If you are interested in these files, it is recommended to use the exectuable. - -## Run All Test Suites - -You can use cert-csi to launch a test run against multiple storage classes to check if the driver adheres to advertised capabilities. - -### Preparing Config - -To run the test suites you need to provide `.yaml` config with storage classes and their capabilities. You can use `example-certify-config.yaml` as an example. - -Template: -```yaml -storageClasses: - - name: # storage-class-name (ex. powerstore) - minSize: # minimal size for your sc (ex. 1Gi) - rawBlock: # is Raw Block supported (true or false) - expansion: # is volume expansion supported (true or false) - clone: # is volume cloning supported (true or false) - snapshot: # is volume snapshotting supported (true or false) - RWX: # is ReadWriteMany volume access mode supported for non RawBlock volumes (true or false) - volumeHealth: # set this to enable the execution of the VolumeHealthMetricsSuite (true or false) - # Make sure to enable healthMonitor for the driver's controller and node pods before running this suite. It is recommended to use a smaller interval time for this sidecar and pass the required arguments. - VGS: # set this to enable the execution of the VolumeGroupSnapSuite (true or false) - # Additionally, make sure to provide the necessary required arguments such as volumeSnapshotClass, vgs-volume-label, and any others as needed. - RWOP: # set this to enable the execution of the MultiAttachSuite with the AccessMode set to ReadWriteOncePod (true or false) - ephemeral: # if exists, then run EphemeralVolumeSuite. See the Ephemeral Volumes suite section for example Volume Attributes - driver: # driver name for EphemeralVolumeSuite (e.g., csi-vxflexos.dellemc.com) - fstype: # fstype for EphemeralVolumeSuite - volumeAttributes: # volume attrs for EphemeralVolumeSuite. - attr1: # volume attr for EphemeralVolumeSuite - attr2: # volume attr for EphemeralVolumeSuite - capacityTracking: - driverNamespace: # namepsace where driver is installed - pollInterval: # duration to poll capacity (e.g., 2m) -``` - -Driver specific examples: - - {{< tabs name="cerity-config-examples" >}} - {{% tab name="CSI PowerFlex" %}} - -```yaml -storageClasses: - - name: vxflexos - minSize: 8Gi - rawBlock: true - expansion: true - clone: true - snapshot: true - RWX: false - ephemeral: - driver: csi-powerstore.dellemc.com - fstype: ext4 - volumeAttributes: - volumeName: "my-ephemeral-vol" - size: "8Gi" - storagepool: "sample" - systemID: "sample" - - name: vxflexos-nfs - minSize: 8Gi - rawBlock: false - expansion: true - clone: true - snapshot: true - RWX: true - RWOP: true - ephemeral: - driver: csi-vxflexos.dellemc.com - fstype: "nfs" - volumeAttributes: - volumeName: "my-ephemeral-vol" - size: "8Gi" - storagepool: "sample" - systemID: "sample" - capacityTracking: - driverNamespace: powerstore - pollInterval: 2m -``` - - {{% /tab %}} - {{% tab name="CSI PowerScale" %}} - -```yaml -storageClasses: - - name: isilon - minSize: 8Gi - rawBlock: false - expansion: true - clone: true - snapshot: true - RWX: false - ephemeral: - driver: csi-isilon.dellemc.com - fstype: nfs - volumeAttributes: - size: "10Gi" - ClusterName: "sample" - AccessZone: "sample" - IsiPath: "/ifs/data/sample" - IsiVolumePathPermissions: "0777" - AzServiceIP: "192.168.2.1" -``` - - {{% tab name="CSI PowerMax" %}} - -```yaml -storageClasses: - - name: powermax-iscsi - minSize: 5Gi - rawBlock: true - expansion: true - clone: true - snapshot: true - capacityTracking: - driverNamespace: powerstore - pollInterval: 2m - - name: powermax-nfs - minSize: 5Gi - rawBlock: false - expansion: true - clone: true - snapshot: true - RWX: true - RWOP: true - capacityTracking: - driverNamespace: powerstore - pollInterval: 2m -``` - - {{% /tab %}} - - {{% /tab %}} - {{% tab name="CSI PowerStore" %}} - -```yaml -storageClasses: - - name: powerstore - minSize: 5Gi - rawBlock: true - expansion: true - clone: true - snapshot: true - RWX: false - ephemeral: - driver: csi-powerstore.dellemc.com - fstype: ext4 - volumeAttributes: - arrayID: "arrayid" - protocol: iSCSI - size: 5Gi - - name: powerstore-nfs - minSize: 5Gi - rawBlock: false - expansion: true - clone: true - snapshot: true - RWX: true - RWOP: true - ephemeral: - driver: csi-powerstore.dellemc.com - fstype: "nfs" - volumeAttributes: - arrayID: "arrayid" - protocol: NFS - size: 5Gi - nasName: "nas-server" - capacityTracking: - driverNamespace: powerstore - pollInterval: 2m -``` - - {{% /tab %}} - {{% tab name="CSI Unity" %}} - -```yaml -storageClasses: - - name: unity-iscsi - minSize: 3Gi - rawBlock: true - expansion: true - clone: false - snapshot: true - RWX: false - ephemeral: - driver: csi-unity.dellemc.com - fstype: ext4 - volumeAttributes: - arrayId: "array-id" - storagePool: pool-name - protocol: NFS - size: 5Gi - - name: unity-nfs - minSize: 3Gi - rawBlock: false - expansion: true - clone: false - snapshot: true - RWX: true - RWOP: true - ephemeral: - driver: csi-unity.dellemc.com - fstype: "nfs" - volumeAttributes: - arrayId: "array-id" - storagePool: pool-name - protocol: NFS - size: 5Gi - nasServer: "nas-server" - nasName: "nas-name" - capacityTracking: - driverNamespace: unity - pollInterval: 2m -``` - - {{% /tab %}} - {{< /tabs >}} - -### Launching Test Run -1. Executes the [VolumeIO](#volume-io) suite. -2. Executes the [Scaling](#scalability) suite. -3. If `storageClasses.clone` is `true`, executes the [Volume Cloning](#volume-cloning) suite. -4. If `storageClasses.expansion` is `true`, executes the [Volume Expansion](#volume-expansion) suite. -5. If `storageClasses.expansion` is `true` and `storageClasses.rawBlock` is `true`, executes the [Volume Expansion](#volume-expansion) suite with raw block volumes. -6. If `storageClasses.snapshot` is `true`, exeuctes the [Snapshot](#snapshots) suite and the [Replication](#replication) suite. -7. If `storageClasses.rawBlock` is `true`, executes the [Multi-Attach Volume](#multi-attach-volume) suite with raw block volumes. -8. If `storageClasses.rwx` is `true`, executes the [Multi-Attach Volume](#multi-attach-volume) suite. (Storgae Class must be NFS.) -9. If `storageClasses.volumeHealth` is `true`, executes the [Volume Health Metrics](#volume-health-metrics) suite. -10. If `storageClasses.rwop` is `true`, executes the [Multi-Attach Volume](#multi-attach-volume) suite with the volume access mode `ReadWriteOncePod`. -11. If `storageClasses.ephemeral` exists, executes the [Ephemeral Volumes](#ephemeral-volumes) suite. -12. If `storageClasses.vgs` is `true`, executes the [Volume Group Snapshot]() suite. -13. If `storageClasses.capacityTracking` exists, exeuctes the [Storage Class Capacity Tracking](#storage-capacity-tracking) suite. - -> NOTE: For testing/debugging purposes, it can be useful to use the `--no-cleanup` so resources do not get deleted. - -> NOTE: If you are using CSI PowerScale with [SmartQuotas](../../features/powerscale/#usage-of-smartquotas-to-limit-storage-consumption) disabled, the `Volume Expansion` suite is expected to timeout due to the way PowerScale provisions storage. Set `storageClasses.expansion` to `false` to skip this suite. - -```bash -cert-csi certify --cert-config --vsc -``` - -Withold the `--vsc` argument if Snapshot capabilities are disabled. - -```bash -cert-csi certify --cert-config -Optional Params: - --vsc: volume snapshot class, required if you specified snapshot capability -``` - -Run `cert-csi certify -h` for more options. - -If you are using the container image, the `cert-config` file must be mounted into the container. Assuming your `cert-config` file is `/home/user/example-certify-config.yaml`, here are examples of how to exeucte this suite with the container image. - -{{< tabs name="running-container-certify" >}} -{{% tab name="Docker" %}} -```bash - docker run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/example-certify-config.yaml:/example-certify-config.yaml dellemc/cert-csi:v1.3.1 certify --cert-config /example-certify-config.yaml --vsc -``` -{{% /tab %}} -{{% tab name="Podman" %}} -```bash - podman run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/example-certify-config.yaml:/example-certify-config.yaml dellemc/cert-csi:v1.3.1 certify --cert-config /example-certify-config.yaml --vsc -``` - -{{% /tab %}} -{{< /tabs >}} - -### Running Invidual Test Suites - -> NOTE: For testing/debugging purposes, it can useful to use the `--no-cleanup` flag so resources do not get deleted. - -#### Volume I/O -1. Creates the namespace `volumeio-test-*` where resources will be created. -2. Creates Persistent Volume Claims. -3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claims to be bound to Persistent Volumes. -4. For each Persistent Volume Claim, executes the following workflow concurrently: - 1. Creates a Pod to consume the Persistent Volume Claim. - 2. Writes data to the volume and verifies the checksum of the data. - 3. Deletes the Pod. - 4. Waits for the associated Volume Attachment to be deleted. - -```bash -cert-csi test vio --sc -``` - -Run `cert-csi test vio -h` for more options. - -#### Scalability -1. Creates the namespace `scale-test-*` where resources will be created. -2. Creates a StatefulSet. -3. Scales up the StatefulSet. -4. Scales down the StatefulSet to zero. - -```bash -cert-csi test scaling --sc -``` - -Run `cert-csi test scaling -h` for more options. - -#### Snapshots -1. Creates the namespace `snap-test-*` where resources will be created. -2. Creates Persistent Volume Claim. -3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claim to be bound to Persistent Volumes. -4. Create Pod to consume the Persistent Volume Claim. -5. Writes data to the volume. -6. Deletes the Pod. -7. Creates a Volume Snapshot from the Persistent Volume Claim. -8. Waits for the Volume Snapshot to be Ready. -9. Creates a new Persistent Volume Claim from the Volume Snapshot. -10. Creates a new Pod to consume the new Persistent Volume Claim. -11. Verifies the checksum of the data. - -```bash -cert-csi test snap --sc --vsc -``` - -Run `cert-csi test snap -h` for more options. - -#### Volume Group Snapshots -1. Creates the namespace `vgs-snap-test-*` where resources will be created. -2. Creates Persistent Volume Claims. -3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claim to be bound to Persistent Volumes. -4. Create Pods to consume the Persistent Volume Claims. -5. Creates Volume Group Snapshot. -6. Waits for Volume Group Snapshot state to be COMPLETE. - -> Note: Volume Group Snapshots are only supported by CSI PowerFlex and CSI PowerStore. - -#### Multi-Attach Volume -1. Creates the namespace `mas-test-*` where resources will be created. -2. Creates Persistent Volume Claim. -3. Creates Pod to consume the Persistent Volume Claim. -4. Waits for Pod to be in the Ready state. -5. Creates additional Pods to consume the same Persistent Volume Claim. -6. Watis for Pods to be in the Ready state. -7. Writes data to the volumes on the Pods and verifies checksum of the data. - -```bash -cert-csi test multi-attach-vol --sc -``` - -> The storage class must be an NFS storage class. Otherwise, raw block volumes must be used. - -```bash -cert-csi test multi-attach-vol --sc --block -``` - -Run `cert-csi test multi-attach-vol -h` for more options. - -#### Replication -1. Creates the namespace `replication-suite-*` where resources will be created. -2. Creates Persistent Volume Claims. -3. Create Pods to consume the Persistent Volume Claims. -4. Waits for Pods to be in the Ready state. -5. Creates a Volume Snapshot from each Persistent Volume Claim. -6. Waits for the Volume Snapshots to be Ready. -7. Creates Persistent Volume Claims from the Volume Snapshots. -8. Creates Pods to consume the Persistent Volume Claims. -9. Waits for Pods to be in the Ready state. -10. Verifies the replication group name on ersistent Volume Claims. - -```bash -cert-csi test replication --sc --vsc -``` - -Run `cert-csi test replication -h` for more options. - -#### Volume Cloning -1. Creates the namespace `clonevolume-suite-*` where resources will be created. -2. Creates Persistent Volume Claims. -3. Create Pods to consume the Persistent Volume Claims. -4. Waits for Pods to be in the Ready state. -5. Creates Persistent Volume Claims with the source volume being from the volumes in step 2. -6. Create Pods to consume the Persistent Volume Claims. -7. Waits for Pods to be in the Ready state. - -```bash -cert-csi test clone-volume --sc -``` - -Run `cert-csi test clone-volume -h` for more options. - -#### Volume Expansion -1. Creates the namespace `volume-expansion-suite-*` where resources will be created. -2. Creates Persistent Volume Claims. -3. Create Pods to consume the Persistent Volume Claims. -4. Waits for Pods to be in the Ready state. -5. Expands the size in the Persistent Volume Claims. -6. Verifies that the volumes mounted to the Pods were expanded. - -> Raw block volumes cannot be verified since there is no filesystem. - -> If you are using CSI PowerScale with [SmartQuotas](../../features/powerscale/#usage-of-smartquotas-to-limit-storage-consumption) disabled, the `Volume Expansion` suite is expected to timeout due to the way PowerScale provisions storage. - -```bash -cert-csi test expansion --sc -``` - -Run `cert-csi test expansion -h` for more options. - -#### Blocksnap suite -1. Creates the namespace `block-snap-test-*` where resources will be created. -2. Creates Persistent Volume Claim. -3. If the specified storage class binding mode is not `WaitForFirstConsumer`, waits for Persistent Volume Claim to be bound to Persistent Volumes. -4. Creates Pod to consume the Persistent Volume Claim. -5. Writes data to the volume. -5. Creates a Volume Snapshot from the Persistent Volume Claim. -6. Waits for the Volume Snapshot to be Ready. -7. Create a Persistent Volume Claim with raw block volume mode from the Volume Snapshot. -8. Creates Pod to consume the Persistent Volume Claim. -9. Mounts the raw block volume and verifes the checksum of the data. - -```bash -cert-csi test blocksnap --sc --vsc -``` - -Run `cert-csi test blocksnap -h` for more options. - -#### Volume Health Metrics -1. Creates the namespace `volume-health-metrics-*` where resources will be created. -2. Creates Persistent Volume Claim. -3. Creates Pod to consume the Persistent Volume Claim. -4. Waits for Pod to be in the Ready state. -4. Veries that ControllerGetVolume and NodeGetVolumeStats are being executed in the controller and node pods, respectively. - -```bash -cert-csi test volumehealthmetrics --sc --driver-ns -``` - -Run `cert-csi test volumehealthmetrics -h` for more options. - -> Note: Make sure to enable healthMonitor for the driver's controller and node pods before running this suite. It is recommended to use a smaller interval time for this sidecar. - -#### Ephemeral Volumes -1. Creates namespace `functional-test` where resources will be created. -2. Creates Pods with one ephemeral inline volume each. -3. Waits for Pods to be in the Ready state. -4. Writes data to the volume on each Pod. -5. Verifies the checksum of the data. - -```bash -cert-csi test ephemeral-volume --driver --attr ephemeral-config.properties -``` - -Run `cert-csi test ephemeral-volume -h` for more options. - -> `--driver` is the name of a CSI Driver from the output of `kubectl get csidriver` (e.g, csi-vxflexos.dellemc.com). -> This suite does not delete resources on success. - -If you are using the container image, the `attr` file must be mounted into the container. Assuming your `attr` file is `/home/user/ephemeral-config.properties`, here are examples of how to exeucte this suite with the container image. - -{{< tabs name="running-container-ephemeral-volume" >}} -{{% tab name="Docker" %}} -```bash - docker run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/ephemeral-config.properties:/ephemeral-config.properties dellemc/cert-csi:v1.3.1 test ephemeral-volume --driver --attr /ephemeral-config.properties -``` -{{% /tab %}} -{{% tab name="Podman" %}} -```bash - podman run --rm -it -v ~/.kube/config:/root/.kube/config -v /home/user/ephemeral-config.properties:/ephemeral-config.properties dellemc/cert-csi:v1.3.1 test ephemeral-volume --driver --attr /ephemeral-config.properties -``` - -{{% /tab %}} -{{< /tabs >}} - -Sample ephemeral-config.properties (key/value pair) - {{< tabs name="volume-attributes-examples" >}} - {{% tab name="CSI PowerFlex" %}} - - ```yaml - volumeName: "my-ephemeral-vol" - size: "10Gi" - storagepool: "sample" - systemID: "sample" - ``` - - {{% /tab %}} - {{% tab name="CSI PowerScale" %}} - - ```yaml - size: "10Gi" - ClusterName: "sample" - AccessZone: "sample" - IsiPath: "/ifs/data/sample" - IsiVolumePathPermissions: "0777" - AzServiceIP: "192.168.2.1" - ``` - - {{% /tab %}} - {{% tab name="CSI PowerStore" %}} - - ```yaml - size: "10Gi" - arrayID: "sample" - nasName: "sample" - nfsAcls: "0777" - ``` - - {{% /tab %}} - {{% tab name="CSI Unity" %}} - - ```yaml - size: "10Gi" - arrayID: "sample" - protocol: iSCSI - thinProvisioned: "true" - isDataReductionEnabled: "false" - tieringPolicy: "1" - storagePool: pool_2 - nasName: "sample" - ``` - - {{% /tab %}} - {{< /tabs >}} - -#### Storage Capacity Tracking -1. Creates namespace `functional-test` where resources will be created. -2. Creates a duplicate of the provided storge class using prefix `capacity-tracking`. -3. Waits for the associated CSIStorageCapacity object to be created. -4. Deletes the duplicate storge class. -5. Waits for the associated CSIStorageCapacity to be deleted. -6. Sets the capacity of the CSIStorageCapacity of the provided storage class to zero. -7. Creates Pod with a volume using the provided storage class. -8. Verifies that the Pod is in the Pending state. -9. Waits for storage capacity to be polled by the driver. -10. Waits for Pod to be Running. - -> Storage class must use volume binding mode `WaitForFirstConsumer`.\ -> This suite does not delete resources on success. - -```bash -cert-csi functional-test capacity-tracking --sc --drns -``` - -Run `cert-csi test capacity-tracking -h` for more options. - -### Running Longevity mode - -```bash -cert-csi test --sc --longevity -``` -### Use configurable container images - -To use custom images for creating containers pass an image config YAML file as an argument. The YAML file should have linux(test) and postgres images name with their corresponding image URL. For example - -Example: -```yaml -images: - - test: "docker.io/centos:centos7" # change this to your url - postgres: "docker.io/bitnami/postgresql:11.8.0-debian-10-r72" # change this to your url -``` -To use this feature, run cert-csi with the option `--image-config /path/to/config.yaml` along with any other arguments. - - -## Kubernetes End-To-End Tests -All Kubernetes end to end tests require that you provide the driver config based on the storage class you want to test and the version of the kubernetes you want to test against. These are the mandatory parameters that you can provide in command like.. -```bash - --driver-config and --version "v1.25.0" - ``` - -### Running kubernetes end-to-end tests -To run kubernetes end-to-end tests, run the command: -```bash - -cert-csi k8s-e2e --config --driver-config --focus --timeout --version < version of k8s Ex: "v1.25.0"> --skip-tests --skip -``` - -### Kubernetes end-to-end reporting - -- All the reports generated by kubernetes end-to-end tests will be under `$HOME/reports` directory by default if user doesn't mention the report path. -- Kubernetes end to end tests Execution log file will be placed under `$HOME/reports/execution_[storage class name].log` -- Cert-CSI logs will be present in the execution directory `info.log` , `error.log` - -### Test config files format -- #### [driver-config](https://github.com/dell/cert-csi/blob/main/pkg/utils/testdata/config-nfs.yaml) -- #### [ignore-tests](https://github.com/dell/cert-csi/blob/main/pkg/utils/ignore.yaml) - -### Example Commands -- ```bash - - cert-csi k8s-e2e --config "/root/.kube/config" --driver-config "/root/e2e_config/config-nfs.yaml" --focus "External.Storage.*" --timeout "2h" --version "v1.25.0" --skip-tests "/root/e2e_config/ignore.yaml" - ``` -- ```bash - - ./cert-csi k8s-e2e --config "/root/.kube/config" --driver-config "/root/e2e_config/config-iscsi.yaml" --focus "External.Storage.*" --timeout "2h" --version "v1.25.0" --focus-file "capacity.go" - ``` - -### Interacting with DB - -#### Generating report from runs without running tests - -To generate test report from the database, run the command: -```bash -cert-csi --db report --testrun --html --txt -Report types: ---html: performance html report ---txt: performance txt report ---xml: junit compatible xml report, contains basic run infomation ---tabular: tidy html report with basic run information -``` - -To generate tabular report from the database, run the command: -```bash -cert-csi -db ./cert-csi-functional.db functional-report -tabular -``` - -To generate XML report from the database, run the command: -```bash -cert-csi -db ./cert-csi-functional.db functional-report -xml -``` - -#### Customizing report folder - -To specify test report folder path, use --path option as follows: -```bash - -cert-csi --db report --testrun --path -Options: ---path: path to folder where reports will be created (if not specified ~/.cert-csi/ will be used) -``` - -#### Generating report from multiple databases and test runs - -To generate report from multiple databases, run the command: -```bash -cert-csi report --tr : --tr ... --tabular --xml -Supported report types: ---xml ---tabular -``` - -#### Listing all known test runs - -To list all test runs, run the command: -```bash -cert-csi --db list test-runs -``` - -### Other options - -#### Customizing report folder - -To specify test report folder path, use --path option as follows: -```bash -cert-csi --path -Commands: - test - certify - report -``` - -#### Running with enabled driver resource usage metrics - -To run tests with driver resource usage metrics enabled, run the command: -```bash -cert-csi test --sc <...> --ns -``` - -#### Running custom hooks from program - -To run tests with custom hooks, run the command: -```bash - -cert-csi test --sc <...> --sh ./hooks/start.sh --rh ./hooks/ready.sh --fh ./hooks/finish.sh -``` - -## Screenshots - -### Running provisioning test - -![img1](../img/unifiedTest.png) - -You can interrupt the application by sending an interruption signal (for example pressing Ctrl + C). -It will stop polling and try to cleanup resources. - -![img2](../img/interruptTest.png) - -### Running scaling test - -![img3](../img/scaling.PNG) - -### Listing available test runs - -![img4](../img/listRuns.png) - -### Running longevity mode - -![img5](../img/longevity.png) - -### Multi DB Tabular report example - -![img6](../img/multiDBTabularReport.png) - -Text report example - -![img7](../img/textReport.png) - -Tabular Report example - -![img9](../img/tabularReport.png) - -### HTML report example - -![img8](../img/HTMLReport.png) - -### Resource usage example chart - -![img9](../img/resourceUsage.png) From eec452691eabae35e91f6805892fde259b1a7c1b Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 16:51:07 -0300 Subject: [PATCH 24/49] Removed Tanzu from Support Matrix --- content/docs/prerequisites/_index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index ce6443eee1..c8587ccc02 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -14,7 +14,6 @@ weight: 3 | Red Hat OpenShift | 4.13, 4.14 | | Mirantis Kubernetes Engine | 3.7.x | | Google Anthos | 1.15 | -| VMware Tanzu | ?? | | Rancher Kubernetes Engine | 1.4.x| | Amazon Elastic Kubernetes Service
Anywhere | ?? | {{
}} From 4ad0786a772b9dcfdcb50c77656eb7638e55369c Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 16:53:09 -0300 Subject: [PATCH 25/49] Update qualified.md --- content/docs/cert-csi/qualified.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/docs/cert-csi/qualified.md b/content/docs/cert-csi/qualified.md index fd4798725a..9f55d9e5b4 100644 --- a/content/docs/cert-csi/qualified.md +++ b/content/docs/cert-csi/qualified.md @@ -1,7 +1,7 @@ --- -title: Community Qualified Configuration -linktitle: Community Qualified Configuration -description: Community Qualified Configuration +title: Community Qualified Configurations +linktitle: Community Qualified Configurations +description: Community Qualified Configurations --- {{}} From c9b4b59c7cd68920502da7fd40fb5a31f1731f91 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 16:53:35 -0300 Subject: [PATCH 26/49] changed weight --- content/docs/deployment/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/docs/deployment/_index.md b/content/docs/deployment/_index.md index fea74b27e2..2cbef062d9 100644 --- a/content/docs/deployment/_index.md +++ b/content/docs/deployment/_index.md @@ -3,7 +3,7 @@ title: "Deployment" linkTitle: "Deployment" no_list: true description: Deployment of CSM -weight: 1 +weight: 2 --- The Container Storage Modules along with the required CSI Drivers can each be deployed using CSM operator. @@ -70,4 +70,4 @@ The Container Storage Modules and the required CSI Drivers can each be deployed Encryption can be optionally installed via the PowerScale CSI driver Helm chart. [...More on installation instructions](helm/modules/installation/encryption) {{< /card >}} -{{< /cardpane >}} \ No newline at end of file +{{< /cardpane >}} From 9a061afadd8ff7dd773dc54b718868c572fed9ae Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 16:53:57 -0300 Subject: [PATCH 27/49] changed weight --- content/docs/prerequisites/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index c8587ccc02..ad514ff4a4 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -2,7 +2,7 @@ title: "Prerequisites" linkTitle: "Prerequisites" description: Prerequisites for CSM -weight: 3 +weight: 1 --- ## Supported Container Orchestrator Platforms From 8531ae5198650848ff68d7dde2cb4af21295d2dd Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 16:54:53 -0300 Subject: [PATCH 28/49] Update _index.md --- content/docs/prerequisites/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index ad514ff4a4..b6e3019a70 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -15,7 +15,7 @@ weight: 1 | Mirantis Kubernetes Engine | 3.7.x | | Google Anthos | 1.15 | | Rancher Kubernetes Engine | 1.4.x| -| Amazon Elastic Kubernetes Service
Anywhere | ?? | +| Amazon Elastic Kubernetes Service | ?? | {{
}} > Notes: From c47ae73cb7e6acd3e140dd11cefb97ffd9a8fca1 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Thu, 14 Mar 2024 17:01:18 -0300 Subject: [PATCH 29/49] Update _index.md --- content/docs/prerequisites/_index.md | 34 ++++++++++++++-------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index b6e3019a70..8f26379d38 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -5,23 +5,6 @@ description: Prerequisites for CSM weight: 1 --- -## Supported Container Orchestrator Platforms - -{{}} -| Platform | Version | -|---------------|:----------------:| -| Kubernetes | 1.27, 1.28, 1.29 | -| Red Hat OpenShift | 4.13, 4.14 | -| Mirantis Kubernetes Engine | 3.7.x | -| Google Anthos | 1.15 | -| Rancher Kubernetes Engine | 1.4.x| -| Amazon Elastic Kubernetes Service | ?? | -{{
}} - -> Notes: -> * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. -> * Any orchestrator platform or version that's not mentioned here must be self-certified using [Cert-CSI](https://dell.github.io/csm-docs/docs/csidriver/installation/test/certcsi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. - ## Supported Storage Platforms {{}} @@ -40,3 +23,20 @@ weight: 1 > * [CSM Authorization](https://dell.github.io/csm-docs/docs/authorization/) only supports Kubernetes and the following Storage Platforms: PowerMax, PowerFlex, and PowerScale. > * [CSM Observability](https://dell.github.io/csm-docs/docs/observability/) does not support the Unity XT Storage Platform. > * [CSM Replication](https://dell.github.io/csm-docs/docs/replication/) does not support the Unity XT Storage Platform or File Replication for PowerMax. + +## Supported Container Orchestrator Platforms + +{{
}} +| Platform | Version | +|---------------|:----------------:| +| Kubernetes | 1.27, 1.28, 1.29 | +| Red Hat OpenShift | 4.13, 4.14 | +| Mirantis Kubernetes Engine | 3.7.x | +| Google Anthos | 1.15 | +| Rancher Kubernetes Engine | 1.4.x| +| Amazon Elastic Kubernetes Service | ?? | +{{
}} + +> Notes: +> * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. +> * Any orchestrator platform or version that's not mentioned here must be self-certified using [Cert-CSI](https://dell.github.io/csm-docs/docs/csidriver/installation/test/certcsi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. From af5e39f8560bc40530c50ae857950c704a05caf5 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Fri, 15 Mar 2024 08:48:36 -0300 Subject: [PATCH 30/49] updated links --- content/docs/prerequisites/_index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index 8f26379d38..1aa96fdb80 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -20,9 +20,9 @@ weight: 1 > Notes: > * The required OS dependencies are only for the protocol needed (e.g. if NVMe isn't the storage access protocol then nvme-cli is not required). -> * [CSM Authorization](https://dell.github.io/csm-docs/docs/authorization/) only supports Kubernetes and the following Storage Platforms: PowerMax, PowerFlex, and PowerScale. -> * [CSM Observability](https://dell.github.io/csm-docs/docs/observability/) does not support the Unity XT Storage Platform. -> * [CSM Replication](https://dell.github.io/csm-docs/docs/replication/) does not support the Unity XT Storage Platform or File Replication for PowerMax. +> * [CSM Authorization](../authorization/) only supports Kubernetes and the following Storage Platforms: PowerMax, PowerFlex, and PowerScale. +> * [CSM Observability](../observability/) does not support the Unity XT Storage Platform. +> * [CSM Replication](../replication/) does not support the Unity XT Storage Platform or File Replication for PowerMax. ## Supported Container Orchestrator Platforms @@ -39,4 +39,4 @@ weight: 1 > Notes: > * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. -> * Any orchestrator platform or version that's not mentioned here must be self-certified using [Cert-CSI](https://dell.github.io/csm-docs/docs/csidriver/installation/test/certcsi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. +> * Any orchestrator platform or version that's not mentioned here must be self-certified using [Cert-CSI](../certcsi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. From df15600055cf7c264185758f30d14c3620f7e59c Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Fri, 15 Mar 2024 08:51:19 -0300 Subject: [PATCH 31/49] Update _index.md --- content/docs/prerequisites/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index 1aa96fdb80..303c304d3d 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -39,4 +39,4 @@ weight: 1 > Notes: > * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. -> * Any orchestrator platform or version that's not mentioned here must be self-certified using [Cert-CSI](../certcsi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. +> * Any orchestrator platform or version that's not mentioned here must be self-certified using [Cert-CSI](../cert-csi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. From 0849d83811e9d2cd868e81ce07526372dd4400b6 Mon Sep 17 00:00:00 2001 From: Sean Gallacher Date: Fri, 15 Mar 2024 08:00:20 -0400 Subject: [PATCH 32/49] moved images --- .../{csidriver/test => cert-csi}/img/HTMLReport.png | Bin .../test => cert-csi}/img/interruptTest.png | Bin .../{csidriver/test => cert-csi}/img/listRuns.png | Bin .../{csidriver/test => cert-csi}/img/longevity.png | Bin .../test => cert-csi}/img/multiDBTabularReport.png | Bin .../test => cert-csi}/img/resourceUsage.png | Bin .../{csidriver/test => cert-csi}/img/scaling.PNG | Bin .../test => cert-csi}/img/tabularReport.png | Bin .../{csidriver/test => cert-csi}/img/textReport.png | Bin .../test => cert-csi}/img/unifiedTest.png | Bin 10 files changed, 0 insertions(+), 0 deletions(-) rename content/docs/{csidriver/test => cert-csi}/img/HTMLReport.png (100%) rename content/docs/{csidriver/test => cert-csi}/img/interruptTest.png (100%) rename content/docs/{csidriver/test => cert-csi}/img/listRuns.png (100%) rename content/docs/{csidriver/test => cert-csi}/img/longevity.png (100%) rename content/docs/{csidriver/test => cert-csi}/img/multiDBTabularReport.png (100%) rename content/docs/{csidriver/test => cert-csi}/img/resourceUsage.png (100%) rename content/docs/{csidriver/test => cert-csi}/img/scaling.PNG (100%) rename content/docs/{csidriver/test => cert-csi}/img/tabularReport.png (100%) rename content/docs/{csidriver/test => cert-csi}/img/textReport.png (100%) rename content/docs/{csidriver/test => cert-csi}/img/unifiedTest.png (100%) diff --git a/content/docs/csidriver/test/img/HTMLReport.png b/content/docs/cert-csi/img/HTMLReport.png similarity index 100% rename from content/docs/csidriver/test/img/HTMLReport.png rename to content/docs/cert-csi/img/HTMLReport.png diff --git a/content/docs/csidriver/test/img/interruptTest.png b/content/docs/cert-csi/img/interruptTest.png similarity index 100% rename from content/docs/csidriver/test/img/interruptTest.png rename to content/docs/cert-csi/img/interruptTest.png diff --git a/content/docs/csidriver/test/img/listRuns.png b/content/docs/cert-csi/img/listRuns.png similarity index 100% rename from content/docs/csidriver/test/img/listRuns.png rename to content/docs/cert-csi/img/listRuns.png diff --git a/content/docs/csidriver/test/img/longevity.png b/content/docs/cert-csi/img/longevity.png similarity index 100% rename from content/docs/csidriver/test/img/longevity.png rename to content/docs/cert-csi/img/longevity.png diff --git a/content/docs/csidriver/test/img/multiDBTabularReport.png b/content/docs/cert-csi/img/multiDBTabularReport.png similarity index 100% rename from content/docs/csidriver/test/img/multiDBTabularReport.png rename to content/docs/cert-csi/img/multiDBTabularReport.png diff --git a/content/docs/csidriver/test/img/resourceUsage.png b/content/docs/cert-csi/img/resourceUsage.png similarity index 100% rename from content/docs/csidriver/test/img/resourceUsage.png rename to content/docs/cert-csi/img/resourceUsage.png diff --git a/content/docs/csidriver/test/img/scaling.PNG b/content/docs/cert-csi/img/scaling.PNG similarity index 100% rename from content/docs/csidriver/test/img/scaling.PNG rename to content/docs/cert-csi/img/scaling.PNG diff --git a/content/docs/csidriver/test/img/tabularReport.png b/content/docs/cert-csi/img/tabularReport.png similarity index 100% rename from content/docs/csidriver/test/img/tabularReport.png rename to content/docs/cert-csi/img/tabularReport.png diff --git a/content/docs/csidriver/test/img/textReport.png b/content/docs/cert-csi/img/textReport.png similarity index 100% rename from content/docs/csidriver/test/img/textReport.png rename to content/docs/cert-csi/img/textReport.png diff --git a/content/docs/csidriver/test/img/unifiedTest.png b/content/docs/cert-csi/img/unifiedTest.png similarity index 100% rename from content/docs/csidriver/test/img/unifiedTest.png rename to content/docs/cert-csi/img/unifiedTest.png From b3ab3dbe8498c67a3b5ab1496199ba875ee8e645 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Fri, 15 Mar 2024 09:24:50 -0300 Subject: [PATCH 33/49] Update _index.md --- content/docs/cert-csi/_index.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/content/docs/cert-csi/_index.md b/content/docs/cert-csi/_index.md index 44a66a6efc..ccb27d5d5b 100644 --- a/content/docs/cert-csi/_index.md +++ b/content/docs/cert-csi/_index.md @@ -803,41 +803,41 @@ cert-csi test --sc <...> --sh ./hooks/start.sh --rh ### Running provisioning test -![img1](../img/unifiedTest.png) +![img1](./img/unifiedTest.png) You can interrupt the application by sending an interruption signal (for example pressing Ctrl + C). It will stop polling and try to cleanup resources. -![img2](../img/interruptTest.png) +![img2](./img/interruptTest.png) ### Running scaling test -![img3](../img/scaling.PNG) +![img3](./img/scaling.PNG) ### Listing available test runs -![img4](../img/listRuns.png) +![img4](./img/listRuns.png) ### Running longevity mode -![img5](../img/longevity.png) +![img5](./img/longevity.png) ### Multi DB Tabular report example -![img6](../img/multiDBTabularReport.png) +![img6](./img/multiDBTabularReport.png) Text report example -![img7](../img/textReport.png) +![img7](./img/textReport.png) Tabular Report example -![img9](../img/tabularReport.png) +![img9](./img/tabularReport.png) ### HTML report example -![img8](../img/HTMLReport.png) +![img8](./img/HTMLReport.png) ### Resource usage example chart -![img9](../img/resourceUsage.png) +![img9](./img/resourceUsage.png) From 291b6b3f248699437d4c61cdf4d71b5d7d39496d Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Fri, 15 Mar 2024 09:44:59 -0300 Subject: [PATCH 34/49] Update _index.md --- content/docs/cert-csi/_index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/docs/cert-csi/_index.md b/content/docs/cert-csi/_index.md index ccb27d5d5b..a948609472 100644 --- a/content/docs/cert-csi/_index.md +++ b/content/docs/cert-csi/_index.md @@ -1,6 +1,7 @@ --- title: Cert-CSI linktitle: Cert-CSI +weight: 13 description: Tool to validate Dell CSI Drivers --- From 57a53024faeb653e9f2e3dbfa1f434df8429e18e Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Fri, 15 Mar 2024 10:30:54 -0300 Subject: [PATCH 35/49] Added EKS qual --- content/docs/cert-csi/qualified.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/docs/cert-csi/qualified.md b/content/docs/cert-csi/qualified.md index 9f55d9e5b4..c13a72d0d7 100644 --- a/content/docs/cert-csi/qualified.md +++ b/content/docs/cert-csi/qualified.md @@ -7,5 +7,6 @@ description: Community Qualified Configurations {{}} | cert-csi results | OS | CO | Storage Platform | Protocol | CSM | |--------------------------------------------------------|:----------:|:----------------:|:-----------------------:|:---------:|:----------:| -| [Ticket 1079](https://github.com/dell/csm/issues/1079) | Debian 10 | K3s v1.24.7+k3s1 | Unity VSA 5.3.1.0.5.008 | iSCSI | CSI v1.8.0 | +| [Ticket 1079](https://github.com/dell/csm/issues/1079) | Debian 10 | K3s v1.24.7+k3s1 | Unity VSA 5.3.1.0.5.008 | iSCSI | CSI v2.8.0 | +| [Ticket 1177](https://github.com/dell/csm/issues/1177) | Ubuntu OS 22.04 | Amazon EKS (K8s 1.29) | PowerFlex | SCINI | CSI v2.9.2 | {{
}} From 8b980f1ae3726d6b6e55599dde77b5e8a3533aea Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Fri, 15 Mar 2024 10:36:08 -0300 Subject: [PATCH 36/49] Removed Elastic Kubernetes Anywhere --- content/docs/prerequisites/_index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index 303c304d3d..7952e3ab65 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -34,7 +34,6 @@ weight: 1 | Mirantis Kubernetes Engine | 3.7.x | | Google Anthos | 1.15 | | Rancher Kubernetes Engine | 1.4.x| -| Amazon Elastic Kubernetes Service | ?? | {{}} > Notes: From 52919f23c2283740d475a8de7de9e61da5b15965 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Fri, 15 Mar 2024 10:38:13 -0300 Subject: [PATCH 37/49] Added VMware Tanzu --- content/docs/prerequisites/_index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index 7952e3ab65..d5ac28e7a9 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -33,7 +33,8 @@ weight: 1 | Red Hat OpenShift | 4.13, 4.14 | | Mirantis Kubernetes Engine | 3.7.x | | Google Anthos | 1.15 | -| Rancher Kubernetes Engine | 1.4.x| +| Rancher Kubernetes Engine | 1.4.x | +| VMware Tanzu | 7.0 | {{}} > Notes: From 15e8630b47440dad83bcd28f06f2e22c60753929 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Fri, 15 Mar 2024 10:44:40 -0300 Subject: [PATCH 38/49] updated cert-csi note --- content/docs/prerequisites/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index d5ac28e7a9..968afc6a46 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -39,4 +39,4 @@ weight: 1 > Notes: > * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. -> * Any orchestrator platform or version that's not mentioned here must be self-certified using [Cert-CSI](../cert-csi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. +> * Any orchestrator platform, operating system or version that's not mentioned here must be self-certified using [Cert-CSI](../cert-csi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. From 3b52fcde4d2b833d3aa497ab92f1fa56aed0ce01 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Fri, 15 Mar 2024 12:06:38 -0300 Subject: [PATCH 39/49] Removed storage platforms for page --- content/docs/csidriver/_index.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/content/docs/csidriver/_index.md b/content/docs/csidriver/_index.md index bbdd747ffc..31bf3ed257 100644 --- a/content/docs/csidriver/_index.md +++ b/content/docs/csidriver/_index.md @@ -34,12 +34,6 @@ The CSI Drivers by Dell implement an interface between [CSI](https://kubernetes- | Storage Capacity Tracking | yes | yes | yes | yes | yes | | Volume Limit | yes | yes | yes | yes | yes | {{}} -### Supported Storage Platforms -{{}} -| | PowerMax | PowerFlex | Unity XT | PowerScale | PowerStore | -|---------------|:-------------------------------------------------------:|:----------------:|:--------------------------:|:----------------------------------:|:----------------:| -| Storage Array | PowerMax 2500/8500 PowerMaxOS 10 (6079) , PowerMaxOS 10.0.1 (6079) , PowerMaxOS 10.1 (6079)
PowerMax 2000/8000 - 5978.711.xxx, 5978.479.xxx
Unisphere 10.0,10.0.1,10.1 | 3.6.x, 4.0.x, 4.5.x | 5.1.x, 5.2.x, 5.3.0 | OneFS 9.3, 9.4, 9.5.0.x (x >=5) | 3.0, 3.2, 3.5, 3.6 | -{{
}} ### Backend Storage Details {{}} From ffe0f958d3daff516216e8c1f4b68ad8f2e54ea4 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Mon, 18 Mar 2024 13:37:10 -0300 Subject: [PATCH 40/49] Removed support matrix --- content/docs/deployment/csmoperator/_index.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/content/docs/deployment/csmoperator/_index.md b/content/docs/deployment/csmoperator/_index.md index 6307c61afd..34bfcf87fa 100644 --- a/content/docs/deployment/csmoperator/_index.md +++ b/content/docs/deployment/csmoperator/_index.md @@ -7,17 +7,6 @@ weight: 1 The Dell Container Storage Modules Operator is a Kubernetes Operator, which can be used to install and manage the CSI Drivers and CSM Modules provided by Dell for various storage platforms. This operator is available as a community operator for upstream Kubernetes and can be deployed using OperatorHub.io. The operator can be installed using OLM (Operator Lifecycle Manager) or manually. -## Support Matrix - -Dell CSM Operator has been tested and qualified on Upstream Kubernetes and OpenShift. Supported versions are listed below: - -| Kubernetes Version | OpenShift Version | -| -------------------------- | -------------------- | -| 1.27, 1.28, 1.29 | 4.13, 4.14 | - ->NOTE: ->- Authorization module is only supported on Kubernetes platforms. - ## Supported CSM Components The table below lists the driver and modules versions installable with the CSM Operator: From 6aa1ef1b9675511fc6d5e6d1a3c754209aedba3b Mon Sep 17 00:00:00 2001 From: vyshakav-dell <136616665+vyshakav-dell@users.noreply.github.com> Date: Mon, 18 Mar 2024 11:48:40 -0500 Subject: [PATCH 41/49] Update _index.md (#1031) * Update _index.md Added few lines to demonstrate the need for cert-csi, re-iterate support for self-certified customers, and enable customers to submit test results. * Update _index.md Removed line - 'As we begin to see more flavors of operating systems and orchestrator platforms, qualifying Dell CSI drivers against different storage arrays is a challenging task. Validating five different storage arrays with different permutations of orchestrators, operating systems, and their versions is not feasible.' --- content/docs/cert-csi/_index.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/content/docs/cert-csi/_index.md b/content/docs/cert-csi/_index.md index a948609472..af720fce0d 100644 --- a/content/docs/cert-csi/_index.md +++ b/content/docs/cert-csi/_index.md @@ -5,7 +5,13 @@ weight: 13 description: Tool to validate Dell CSI Drivers --- -Cert-CSI is a tool to validate Dell CSI Drivers. It contains various test suites to validate the drivers. +Dell's Cert-CSI is a tool to validate Dell CSI Drivers. It contains various test suites to validate the drivers and addresses the complexity involved with certifiying Dell CSI drivers in different customer environments. + +Any orchestrator platform, operating system or version that is not mentioned in the support matrix but self-certified by the customer using cert-csi is supported for customer use. + +## Submit Test Results + +Please submit your test results for our review here (https://github.com/dell/csm/issues/) by clicking on 'New Issue'. If the results are a success, the orchestrator platform and version will be published under Community Qualified Configurations. ## Installation From f953e1f2339fa3c115249956c375e275d7a3f8d3 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Mon, 18 Mar 2024 14:12:07 -0300 Subject: [PATCH 42/49] Added module support matrix --- content/docs/prerequisites/_index.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index 968afc6a46..c505d8af52 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -40,3 +40,19 @@ weight: 1 > Notes: > * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. > * Any orchestrator platform, operating system or version that's not mentioned here must be self-certified using [Cert-CSI](../cert-csi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. + +## Supported CSM Modules + +{{
}} +| CSM Module | PowerMax | PowerFlex | Unity XT | PowerScale | PowerStore | +|---------------|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:| +| Authorization | Yes | Yes | Yes | Yes | No | +| Observability | Yes | Yes | No | Yes | Yes | +| Replication | Yes | Yes | No | Yes | Yes | +| Resiliency | No | Yes | Yes | Yes | Yes | +| Encryption | No | No | No | Yes | No | +| Application Mobility | Yes | Yes | Yes | Yes | Yes | +{{
}} + +> Notes: +> * Encryption and Application Mobility are available as a Technical Preview only and are not officially supported. From d42f1e5c698b47e32e157a1a079b913a50e12e10 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Mon, 18 Mar 2024 14:15:07 -0300 Subject: [PATCH 43/49] Added links to table --- content/docs/prerequisites/_index.md | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index c505d8af52..3b0ab12c1f 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -19,10 +19,7 @@ weight: 1 {{}} > Notes: -> * The required OS dependencies are only for the protocol needed (e.g. if NVMe isn't the storage access protocol then nvme-cli is not required). -> * [CSM Authorization](../authorization/) only supports Kubernetes and the following Storage Platforms: PowerMax, PowerFlex, and PowerScale. -> * [CSM Observability](../observability/) does not support the Unity XT Storage Platform. -> * [CSM Replication](../replication/) does not support the Unity XT Storage Platform or File Replication for PowerMax. +> * The required OS dependencies are only for the protocol needed (e.g. if NVMe isn't the storage access protocol then nvme-cli is not required).. ## Supported Container Orchestrator Platforms @@ -46,12 +43,12 @@ weight: 1 {{}} | CSM Module | PowerMax | PowerFlex | Unity XT | PowerScale | PowerStore | |---------------|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:| -| Authorization | Yes | Yes | Yes | Yes | No | -| Observability | Yes | Yes | No | Yes | Yes | -| Replication | Yes | Yes | No | Yes | Yes | -| Resiliency | No | Yes | Yes | Yes | Yes | -| Encryption | No | No | No | Yes | No | -| Application Mobility | Yes | Yes | Yes | Yes | Yes | +| [CSM Authorization](../authorization/) | Yes | Yes | Yes | Yes | No | +| [CSM Observability](../observability/) | Yes | Yes | No | Yes | Yes | +| [CSM Replication](../replication/) | Yes | Yes | No | Yes | Yes | +| [CSM Resiliency](../resiliency/) | No | Yes | Yes | Yes | Yes | +| [CSM Encryption](../secure/) | No | No | No | Yes | No | +| [CSM Application Mobility](../applicationmobility/) | Yes | Yes | Yes | Yes | Yes | {{
}} > Notes: From 924494d0eb8c89e45a7e3bdd6142c1c700072ae0 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Mon, 18 Mar 2024 14:26:52 -0300 Subject: [PATCH 44/49] updated link --- content/docs/prerequisites/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index 3b0ab12c1f..500ac71741 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -47,7 +47,7 @@ weight: 1 | [CSM Observability](../observability/) | Yes | Yes | No | Yes | Yes | | [CSM Replication](../replication/) | Yes | Yes | No | Yes | Yes | | [CSM Resiliency](../resiliency/) | No | Yes | Yes | Yes | Yes | -| [CSM Encryption](../secure/) | No | No | No | Yes | No | +| [CSM Encryption](../secure/encryption/) | No | No | No | Yes | No | | [CSM Application Mobility](../applicationmobility/) | Yes | Yes | Yes | Yes | Yes | {{}} From cd0b731e09d4f63f15f998988af3c0c32cdd4381 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Tue, 19 Mar 2024 11:31:21 -0300 Subject: [PATCH 45/49] Added tested host operating systems --- content/docs/prerequisites/_index.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index 500ac71741..38a5ce1eca 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -38,6 +38,13 @@ weight: 1 > * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. > * Any orchestrator platform, operating system or version that's not mentioned here must be self-certified using [Cert-CSI](../cert-csi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. +## Tested Host Operating Systems + +Container Storage Modules (CSM) does not officially support specific operating systems. However, the following operating systems are known to work: +- RedHat CoreOS (RHCOS) versions as supported by OpenShift Container Platform +- RHEL 8+ +- SLES 15SP5 + ## Supported CSM Modules {{}} From cf3c35bc1e4cb02decd27855816fa5bee5b2b5b6 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Tue, 19 Mar 2024 11:40:32 -0300 Subject: [PATCH 46/49] Removed host operating system note --- content/docs/prerequisites/_index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index 38a5ce1eca..955b128d45 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -35,7 +35,6 @@ weight: 1 {{
}} > Notes: -> * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. > * Any orchestrator platform, operating system or version that's not mentioned here must be self-certified using [Cert-CSI](../cert-csi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. ## Tested Host Operating Systems From 62081dc90e26eb118e6cc62de8b3c083ca60f3d0 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Tue, 19 Mar 2024 16:34:41 -0300 Subject: [PATCH 47/49] Updated cert-csi note --- content/docs/prerequisites/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index 955b128d45..e26a5f1686 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -35,7 +35,7 @@ weight: 1 {{}} > Notes: -> * Any orchestrator platform, operating system or version that's not mentioned here must be self-certified using [Cert-CSI](../cert-csi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. +> * Any orchestrator platform or version that's not mentioned here must be self-certified using [Cert-CSI](../cert-csi/) in order to be supported. Although not mandatory, we recommend users to use orchestrator platforms and versions that have not met their end of life. ## Tested Host Operating Systems From b0650e7fd8f021bc7555ef0774534ecf737ff120 Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Wed, 20 Mar 2024 08:23:55 -0300 Subject: [PATCH 48/49] Update _index.md --- content/docs/prerequisites/_index.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/content/docs/prerequisites/_index.md b/content/docs/prerequisites/_index.md index e26a5f1686..45f270a75b 100644 --- a/content/docs/prerequisites/_index.md +++ b/content/docs/prerequisites/_index.md @@ -43,6 +43,11 @@ Container Storage Modules (CSM) does not officially support specific operating s - RedHat CoreOS (RHCOS) versions as supported by OpenShift Container Platform - RHEL 8+ - SLES 15SP5 +- Ubuntu 22.04 + +> Notes: +> * The host operating system/version being used must align with what each Dell Storage platform supports. Please visit [E-Lab Navigator](https://elabnavigator.dell.com/eln/modernHomeSSM) for specific Dell Storage platform host operating system level support matrices. +> * Any operating system or version that's not mentioned here must be self-certified using [Cert-CSI](../cert-csi/) in order to be supported. Although not mandatory, we recommend users to use operating systems and versions that have not met their end of life. ## Supported CSM Modules From 62368ebf36af654f3a9f799b15a023844e0dfb4b Mon Sep 17 00:00:00 2001 From: gallacher <35462391+gallacher@users.noreply.github.com> Date: Wed, 20 Mar 2024 09:24:16 -0300 Subject: [PATCH 49/49] Update qualified.md