Running CERT-CSI tests.....
[2024-10-21 10:53:12]  INFO Starting cert-csi; ver. 1.5.0
Does it look OK? (Y)es/(n)o
-> [2024-10-21 10:53:12]  INFO Suites to run with powermax-iscsi storage class:
[2024-10-21 10:53:12]  INFO 1. VolumeIoSuite {volumes: 2, volumeSize: 5Gi chains: 2-2}
[2024-10-21 10:53:12]  INFO 2. ScalingSuite {replicas: 2, volumes: 5, volumeSize: 5Gi}
[2024-10-21 10:53:12]  INFO 3. CloneVolumeSuite {pods: 2, volumes: 1, volumeSize: 5Gi}
[2024-10-21 10:53:12]  INFO 4. VolumeExpansionSuite {pods: 1, volumes: 1, size: 5Gi, expSize: 10Gi, block: false}
[2024-10-21 10:53:12]  INFO 5. VolumeExpansionSuite {pods: 1, volumes: 1, size: 5Gi, expSize: 10Gi, block: true}
[2024-10-21 10:53:12]  INFO 6. SnapSuite {snapshots: 3, volumeSize; 5Gi}
[2024-10-21 10:53:12]  INFO 7. ReplicationSuite {pods: 2, volumes: 5, volumeSize: 5Gi}
[2024-10-21 10:53:12]  INFO 8. MultiAttachSuite {pods: 5, rawBlock: true, size: 5Gi, accMode: ReadWriteMany}
[2024-10-21 10:53:12]  INFO 9. CapacityTrackingSuite {DriverNamespace: powermax, volumeSize: 5Gi, pollInterval: 0s}
[2024-10-21 10:53:12] ERROR EOF
[2024-10-21 10:53:12]  INFO Using EVENT observer type
[2024-10-21 10:53:12]  INFO Using default config
[2024-10-21 10:53:12]  INFO Successfully loaded config. Host: https://api.ocppmx.hop.lab.emc.com:6443
[2024-10-21 10:53:12]  INFO Created new KubeClient
[2024-10-21 10:53:12]  INFO Running 1 iteration(s)
[2024-10-21 10:53:12]  INFO 	*** ITERATION NUMBER 1 ***	
[2024-10-21 10:53:12]  INFO Starting CloneVolumeSuite with powermax-iscsi storage class
[2024-10-21 10:53:12]  INFO Starting VolumeIoSuite with powermax-iscsi storage class
[2024-10-21 10:53:12]  INFO Starting VolumeExpansionSuite with powermax-iscsi storage class
[2024-10-21 10:53:12]  INFO Starting ReplicationSuite with powermax-iscsi storage class
[2024-10-21 10:53:12]  INFO Starting MultiAttachSuite with powermax-iscsi storage class
[2024-10-21 10:53:12]  INFO Successfully created namespace clonevolume-suite-4010a19a
[2024-10-21 10:53:12]  INFO Using default image: docker.io/centos:latest
[2024-10-21 10:53:12]  INFO Creating 2 pods, each with 1 volumes
[2024-10-21 10:53:12]  INFO Starting SnapSuite with powermax-iscsi storage class
[2024-10-21 10:53:12]  INFO Starting ScalingSuite with powermax-iscsi storage class
[2024-10-21 10:53:12]  INFO Successfully created namespace volumeio-test-be2a55bf
[2024-10-21 10:53:12]  INFO Using default image: docker.io/centos:latest
[2024-10-21 10:53:12]  INFO Starting CapacityTrackingSuite with powermax-iscsi storage class
[2024-10-21 10:53:12]  INFO Successfully created namespace volume-expansion-suite-19bc2ef9
[2024-10-21 10:53:12]  INFO Using default image: docker.io/centos:latest
[2024-10-21 10:53:12]  INFO Creating 1 pods, each with 1 volumes of size (5Gi)
[2024-10-21 10:53:12]  INFO Starting VolumeExpansionSuite with powermax-iscsi storage class
[2024-10-21 10:53:12]  INFO Successfully created namespace replication-suite-13e60029
[2024-10-21 10:53:12]  INFO Successfully created namespace snap-test-f3ed315a
[2024-10-21 10:53:12]  INFO Creating IO pod
[2024-10-21 10:53:12]  INFO Successfully created namespace mas-test-ae482977
[2024-10-21 10:53:12]  INFO Using default image: docker.io/centos:latest
[2024-10-21 10:53:12]  INFO Creating Volume
[2024-10-21 10:53:12]  INFO Successfully created namespace scale-test-ff251afe
[2024-10-21 10:53:12]  INFO Using default image: docker.io/centos:latest
[2024-10-21 10:53:12]  INFO Using default image: docker.io/centos:latest
[2024-10-21 10:53:12]  INFO Creating Snapshot pod
[2024-10-21 10:53:12]  INFO Using default image: docker.io/centos:latest
[2024-10-21 10:53:12]  INFO Creating 2 pods, each with 5 volumes
[2024-10-21 10:53:12]  INFO Successfully created namespace capacity-tracking-test-f4e8cb81
[2024-10-21 10:53:13]  INFO Found 1 topology segment(s) in csinode
[2024-10-21 10:53:13]  INFO Using default image: docker.io/centos:latest
[2024-10-21 10:53:13]  INFO Creating capacity-tracking-e07f8950 storage class
[2024-10-21 10:53:13]  INFO Waiting for CSIStorageCapacity to be CREATED for capacity-tracking-e07f8950 storage class in powermax namespace
[2024-10-21 10:53:13]  INFO Successfully created namespace volume-expansion-suite-3149a088
[2024-10-21 10:53:13]  INFO Using default image: docker.io/centos:latest
[2024-10-21 10:53:13]  INFO Creating 1 pods, each with 1 volumes of size (5Gi)
I1021 10:53:13.643586  143590 request.go:697] Waited for 1.147342945s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/volumeio-test-be2a55bf/persistentvolumeclaims
[2024-10-21 10:53:13]  INFO Created stateful set sts-scale-testjm5bn
[2024-10-21 10:53:13]  INFO Waiting for sts pods to become ready
W1021 10:53:14.053190  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "prov-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "prov-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "prov-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "prov-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:53:14]  INFO Attaching Volume to original pod
[2024-10-21 10:53:15]  INFO All CSIStorageCapacity are created in 2.008383211s
[2024-10-21 10:53:15]  INFO Deleting capacity-tracking-e07f8950 storage class,
[2024-10-21 10:53:15]  INFO Waiting for CSIStorageCapacity to be DELETED for capacity-tracking-e07f8950 storage class in powermax namespace
W1021 10:53:16.058295  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "prov-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "prov-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "prov-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "prov-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:53:16]  INFO Waiting for all pods in volume-expansion-suite-19bc2ef9 to be READY
[2024-10-21 10:53:17]  INFO All CSIStorageCapacity are deleted in 2.009661162s
[2024-10-21 10:53:17]  INFO Updating CSIStorageCapacity for powermax-iscsi storage class, setting capacity to 0
[2024-10-21 10:53:17]  INFO Creating capacity-tracking-pod-147e3e35 pod using powermax-iscsi storage class
W1021 10:53:17.450800  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "iowriter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "iowriter" must set securityContext.capabilities.drop=["ALL"]; container "iowriter" must not include "SYS_ADMIN" in securityContext.capabilities.add), runAsNonRoot != true (pod or container "iowriter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "iowriter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:53:17]  INFO Waiting for all pods in mas-test-ae482977 to be READY
W1021 10:53:18.058826  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "iowriter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "iowriter" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "iowriter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "iowriter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:53:18]  INFO Waiting for pod iowriter-test-4xppx to be READY
W1021 10:53:19.255125  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "prov-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "prov-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "prov-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "prov-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:53:19]  INFO Waiting for all pods in volume-expansion-suite-3149a088 to be READY
W1021 10:53:19.651965  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "iowriter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "iowriter" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "iowriter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "iowriter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:53:19]  INFO Waiting for pod iowriter-test-2mk76 to be READY
W1021 10:53:19.856982  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "prov-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "prov-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "prov-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "prov-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:53:19]  INFO Waiting for all pods in clonevolume-suite-4010a19a to be READY
W1021 10:53:22.256546  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "capacity-tracking-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "capacity-tracking-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "capacity-tracking-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "capacity-tracking-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W1021 10:53:23.451203  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "iowriter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "iowriter" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "iowriter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "iowriter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:53:23]  INFO Waiting for pod iowriter-test-vdcjk to be READY
I1021 10:53:23.843543  143590 request.go:697] Waited for 1.595783615s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/mas-test-ae482977/pods
W1021 10:53:28.252107  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "prov-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "prov-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "prov-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "prov-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:53:30]  INFO capacity-tracking-pod-147e3e35 pod is PENDING
[2024-10-21 10:53:30]  INFO Waiting for provisioner to POLL GetCapacity for powermax-iscsi storage class in 0s
I1021 10:53:33.843723  143590 request.go:697] Waited for 1.595930504s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/mas-test-ae482977/pods
W1021 10:53:42.660671  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "prov-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "prov-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "prov-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "prov-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:53:42]  INFO Waiting for all pods in replication-suite-13e60029 to be READY
I1021 10:53:44.042989  143590 request.go:697] Waited for 1.595122171s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/snap-test-f3ed315a/pods/iowriter-test-4xppx
I1021 10:53:54.043337  143590 request.go:697] Waited for 1.795876134s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/mas-test-ae482977/pods
I1021 10:54:04.043757  143590 request.go:697] Waited for 1.596387157s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/snap-test-f3ed315a/pods/iowriter-test-4xppx
[2024-10-21 10:54:09]  INFO All pods are ready in 53.78776508s
[2024-10-21 10:54:09]  INFO Requested 5242880 KiB, got 5177984 KiB -- delta is -64896 KiB
[2024-10-21 10:54:09]  INFO Expanding the size of volumes from 5Gi to 10Gi
[2024-10-21 10:54:10]  INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data]
[2024-10-21 10:54:11]  INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512]
[2024-10-21 10:54:11]  INFO Executing command: [dd if=/dev/urandom of=/data0/writer-0.data bs=1M count=128 oflag=sync]
[2024-10-21 10:54:11]  INFO Scaling to 2 replicas
[2024-10-21 10:54:11]  INFO Waiting for sts pods to become ready
[2024-10-21 10:54:13]  INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-1.data]
[2024-10-21 10:54:13]  INFO All pods are ready in 53.590815367s
[2024-10-21 10:54:13]  INFO Creating new pods with cloned volumes mounted on them
[2024-10-21 10:54:13]  INFO Executing command: [/bin/bash -c sha512sum /data0/writer-1.data > /data0/writer-1.sha512]
I1021 10:54:14.242808  143590 request.go:697] Waited for 1.394424187s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/snap-test-f3ed315a/pods
W1021 10:54:17.252334  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "prov-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "prov-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "prov-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "prov-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:54:18]  INFO Waiting until no Volume Attachments with PV  left
[2024-10-21 10:54:18]  INFO VolumeAttachment deleted
W1021 10:54:21.463775  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "iowriter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "iowriter" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "iowriter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "iowriter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:54:21]  INFO Waiting for pod iowriter-test-xpp4c to be READY
W1021 10:54:22.252374  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "prov-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "prov-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "prov-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "prov-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:54:22]  INFO Waiting for all pods in clonevolume-suite-4010a19a to be READY
[2024-10-21 10:54:23]  INFO All pods are ready in 1m5.59613572s
[2024-10-21 10:54:23]  INFO Checking if pod-prov-test-gpxk2/data0 was properly resized
[2024-10-21 10:54:24]  INFO Waiting until no Volume Attachments with PV  left
[2024-10-21 10:54:24]  INFO VolumeAttachment deleted
I1021 10:54:24.243734  143590 request.go:697] Waited for 2.778892336s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/volumeio-test-be2a55bf/pods/iowriter-test-xpp4c
[2024-10-21 10:54:25]  INFO Creating new pods with original Volume attached to them
[2024-10-21 10:54:26]  INFO All pods are ready in 1m6.992947709s
[2024-10-21 10:54:26]  INFO Expanding the size of volumes from 5Gi to 10Gi
W1021 10:54:27.251120  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "iowriter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "iowriter" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "iowriter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "iowriter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:54:27]  INFO Waiting for pod iowriter-test-qgvtm to be READY
W1021 10:54:28.070622  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "iowriter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "iowriter" must set securityContext.capabilities.drop=["ALL"]; container "iowriter" must not include "SYS_ADMIN" in securityContext.capabilities.add), runAsNonRoot != true (pod or container "iowriter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "iowriter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:54:29]  INFO All pods are ready in 46.587664992s
[2024-10-21 10:54:29]  INFO Creating a snapshot on each of the volumes
W1021 10:54:30.455953  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "iowriter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "iowriter" must set securityContext.capabilities.drop=["ALL"]; container "iowriter" must not include "SYS_ADMIN" in securityContext.capabilities.add), runAsNonRoot != true (pod or container "iowriter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "iowriter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:54:30]  INFO Deleting all resources in namespace volume-expansion-suite-3149a088
W1021 10:54:31.854978  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "iowriter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "iowriter" must set securityContext.capabilities.drop=["ALL"]; container "iowriter" must not include "SYS_ADMIN" in securityContext.capabilities.add), runAsNonRoot != true (pod or container "iowriter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "iowriter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W1021 10:54:33.460502  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "iowriter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "iowriter" must set securityContext.capabilities.drop=["ALL"]; container "iowriter" must not include "SYS_ADMIN" in securityContext.capabilities.add), runAsNonRoot != true (pod or container "iowriter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "iowriter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
I1021 10:54:34.443279  143590 request.go:697] Waited for 1.79334285s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/volume-expansion-suite-19bc2ef9/persistentvolumeclaims
W1021 10:54:36.063651  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "iowriter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "iowriter" must set securityContext.capabilities.drop=["ALL"]; container "iowriter" must not include "SYS_ADMIN" in securityContext.capabilities.add), runAsNonRoot != true (pod or container "iowriter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "iowriter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:54:36]  INFO Waiting for all pods in mas-test-ae482977 to be READY
[2024-10-21 10:54:37]  INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-1.sha512]
[2024-10-21 10:54:44]  INFO Writer pod: iowriter-test-4xppx
[2024-10-21 10:54:44]  INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512]
I1021 10:54:44.643173  143590 request.go:697] Waited for 1.795863228s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/scale-test-ff251afe/pods?labelSelector=app%3Dunified-test
[2024-10-21 10:54:47]  INFO Scaling to 0 replicas
[2024-10-21 10:54:47]  INFO Waiting for sts pods to become ready
[2024-10-21 10:54:48]  INFO Namespace volume-expansion-suite-3149a088 was deleted in 17.193610003s
[2024-10-21 10:54:48]  INFO Waiting for volumeattachments to be deleted
[2024-10-21 10:54:49]  INFO Deleting all resources in namespace scale-test-ff251afe
[2024-10-21 10:54:51]  INFO Waiting for the snapshots in replication-suite-13e60029 to be READY
[2024-10-21 10:54:52]  INFO Creating 3 snapshots
[2024-10-21 10:54:52]  INFO Waiting for Snapshot 'snap' to be READY
[2024-10-21 10:54:53]  INFO Hashes match
[2024-10-21 10:54:53]  INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-1.data]
I1021 10:54:54.643748  143590 request.go:697] Waited for 1.389921918s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/replication-suite-13e60029/pods
[2024-10-21 10:54:55]  INFO Executing command: [/bin/bash -c sha512sum /data0/writer-1.data > /data0/writer-1.sha512]
[2024-10-21 10:54:56]  INFO All pods are ready in 34.197126761s
[2024-10-21 10:54:56]  INFO Deleting all resources in namespace clonevolume-suite-4010a19a
[2024-10-21 10:55:03]  INFO Waiting until no Volume Attachments with PV  left
[2024-10-21 10:55:03]  INFO VolumeAttachment deleted
I1021 10:55:04.843583  143590 request.go:697] Waited for 1.589942943s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/replication-suite-13e60029/pods
[2024-10-21 10:55:08]  INFO Namespace scale-test-ff251afe was deleted in 18.805862925s
[2024-10-21 10:55:08]  INFO Waiting for volumeattachments to be deleted
[2024-10-21 10:55:12]  INFO Namespace clonevolume-suite-4010a19a was deleted in 16.403659238s
[2024-10-21 10:55:12]  INFO Waiting for volumeattachments to be deleted
I1021 10:55:15.843390  143590 request.go:697] Waited for 1.191589771s due to client-side throttling, not priority and fairness, request: GET:https://api.ocppmx.hop.lab.emc.com:6443/api/v1/namespaces/volumeio-test-be2a55bf/persistentvolumeclaims
[2024-10-21 10:55:25]  INFO SUCCESS: VolumeExpansionSuite in 2m12.995671033s
[2024-10-21 10:55:26]  INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-0.sha512]
[2024-10-21 10:55:34]  WARN pod-prov-test-gpxk2/data0 expanded within ±5% of request size
[2024-10-21 10:55:34]  INFO Deleting all resources in namespace volume-expansion-suite-19bc2ef9
[2024-10-21 10:55:34]  INFO SUCCESS: CloneVolumeSuite in 2m22.268166123s
[2024-10-21 10:55:34]  INFO SUCCESS: ScalingSuite in 2m22.343719516s
[2024-10-21 10:55:35]  INFO Creating new pods with replicated volumes mounted on them
W1021 10:55:35.719112  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "prov-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "prov-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "prov-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "prov-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W1021 10:55:36.285231  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "prov-test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "prov-test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "prov-test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "prov-test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:55:36]  INFO Waiting for all pods in replication-suite-13e60029 to be READY
[2024-10-21 10:55:41]  INFO Hashes match
[2024-10-21 10:55:41]  INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data]
[2024-10-21 10:55:44]  INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512]
[2024-10-21 10:55:46]  INFO Namespace volume-expansion-suite-19bc2ef9 was deleted in 12.034208444s
[2024-10-21 10:55:46]  INFO Waiting for volumeattachments to be deleted
[2024-10-21 10:55:48]  INFO Waiting until no Volume Attachments with PV  left
[2024-10-21 10:55:48]  INFO VolumeAttachment deleted
[2024-10-21 10:55:48]  INFO Deleting all resources in namespace volumeio-test-be2a55bf
[2024-10-21 10:55:50]  INFO All pods are ready in 1m14.591683248s
[2024-10-21 10:55:50]  INFO Executing command: [dd if=/dev/urandom of=/dev/data0 bs=1M count=128 oflag=sync]
[2024-10-21 10:55:54]  INFO Waiting for Snapshot 'snap1' to be READY
[2024-10-21 10:55:56]  INFO SUCCESS: VolumeExpansionSuite in 2m44.242709883s
[2024-10-21 10:56:02]  INFO Namespace volumeio-test-be2a55bf was deleted in 14.021573323s
[2024-10-21 10:56:02]  INFO Waiting for volumeattachments to be deleted
[2024-10-21 10:56:08]  INFO SUCCESS: VolumeIoSuite in 2m56.299464363s
[2024-10-21 10:56:18]  INFO Waiting for Snapshot 'snap2' to be READY
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 28.8113 s, 4.7 MB/s
[2024-10-21 10:56:19]  INFO Executing command: [dd if=/dev/data0 of=/tmp/blob.data bs=1M count=128]
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 0.128041 s, 1.0 GB/s
[2024-10-21 10:56:19]  INFO Executing command: [sha512sum /tmp/blob.data]
[2024-10-21 10:56:20]  INFO OriginalPod: iowriter-test-m7blf
[2024-10-21 10:56:20]  INFO hash sum is:d8cfbfbcfc0fe8bc83ecbe34026c011bf13e5bd1af8e99942d712b327e8aa748f9f047fe85f5f9bd1dff18043348807b885f69a8b99df2dc88bde1c06051d2b8  /tmp/blob.data

[2024-10-21 10:56:20]  INFO Checking hash sum on all of the other pods
[2024-10-21 10:56:20]  INFO Executing command: [blockdev --flushbufs /dev/data0]
[2024-10-21 10:56:20]  INFO Executing command: [dd if=/dev/data0 of=/tmp/blob.data bs=1M count=128]
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 13.2417 s, 10.1 MB/s
[2024-10-21 10:56:33]  INFO Executing command: [sha512sum /tmp/blob.data]
[2024-10-21 10:56:34]  INFO Pod: iowriter-test-457bn
[2024-10-21 10:56:34]  INFO hash sum is:d8cfbfbcfc0fe8bc83ecbe34026c011bf13e5bd1af8e99942d712b327e8aa748f9f047fe85f5f9bd1dff18043348807b885f69a8b99df2dc88bde1c06051d2b8  /tmp/blob.data

[2024-10-21 10:56:34]  INFO Hashes match
[2024-10-21 10:56:34]  INFO Executing command: [blockdev --flushbufs /dev/data0]
[2024-10-21 10:56:34]  INFO Executing command: [dd if=/dev/data0 of=/tmp/blob.data bs=1M count=128]
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 12.9613 s, 10.4 MB/s
[2024-10-21 10:56:47]  INFO Executing command: [sha512sum /tmp/blob.data]
[2024-10-21 10:56:48]  INFO Pod: iowriter-test-ctx8x
[2024-10-21 10:56:48]  INFO hash sum is:d8cfbfbcfc0fe8bc83ecbe34026c011bf13e5bd1af8e99942d712b327e8aa748f9f047fe85f5f9bd1dff18043348807b885f69a8b99df2dc88bde1c06051d2b8  /tmp/blob.data

[2024-10-21 10:56:48]  INFO Hashes match
[2024-10-21 10:56:48]  INFO Executing command: [blockdev --flushbufs /dev/data0]
[2024-10-21 10:56:48]  INFO Executing command: [dd if=/dev/data0 of=/tmp/blob.data bs=1M count=128]
[2024-10-21 10:56:48]  INFO Restoring from snap2
[2024-10-21 10:56:48]  INFO Creating pvc snap2-restore
W1021 10:56:48.947629  143590 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "iowriter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "iowriter" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "iowriter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "iowriter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
[2024-10-21 10:56:48]  INFO Waiting for pod snap2-restore-pod to be READY
[2024-10-21 10:56:58]  INFO Provisioner POLLED GetCapacity for powermax-iscsi storage class
[2024-10-21 10:56:58]  INFO Waiting for pod capacity-tracking-pod-147e3e35 to be READY
[2024-10-21 10:56:58]  INFO capacity-tracking-pod-147e3e35 pod is RUNNING
[2024-10-21 10:56:58]  INFO Deleting all resources in namespace capacity-tracking-test-f4e8cb81
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 13.1768 s, 10.2 MB/s
[2024-10-21 10:57:01]  INFO Executing command: [sha512sum /tmp/blob.data]
[2024-10-21 10:57:02]  INFO Pod: iowriter-test-6dcfp
[2024-10-21 10:57:02]  INFO hash sum is:d8cfbfbcfc0fe8bc83ecbe34026c011bf13e5bd1af8e99942d712b327e8aa748f9f047fe85f5f9bd1dff18043348807b885f69a8b99df2dc88bde1c06051d2b8  /tmp/blob.data

[2024-10-21 10:57:02]  INFO Hashes match
[2024-10-21 10:57:02]  INFO Executing command: [blockdev --flushbufs /dev/data0]
[2024-10-21 10:57:02]  INFO Executing command: [dd if=/dev/data0 of=/tmp/blob.data bs=1M count=128]
[2024-10-21 10:57:10]  INFO Namespace capacity-tracking-test-f4e8cb81 was deleted in 12.037343728s
[2024-10-21 10:57:10]  INFO Waiting for volumeattachments to be deleted
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 13.1259 s, 10.2 MB/s
[2024-10-21 10:57:15]  INFO Executing command: [sha512sum /tmp/blob.data]
[2024-10-21 10:57:16]  INFO Pod: iowriter-test-c7htx
[2024-10-21 10:57:16]  INFO hash sum is:d8cfbfbcfc0fe8bc83ecbe34026c011bf13e5bd1af8e99942d712b327e8aa748f9f047fe85f5f9bd1dff18043348807b885f69a8b99df2dc88bde1c06051d2b8  /tmp/blob.data

[2024-10-21 10:57:16]  INFO Hashes match
[2024-10-21 10:57:16]  INFO Executing command: [blockdev --flushbufs /dev/data0]
[2024-10-21 10:57:16]  INFO Executing command: [dd if=/dev/data0 of=/tmp/blob.data bs=1M count=128]
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 13.2136 s, 10.2 MB/s
[2024-10-21 10:57:29]  INFO Executing command: [sha512sum /tmp/blob.data]
[2024-10-21 10:57:30]  INFO Pod: iowriter-test-vj29t
[2024-10-21 10:57:30]  INFO hash sum is:d8cfbfbcfc0fe8bc83ecbe34026c011bf13e5bd1af8e99942d712b327e8aa748f9f047fe85f5f9bd1dff18043348807b885f69a8b99df2dc88bde1c06051d2b8  /tmp/blob.data

[2024-10-21 10:57:30]  INFO Hashes match
[2024-10-21 10:57:30]  INFO Deleting all resources in namespace mas-test-ae482977
[2024-10-21 10:57:31]  INFO SUCCESS: CapacityTrackingSuite in 4m19.030847676s
[2024-10-21 10:57:36]  INFO Checker pod: snap2-restore-pod
[2024-10-21 10:57:36]  INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-0.sha512]
[2024-10-21 10:57:40]  INFO All pods are ready in 2m4.603174116s
[2024-10-21 10:57:40]  INFO The replication group name from pvc is  
[2024-10-21 10:57:40]  INFO Deleting all resources in namespace replication-suite-13e60029
[2024-10-21 10:57:42]  INFO Namespace mas-test-ae482977 was deleted in 12.024213547s
[2024-10-21 10:57:42]  INFO Waiting for volumeattachments to be deleted
[2024-10-21 10:57:56]  INFO Hashes match
[2024-10-21 10:57:56]  INFO Deleting all resources in namespace snap-test-f3ed315a
[2024-10-21 10:58:02]  INFO SUCCESS: MultiAttachSuite in 4m50.061237572s
[2024-10-21 10:58:10]  INFO Namespace replication-suite-13e60029 was deleted in 30.023425934s
[2024-10-21 10:58:10]  INFO Deletion callback function is not empty, calling it
[2024-10-21 10:58:10]  INFO Deleting local RG
[2024-10-21 10:58:10]  WARN error when deleting local RG: no matches for kind "DellCSIReplicationGroup" in version "replication.storage.dell.com/v1"
[2024-10-21 10:58:10]  INFO Sleeping for 1 minute...
[2024-10-21 10:58:14]  INFO Namespace snap-test-f3ed315a was deleted in 18.025881257s
[2024-10-21 10:58:14]  INFO Waiting for volumeattachments to be deleted
[2024-10-21 10:58:29]  INFO SUCCESS: SnapSuite in 5m16.562793308s
[2024-10-21 10:59:13]  INFO SUCCESS: ReplicationSuite in 6m1.063632844s
[2024-10-21 10:59:13]  INFO Started generating reports...
Collecting metrics
9 / 9 [------------------------------------------------------------------------------] 100.00% ? p/s[2024-10-21 10:59:13]  INFO Started generating reports...
Collecting metrics
Generating plots
9 / 9 [------------------------------------------------------------------------------] 100.00% ? p/s1 / 9 [-------->______________________________________________________________________] 11.11% ? p/s2 / 9 [----------------->_____________________________________________________________] 22.22% ? p/s3 / 9 [-------------------------->____________________________________________________] 33.33% ? p/s4 / 9 [----------------------------------->___________________________________________] 44.44% 5 p/s5 / 9 [------------------------------------------->___________________________________] 55.56% 5 p/s5 / 9 [------------------------------------------->___________________________________] 55.56% 5 p/s6 / 9 [---------------------------------------------------->__________________________] 66.67% 5 p/s7 / 9 [------------------------------------------------------------->_________________] 77.78% 5 p/s8 / 9 [---------------------------------------------------------------------->________] 88.89% 5 p/s9 / 9 [----------------------------------------------------------------------------->] 100.00% 5 p/s9 / 9 [------------------------------------------------------------------------------] 100.00% 5 p/s[2024-10-21 10:59:15]  WARN No ResourceUsageMetrics provided
[2024-10-21 10:59:15] ERROR no ResourceUsageMetrics provided
report-test-run-5e1fd5ae:
Name: test-run-5e1fd5ae
Host: https://api.ocppmx.hop.lab.emc.com:6443
StorageClass: powermax-iscsi
Minimum and Maximum EntityOverTime charts:

/root/.cert-csi/reports/test-run-5e1fd5ae/PodsCreatingOverTime.png

/root/.cert-csi/reports/test-run-5e1fd5ae/PodsReadyOverTime.png

/root/.cert-csi/reports/test-run-5e1fd5ae/PodsTerminatingOverTime.png

/root/.cert-csi/reports/test-run-5e1fd5ae/PvcsCreatingOverTime.png

/root/.cert-csi/reports/test-run-5e1fd5ae/PvcsBoundOverTime.png

Tests:
--------------------------------------------------------------
1. TestCase: CloneVolumeSuite
            Started:   2024-10-21 10:53:12.437860727 -0500 -0500
            Ended:     2024-10-21 10:55:34.699806658 -0500 -0500
            Result:    SUCCESS

            Stage metrics:
		    PVCAttachment:
			Avg: 12.246878882s
			Min: 3.234692963s
			Max: 19.395593222s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PVCAttachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PVCAttachment_boxplot.png
		    PVCBind:
			Avg: 12.105834786s
			Min: 8.528608821s
			Max: 18.009270645s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PVCBind.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PVCBind_boxplot.png
		    PVCCreation:
			Avg: 24.810193974s
			Min: 12.205738829s
			Max: 37.785393688s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PVCCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PVCCreation_boxplot.png
		    PVCDeletion:
			Avg: 1.957056877s
			Min: 1.5457598s
			Max: 2.561982967s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PVCDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PVCDeletion_boxplot.png
		    PVCUnattachment:
			Avg: 12.487807836s
			Min: 8.522915263s
			Max: 16.210571085s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PVCUnattachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PVCUnattachment_boxplot.png
		    PodCreation:
			Avg: 43.908565758s
			Min: 32.254612508s
			Max: 53.934148933s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PodCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PodCreation_boxplot.png
		    PodDeletion:
			Avg: 1.772824s
			Min: 1.376145516s
			Max: 2.366049251s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PodDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/PodDeletion_boxplot.png
			EntityNumberOverTime:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CloneVolumeSuite1/EntityNumberOverTime.png
--------------------------------------------------------------
2. TestCase: VolumeIoSuite
            Started:   2024-10-21 10:53:12.437845871 -0500 -0500
            Ended:     2024-10-21 10:56:08.743230144 -0500 -0500
            Result:    SUCCESS

            Stage metrics:
		    PVCAttachment:
			Avg: 30.206373921s
			Min: 21.466490278s
			Max: 38.946257565s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PVCAttachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PVCAttachment_boxplot.png
		    PVCBind:
			Avg: 20.418969927s
			Min: 17.787826243s
			Max: 23.050113611s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PVCBind.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PVCBind_boxplot.png
		    PVCCreation:
			Avg: 1m20.846246109s
			Min: 39.806342122s
			Max: 2m1.886150096s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PVCCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PVCCreation_boxplot.png
		    PVCDeletion:
			Avg: 25.457596ms
			Min: 18.81261ms
			Max: 32.102582ms
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PVCDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PVCDeletion_boxplot.png
		    PVCUnattachment:
			Avg: 13.904956296s
			Min: 12.492916466s
			Max: 15.316996127s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PVCUnattachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PVCUnattachment_boxplot.png
		    PodCreation:
			Avg: 42.791218234s
			Min: 9.044438801s
			Max: 1m4.125209338s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PodCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PodCreation_boxplot.png
		    PodDeletion:
			Avg: 4.046560525s
			Min: 1.997916753s
			Max: 6.599648272s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PodDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/PodDeletion_boxplot.png
			EntityNumberOverTime:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeIoSuite2/EntityNumberOverTime.png
--------------------------------------------------------------
3. TestCase: VolumeExpansionSuite
            Started:   2024-10-21 10:53:12.437834347 -0500 -0500
            Ended:     2024-10-21 10:55:56.690946065 -0500 -0500
            Result:    SUCCESS

            Stage metrics:
		    PVCAttachment:
			Avg: 24.794457548s
			Min: 24.794457548s
			Max: 24.794457548s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PVCAttachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PVCAttachment_boxplot.png
		    PVCBind:
			Avg: 15.824572756s
			Min: 15.824572756s
			Max: 15.824572756s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PVCBind.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PVCBind_boxplot.png
		    PVCCreation:
			Avg: 41.635348872s
			Min: 41.635348872s
			Max: 41.635348872s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PVCCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PVCCreation_boxplot.png
		    PVCDeletion:
			Avg: 1.260665107s
			Min: 1.260665107s
			Max: 1.260665107s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PVCDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PVCDeletion_boxplot.png
		    PVCUnattachment:
			Avg: 10.131671119s
			Min: 10.131671119s
			Max: 10.131671119s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PVCUnattachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PVCUnattachment_boxplot.png
		    PodCreation:
			Avg: 53.05098023s
			Min: 53.05098023s
			Max: 53.05098023s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PodCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PodCreation_boxplot.png
		    PodDeletion:
			Avg: 1.363018843s
			Min: 1.363018843s
			Max: 1.363018843s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PodDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/PodDeletion_boxplot.png
			EntityNumberOverTime:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite3/EntityNumberOverTime.png
--------------------------------------------------------------
4. TestCase: ReplicationSuite
            Started:   2024-10-21 10:53:12.437880799 -0500 -0500
            Ended:     2024-10-21 10:59:13.516603628 -0500 -0500
            Result:    SUCCESS

            Stage metrics:
		    PVCAttachment:
			Avg: 29.514830639s
			Min: 9.57933264s
			Max: 1m13.402087189s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PVCAttachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PVCAttachment_boxplot.png
		    PVCBind:
			Avg: 22.153256671s
			Min: 11.560902288s
			Max: 38.96477127s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PVCBind.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PVCBind_boxplot.png
		    PVCCreation:
			Avg: 55.02659335s
			Min: 27.09473718s
			Max: 1m53.038650148s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PVCCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PVCCreation_boxplot.png
		    PVCDeletion:
			Avg: 1.623475257s
			Min: 1.411246833s
			Max: 1.864430545s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PVCDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PVCDeletion_boxplot.png
		    PVCUnattachment:
			Avg: 9.195645057s
			Min: 1.922141017s
			Max: 26.972388114s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PVCUnattachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PVCUnattachment_boxplot.png
		    PodCreation:
			Avg: 1m7.506799329s
			Min: 42.427238063s
			Max: 2m2.841156235s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PodCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PodCreation_boxplot.png
		    PodDeletion:
			Avg: 1.662337123s
			Min: 1.574382139s
			Max: 1.891870875s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PodDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/PodDeletion_boxplot.png
			EntityNumberOverTime:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ReplicationSuite4/EntityNumberOverTime.png
--------------------------------------------------------------
5. TestCase: MultiAttachSuite
            Started:   2024-10-21 10:53:12.437869656 -0500 -0500
            Ended:     2024-10-21 10:58:02.517903746 -0500 -0500
            Result:    SUCCESS

            Stage metrics:
		    PVCAttachment:
			Avg: 53.321668552s
			Min: 53.321668552s
			Max: 53.321668552s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PVCAttachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PVCAttachment_boxplot.png
		    PVCBind:
			Avg: 18.503864039s
			Min: 18.503864039s
			Max: 18.503864039s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PVCBind.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PVCBind_boxplot.png
		    PVCCreation:
			Avg: 2m9.394190805s
			Min: 2m9.394190805s
			Max: 2m9.394190805s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PVCCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PVCCreation_boxplot.png
		    PVCDeletion:
			Avg: 1.555474826s
			Min: 1.555474826s
			Max: 1.555474826s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PVCDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PVCDeletion_boxplot.png
		    PVCUnattachment:
			Avg: 14.238203893s
			Min: 14.238203893s
			Max: 14.238203893s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PVCUnattachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PVCUnattachment_boxplot.png
		    PodCreation:
			Avg: 37.725298133s
			Min: 1.714077078s
			Max: 1m18.783841019s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PodCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PodCreation_boxplot.png
		    PodDeletion:
			Avg: 1.432710431s
			Min: 1.397918649s
			Max: 1.515882873s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PodDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/PodDeletion_boxplot.png
			EntityNumberOverTime:
	/root/.cert-csi/reports/test-run-5e1fd5ae/MultiAttachSuite5/EntityNumberOverTime.png
--------------------------------------------------------------
6. TestCase: SnapSuite
            Started:   2024-10-21 10:53:12.437849201 -0500 -0500
            Ended:     2024-10-21 10:58:29.024515211 -0500 -0500
            Result:    SUCCESS

            Stage metrics:
		    PVCAttachment:
			Avg: 19.089491711s
			Min: 17.215095771s
			Max: 20.963887651s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PVCAttachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PVCAttachment_boxplot.png
		    PVCBind:
			Avg: 18.529667095s
			Min: 17.859021116s
			Max: 19.200313074s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PVCBind.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PVCBind_boxplot.png
		    PVCCreation:
			Avg: 38.58378442s
			Min: 37.456150742s
			Max: 39.711418099s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PVCCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PVCCreation_boxplot.png
		    PVCDeletion:
			Avg: 467.708438ms
			Min: 18.404563ms
			Max: 917.012314ms
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PVCDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PVCDeletion_boxplot.png
		    PVCUnattachment:
			Avg: 16.512782373s
			Min: 11.063733297s
			Max: 21.96183145s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PVCUnattachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PVCUnattachment_boxplot.png
		    PodCreation:
			Avg: 49.245505213s
			Min: 47.369656539s
			Max: 51.121353887s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PodCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PodCreation_boxplot.png
		    PodDeletion:
			Avg: 1.473616004s
			Min: 1.469399123s
			Max: 1.477832886s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PodDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/PodDeletion_boxplot.png
			EntityNumberOverTime:
	/root/.cert-csi/reports/test-run-5e1fd5ae/SnapSuite6/EntityNumberOverTime.png
--------------------------------------------------------------
7. TestCase: ScalingSuite
            Started:   2024-10-21 10:53:12.437851676 -0500 -0500
            Ended:     2024-10-21 10:55:34.817310523 -0500 -0500
            Result:    SUCCESS

            Stage metrics:
		    PVCAttachment:
			Avg: 12.636501113s
			Min: 5.846196945s
			Max: 19.348549655s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PVCAttachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PVCAttachment_boxplot.png
		    PVCBind:
			Avg: 7.877441381s
			Min: 6.320942238s
			Max: 11.733203758s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PVCBind.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PVCBind_boxplot.png
		    PVCCreation:
			Avg: 24.23433386s
			Min: 16.968003953s
			Max: 31.444586599s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PVCCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PVCCreation_boxplot.png
		    PVCDeletion:
			Avg: 64.736944ms
			Min: 18.681535ms
			Max: 120.347237ms
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PVCDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PVCDeletion_boxplot.png
		    PVCUnattachment:
			Avg: 18.565938867s
			Min: 14.643401604s
			Max: 22.476088529s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PVCUnattachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PVCUnattachment_boxplot.png
		    PodCreation:
			Avg: 44.469315335s
			Min: 33.674732501s
			Max: 55.26389817s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PodCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PodCreation_boxplot.png
		    PodDeletion:
			Avg: 1.338730828s
			Min: 1.142545591s
			Max: 1.534916066s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PodDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/PodDeletion_boxplot.png
			EntityNumberOverTime:
	/root/.cert-csi/reports/test-run-5e1fd5ae/ScalingSuite7/EntityNumberOverTime.png
--------------------------------------------------------------
8. TestCase: CapacityTrackingSuite
            Started:   2024-10-21 10:53:12.437787105 -0500 -0500
            Ended:     2024-10-21 10:57:31.508748349 -0500 -0500
            Result:    SUCCESS

            Stage metrics:
		    PVCAttachment:
			Avg: 17.572105701s
			Min: 17.572105701s
			Max: 17.572105701s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PVCAttachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PVCAttachment_boxplot.png
		    PVCBind:
			Avg: 20.794505766s
			Min: 20.794505766s
			Max: 20.794505766s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PVCBind.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PVCBind_boxplot.png
		    PVCCreation:
			Avg: 39.191599495s
			Min: 39.191599495s
			Max: 39.191599495s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PVCCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PVCCreation_boxplot.png
		    PVCDeletion:
			Avg: 2.035165354s
			Min: 2.035165354s
			Max: 2.035165354s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PVCDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PVCDeletion_boxplot.png
		    PVCUnattachment:
			Avg: 13.403728671s
			Min: 13.403728671s
			Max: 13.403728671s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PVCUnattachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PVCUnattachment_boxplot.png
		    PodCreation:
			Avg: 46.869612728s
			Min: 46.869612728s
			Max: 46.869612728s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PodCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PodCreation_boxplot.png
		    PodDeletion:
			Avg: 1.562047479s
			Min: 1.562047479s
			Max: 1.562047479s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PodDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/PodDeletion_boxplot.png
			EntityNumberOverTime:
	/root/.cert-csi/reports/test-run-5e1fd5ae/CapacityTrackingSuite8/EntityNumberOverTime.png
--------------------------------------------------------------
9. TestCase: VolumeExpansionSuite
            Started:   2024-10-21 10:53:12.437859873 -0500 -0500
            Ended:     2024-10-21 10:55:25.478124143 -0500 -0500
            Result:    SUCCESS

            Stage metrics:
		    PVCAttachment:
			Avg: 42.423362932s
			Min: 42.423362932s
			Max: 42.423362932s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PVCAttachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PVCAttachment_boxplot.png
		    PVCBind:
			Avg: 16.481891197s
			Min: 16.481891197s
			Max: 16.481891197s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PVCBind.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PVCBind_boxplot.png
		    PVCCreation:
			Avg: 59.877808806s
			Min: 59.877808806s
			Max: 59.877808806s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PVCCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PVCCreation_boxplot.png
		    PVCDeletion:
			Avg: 1.278939004s
			Min: 1.278939004s
			Max: 1.278939004s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PVCDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PVCDeletion_boxplot.png
		    PVCUnattachment:
			Avg: 30.801769575s
			Min: 30.801769575s
			Max: 30.801769575s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PVCUnattachment.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PVCUnattachment_boxplot.png
		    PodCreation:
			Avg: 1m4.818506811s
			Min: 1m4.818506811s
			Max: 1m4.818506811s
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PodCreation.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PodCreation_boxplot.png
		    PodDeletion:
			Avg: 902.095503ms
			Min: 902.095503ms
			Max: 902.095503ms
			Histogram:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PodDeletion.png
			BoxPlot:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/PodDeletion_boxplot.png
			EntityNumberOverTime:
	/root/.cert-csi/reports/test-run-5e1fd5ae/VolumeExpansionSuite9/EntityNumberOverTime.png

[2024-10-21 10:59:16]  INFO Avg time of a run:	 179.10s
[2024-10-21 10:59:16]  INFO Avg time of a del:	 23.40s
[2024-10-21 10:59:16]  INFO Avg time of all:	 220.54s
[2024-10-21 10:59:16]  INFO During this run 100.0% of suites succeeded
Success: Cert-CSI suite completed successfully.