[root@master-1-90cifA29Om2e7 e2e]# ./run-e2e-test.sh --minimal --pflex --obs /root/e2e/csm-operator/tests/e2e W1014 11:21:25.458558 2964002 test_context.go:538] Unable to find in-cluster config, using default host : https://127.0.0.1:6443 I1014 11:21:25.458877 2964002 test_context.go:561] The --provider flag is not set. Continuing as if --provider=skeleton had been used. Running Suite: CSM Operator End-to-End Tests - /root/e2e/csm-operator/tests/e2e =============================================================================== Random Seed: 1728922878 Will run 1 of 1 specs ------------------------------ [BeforeSuite] /root/e2e/csm-operator/tests/e2e/e2e_test.go:99 STEP: Getting test environment variables @ 10/14/24 11:21:25.466 STEP: [observability powerflex] @ 10/14/24 11:21:25.466 STEP: Reading values file @ 10/14/24 11:21:25.466 STEP: Getting a k8s client @ 10/14/24 11:21:25.468 [BeforeSuite] PASSED [0.006 seconds] ------------------------------ [run-e2e-test] E2E Testing Running all test Given Test Scenarios /root/e2e/csm-operator/tests/e2e/e2e_test.go:140 STEP: Starting: Install PowerFlex Driver(With Observability) @ 10/14/24 11:21:25.471 STEP: Returning false here @ 10/14/24 11:21:25.477 STEP: Executing Given an environment with k8s or openshift, and CSM operator installed @ 10/14/24 11:21:25.477 STEP: Executing Create storageclass with name [op-e2e-vxflexos] and template [testfiles/powerflex-templates/powerflex-storageclass-template.yaml] for [pflex] @ 10/14/24 11:21:25.49 STEP: Executing Set up secret with template [testfiles/powerflex-templates/powerflex-secret-template.yaml] name [test-vxflexos-config] in namespace [test-vxflexos] for [pflex] @ 10/14/24 11:21:26.18 STEP: Executing Apply custom resource [1] @ 10/14/24 11:21:26.61 I1014 11:21:26.610595 2964002 builder.go:121] Running '/usr/bin/kubectl --namespace=test-vxflexos apply --validate=true -f -' I1014 11:21:26.755111 2964002 builder.go:146] stderr: "" I1014 11:21:26.755174 2964002 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/test-vxflexos created\n" STEP: Executing Validate [powerflex] driver from CR [1] is installed @ 10/14/24 11:21:26.755 err: The pod(test-vxflexos-node-7zrrh) is Pending STEP: Executing Validate [observability] module from CR [1] is installed @ 10/14/24 11:22:16.796 err: failed to check for observability installation in default-source-cluster: The pod(karavi-metrics-powerflex-848bb7b58b-qvr7x) is Pending The pod(karavi-topology-69569f78bb-s6jzs) is Pending The pod(otel-collector-6f94bf5456-7c8zs) is Pending err: failed to check for observability installation in default-source-cluster: The pod(karavi-metrics-powerflex-848bb7b58b-qvr7x) is Pending The pod(karavi-topology-69569f78bb-s6jzs) is Pending The pod(otel-collector-6f94bf5456-7c8zs) is Pending err: failed to check for observability installation in default-source-cluster: The pod(karavi-metrics-powerflex-848bb7b58b-qvr7x) is Pending The pod(karavi-topology-69569f78bb-s6jzs) is Pending The pod(otel-collector-6f94bf5456-7c8zs) is Pending err: failed to check for observability installation in default-source-cluster: The pod(karavi-metrics-powerflex-848bb7b58b-qvr7x) is Pending The pod(karavi-topology-69569f78bb-s6jzs) is Pending The pod(otel-collector-6f94bf5456-7c8zs) is Pending err: failed to check for observability installation in default-source-cluster: The pod(karavi-metrics-powerflex-848bb7b58b-qvr7x) is Pending STEP: Executing Run custom test @ 10/14/24 11:24:06.891 I1014 11:24:06.891814 2964002 util.go:650] Running cert-csi [test vio --sc op-e2e-vxflexos --chainNumber 2 --chainLength 2] [2024-10-14 11:24:06] INFO Starting cert-csi; ver. 1.5.0 [2024-10-14 11:24:06] INFO Using EVENT observer type [2024-10-14 11:24:06] INFO Using config from /etc/kubernetes/admin.conf [2024-10-14 11:24:06] INFO Successfully loaded config. Host: https://10.247.72.160:6443 [2024-10-14 11:24:06] INFO Created new KubeClient [2024-10-14 11:24:06] INFO Running 1 iteration(s) [2024-10-14 11:24:06] INFO *** ITERATION NUMBER 1 *** [2024-10-14 11:24:06] INFO Starting VolumeIoSuite with op-e2e-vxflexos storage class [2024-10-14 11:24:06] INFO Successfully created namespace volumeio-test-3038c3a2 [2024-10-14 11:24:06] INFO Using default number of volumes [2024-10-14 11:24:06] INFO Using default image: quay.io/centos/centos:latest [2024-10-14 11:24:06] INFO Creating IO pod [2024-10-14 11:24:06] INFO Waiting for pod iowriter-test-kzm6l to be READY [2024-10-14 11:24:06] INFO Waiting for pod iowriter-test-xv2t6 to be READY [2024-10-14 11:24:24] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data] [2024-10-14 11:24:25] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-1.data] [2024-10-14 11:24:25] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512] [2024-10-14 11:24:25] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-1.data > /data0/writer-1.sha512] [2024-10-14 11:24:32] INFO Waiting until no Volume Attachments with PV left [2024-10-14 11:24:32] INFO VolumeAttachment deleted [2024-10-14 11:24:32] INFO Waiting for pod iowriter-test-7674k to be READY [2024-10-14 11:24:32] INFO Waiting until no Volume Attachments with PV left [2024-10-14 11:24:32] INFO VolumeAttachment deleted [2024-10-14 11:24:32] INFO Waiting for pod iowriter-test-7j4c9 to be READY [2024-10-14 11:24:40] INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-0.sha512] [2024-10-14 11:24:40] INFO Executing command: [/bin/bash -c sha512sum -c /data0/writer-1.sha512] [2024-10-14 11:24:43] INFO Hashes match [2024-10-14 11:24:43] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-0.data] [2024-10-14 11:24:43] INFO Hashes match [2024-10-14 11:24:43] INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-1.data] [2024-10-14 11:24:44] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512] [2024-10-14 11:24:44] INFO Executing command: [/bin/bash -c sha512sum /data0/writer-1.data > /data0/writer-1.sha512] [2024-10-14 11:24:48] INFO Waiting until no Volume Attachments with PV left [2024-10-14 11:24:48] INFO VolumeAttachment deleted [2024-10-14 11:24:49] INFO Waiting until no Volume Attachments with PV left [2024-10-14 11:24:49] INFO VolumeAttachment deleted [2024-10-14 11:24:49] INFO Deleting all resources in namespace volumeio-test-3038c3a2 [2024-10-14 11:25:01] INFO Namespace volumeio-test-3038c3a2 was deleted in 12.011481478s [2024-10-14 11:25:02] INFO SUCCESS: VolumeIoSuite in 55.083392888s Collecting metrics [2024-10-14 11:25:02] INFO Started generating reports... 1 / 1 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s[2024-10-14 11:25:02] INFO Started generating reports... Collecting metrics Generating plots 1 / 1 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s1 / 1 [------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->] 100.00% ? p/s1 / 1 [-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 53 p/s[2024-10-14 11:25:02] WARN No ResourceUsageMetrics provided [2024-10-14 11:25:02] ERROR no ResourceUsageMetrics provided report-test-run-2a084aa3: Name: test-run-2a084aa3 Host: https://10.247.72.160:6443 StorageClass: op-e2e-vxflexos Minimum and Maximum EntityOverTime charts: /root/.cert-csi/reports/test-run-2a084aa3/PodsCreatingOverTime.png /root/.cert-csi/reports/test-run-2a084aa3/PodsReadyOverTime.png /root/.cert-csi/reports/test-run-2a084aa3/PodsTerminatingOverTime.png /root/.cert-csi/reports/test-run-2a084aa3/PvcsCreatingOverTime.png /root/.cert-csi/reports/test-run-2a084aa3/PvcsBoundOverTime.png Tests: -------------------------------------------------------------- 1. TestCase: VolumeIoSuite Started: 2024-10-14 11:24:06.946146115 -0500 -0500 Ended: 2024-10-14 11:25:02.027319741 -0500 -0500 Result: SUCCESS Stage metrics: PVCAttachment: Avg: 155.843027ms Min: 142.32437ms Max: 169.361684ms Histogram: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PVCAttachment.png BoxPlot: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PVCAttachment_boxplot.png PVCBind: Avg: 337.125261ms Min: 329.428958ms Max: 344.821565ms Histogram: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PVCBind.png BoxPlot: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PVCBind_boxplot.png PVCCreation: Avg: 1.263656042s Min: 1.246251548s Max: 1.281060536s Histogram: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PVCCreation.png BoxPlot: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PVCCreation_boxplot.png PVCDeletion: Avg: 20.426125ms Min: 14.352886ms Max: 26.499364ms Histogram: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PVCDeletion.png BoxPlot: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PVCDeletion_boxplot.png PVCUnattachment: Avg: 89.602554ms Min: 86.637604ms Max: 92.567505ms Histogram: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PVCUnattachment.png BoxPlot: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PVCUnattachment_boxplot.png PodCreation: Avg: 11.751264524s Min: 6.984396445s Max: 16.425297302s Histogram: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PodCreation.png BoxPlot: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PodCreation_boxplot.png PodDeletion: Avg: 4.380756098s Min: 3.579006464s Max: 5.672576648s Histogram: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PodDeletion.png BoxPlot: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/PodDeletion_boxplot.png EntityNumberOverTime: /root/.cert-csi/reports/test-run-2a084aa3/VolumeIoSuite19/EntityNumberOverTime.png [2024-10-14 11:25:02] INFO Avg time of a run: 42.25s [2024-10-14 11:25:02] INFO Avg time of a del: 12.01s [2024-10-14 11:25:02] INFO Avg time of all: 55.08s [2024-10-14 11:25:02] INFO During this run 100.0% of suites succeeded STEP: Executing Enable forceRemoveDriver on CR [1] @ 10/14/24 11:25:02.548 STEP: Executing Delete custom resource [1] @ 10/14/24 11:25:02.558 STEP: Executing Restore template [testfiles/powerflex-templates/powerflex-secret-template.yaml] for [pflex] @ 10/14/24 11:25:02.572 STEP: Executing Restore template [testfiles/powerflex-templates/powerflex-storageclass-template.yaml] for [pflex] @ 10/14/24 11:25:02.696 STEP: Ending: Install PowerFlex Driver(With Observability) @ 10/14/24 11:25:02.816 STEP: Starting: Install PowerFlex Driver(With Observability and Custom Cert for Otel-Collector) @ 10/14/24 11:25:07.82 STEP: Returning false here @ 10/14/24 11:25:07.82 STEP: Executing Given an environment with k8s or openshift, and CSM operator installed @ 10/14/24 11:25:07.82 STEP: Executing Create storageclass with name [op-e2e-vxflexos] and template [testfiles/powerflex-templates/powerflex-storageclass-template.yaml] for [pflex] @ 10/14/24 11:25:07.824 STEP: Executing Set up secret with template [testfiles/powerflex-templates/powerflex-secret-template.yaml] name [test-vxflexos-config] in namespace [test-vxflexos] for [pflex] @ 10/14/24 11:25:08.208 STEP: Executing Apply custom resource [1] @ 10/14/24 11:25:08.552 I1014 11:25:08.553101 2964002 builder.go:121] Running '/usr/bin/kubectl --namespace=test-vxflexos apply --validate=true -f -' I1014 11:25:08.660962 2964002 builder.go:146] stderr: "" I1014 11:25:08.661018 2964002 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/test-vxflexos created\n" STEP: Executing Validate [powerflex] driver from CR [1] is installed @ 10/14/24 11:25:08.661 err: The pod(test-vxflexos-node-qx8bc) is Pending STEP: Executing Validate [observability] module from CR [1] is installed @ 10/14/24 11:25:58.7 STEP: Executing Enable forceRemoveDriver on CR [1] @ 10/14/24 11:26:08.723 STEP: Executing Delete custom resource [1] @ 10/14/24 11:26:08.739 STEP: Executing Restore template [testfiles/powerflex-templates/powerflex-secret-template.yaml] for [pflex] @ 10/14/24 11:26:08.752 STEP: Executing Restore template [testfiles/powerflex-templates/powerflex-storageclass-template.yaml] for [pflex] @ 10/14/24 11:26:08.879 STEP: Ending: Install PowerFlex Driver(With Observability and Custom Cert for Otel-Collector) @ 10/14/24 11:26:09.036 • [288.566 seconds] ------------------------------ Ran 1 of 1 Specs in 288.572 seconds SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped PASS Ginkgo ran 1 suite in 4m55.188313867s Test Suite Passed [root@master-1-90cifA29Om2e7 e2e]#