operator: image bundle < 1000 references test #2549

sebrandon1 · 2024-11-06T21:30:05Z

Adds a test for operator image bundles. They must have a bundleCount less than 1000 images. This is new from the v1.6 document.

dcibot · 2024-11-06T21:56:47Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/416f0f84-0cfd-4775-8775-a371add71b85/jobStates

ramperher

I guess you need to update CATALOG.md too, right?

dcibot · 2024-11-07T10:35:22Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/ee8ca2a0-7bd4-4e95-91d2-d058bff2568d/jobStates

ramperher · 2024-11-07T11:12:35Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/ee8ca2a0-7bd4-4e95-91d2-d058bff2568d/jobStates

I tried this new test in our operator-based workload but it's skipped, it complains that no catalog source is found:

[DEBUG] [Nov  7 09:59:55.593] [checksgroup.go: 158] GROUP operator - Running beforeEach for check operator-operator-bundle-count
[INFO] [Nov  7 09:59:55.593] [checksgroup.go: 83] [operator-operator-bundle-count] Skipping check operator-operator-bundle-count, reason: no catalog sources found
[DEBUG] [Nov  7 09:59:55.593] [checksgroup.go: 182] GROUP operator - Running afterEach for check operator-operator-bundle-count

However, all the other operator tests are passing:

[INFO] [Nov  7 09:59:55.593] [checksgroup.go: 412] Recording checks results of group operator
[INFO] [Nov  7 09:59:55.593] [checksdb.go: 115] [operator-install-status-succeeded] Recording result "PASSED", claimID: {Id:operator-install-status-succeeded Suite:operator Tags:common}
[INFO] [Nov  7 09:59:55.593] [checksdb.go: 115] [operator-install-status-no-privileges] Recording result "FAILED", claimID: {Id:operator-install-status-no-privileges Suite:operator Tags:common}
[INFO] [Nov  7 09:59:55.593] [checksdb.go: 115] [operator-install-source] Recording result "PASSED", claimID: {Id:operator-install-source Suite:operator Tags:common}
[INFO] [Nov  7 09:59:55.593] [checksdb.go: 115] [operator-semantic-versioning] Recording result "PASSED", claimID: {Id:operator-semantic-versioning Suite:operator Tags:common}
[INFO] [Nov  7 09:59:55.593] [checksdb.go: 115] [operator-crd-versioning] Recording result "PASSED", claimID: {Id:operator-crd-versioning Suite:operator Tags:common}
[INFO] [Nov  7 09:59:55.593] [checksdb.go: 115] [operator-crd-openapi-schema] Recording result "PASSED", claimID: {Id:operator-crd-openapi-schema Suite:operator Tags:common}
[INFO] [Nov  7 09:59:55.593] [checksdb.go: 115] [operator-single-crd-owner] Recording result "PASSED", claimID: {Id:operator-single-crd-owner Suite:operator Tags:common}
[INFO] [Nov  7 09:59:55.593] [checksdb.go: 115] [operator-pods-no-hugepages] Recording result "PASSED", claimID: {Id:operator-pods-no-hugepages Suite:operator Tags:common}
[INFO] [Nov  7 09:59:55.593] [checksdb.go: 115] [operator-olm-skip-range] Recording result "PASSED", claimID: {Id:operator-olm-skip-range Suite:operator Tags:common}
[INFO] [Nov  7 09:59:55.593] [checksdb.go: 115] [operator-multiple-same-operators] Recording result "PASSED", claimID: {Id:operator-multiple-same-operators Suite:operator Tags:common}
[INFO] [Nov  7 09:59:55.593] [checksdb.go: 115] [operator-operator-bundle-count] Recording result "SKIPPED", claimID: {Id:operator-operator-bundle-count Suite:operator Tags:common}

In our case, we create a CatalogSource called nfv-example-cnf-catalog in the default openshift-marketplace namespace. Is it the reason why it's not autodiscovered for this test (not using the workload namespace)?

sebrandon1 · 2024-11-07T15:15:12Z

At the moment, the requirement is to only test catalogSources that come from a namespace-under-test. Users don't have to label their catalog sources. Let me bring that up with the rest of the team to see if that would make sense to change it.

pkg/provider/catalogsources.go

dcibot · 2024-11-07T17:53:43Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/5b9e152a-6fc6-4092-ba5e-3a2c991cc95e/jobStates

dcibot · 2024-11-07T21:36:19Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/4241e2b3-e13d-47c1-87f4-d236c8de65df/jobStates

sebrandon1 · 2024-11-07T21:47:19Z

redhat-best-practices-for-k8s/certsuite-sample-workload#502

I added a sample catalogsource in the sample-workload repo that gets deployed in the CI so we can mark it as an expected passing test. I will also add a positive and negative test in QE.

dcibot · 2024-11-07T22:48:04Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/fccbc6f2-5510-4336-bd26-4afcd5fd25dd/jobStates

dcibot · 2024-11-08T00:50:58Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/b582e079-8ae8-4508-91af-8b54445f8513/jobStates

dcibot · 2024-11-08T01:16:24Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/93c288fd-2729-41d3-baf6-f44d000ef17b/jobStates

dcibot · 2024-11-08T13:37:19Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/697cf0c4-1f52-4190-b1aa-a885cf3661ef/jobStates

ramperher · 2024-11-08T13:40:44Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/697cf0c4-1f52-4190-b1aa-a885cf3661ef/jobStates

@sebrandon1, the operator-bundle-count test is still skipped. Please tell me if you need some info to troubleshoot, but in the DCI job, in Files section, you can take a look at the claim.json file, certsuite.log (however, no logs are provided to this test, it directly says that no CatalogSource was found) and so on.

dcibot · 2024-12-02T22:35:34Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/7fbacac7-5cd3-4d69-8a4b-f5676380bd06/jobStates

tests/operator/suite.go

dcibot · 2024-12-03T15:39:24Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/2ad7683d-1aaf-4221-9b69-5676116e263c/jobStates

sebrandon1 · 2024-12-03T21:09:12Z

Okay I was able to verify that the logic I added for performing the grpcurl on clusters <= 4.12 works on my QE branch. @ramperher can you verify my latest commit works on your 4.12 setup?

dcibot · 2024-12-03T23:45:34Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/9624aae4-b2ec-4120-8d11-672977b7fb8d/jobStates

ramperher · 2024-12-04T10:54:22Z

Okay I was able to verify that the logic I added for performing the grpcurl on clusters <= 4.12 works on my QE branch. @ramperher can you verify my latest commit works on your 4.12 setup?

I will schedule it when it's available, it will take some time but I'll tell you when it's ready

dcibot · 2024-12-04T14:50:05Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/dd305766-197a-4ba4-9182-d47b56de36d3/jobStates

ramperher · 2024-12-04T15:21:03Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/dd305766-197a-4ba4-9182-d47b56de36d3/jobStates

Tested with 4.12, and the test is skipped in this case, it's returning 0 to all executions. This needs to be reviewed.
I remember that, when testing grpcurl in the past, I needed to have it bavailable in the debug daemonset. @edcdavid created an image for that: #2549 (comment). My impression is that we need this available in the debug daemonset to make it work.

sebrandon1 · 2024-12-04T15:33:47Z

@ramperher Yup it is installed an available in v0.0.11 of the probe image. We are issuing the grpcurl command from the probe pod to the catalogsource service. I'll take a look at the execution logs.

EDIT: I see what's wrong with it. The SkipPMBasedOnChannel is skipping PM's based on channel.entries (something that doesn't exist in 4.12). So everything is just skipped even before it gets to the bundle count lookup. Let me think about how to get around that.....

Look through CSVs and PackageManifests Remove curl lookup; use relatedImages Add filter for already reported catalogsources additional cleanup; catalogsource pkg and tests total up channel entries for bundle count cleanup and rebuild catalog

dcibot · 2024-12-05T15:11:20Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/b41f3ebf-990b-40d9-b7d8-2d8f24639bdc/jobStates

dcibot · 2024-12-05T15:34:43Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/62a4031a-adfc-4b78-8495-6f05459d0e80/jobStates

ramperher · 2024-12-05T15:54:04Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/62a4031a-adfc-4b78-8495-6f05459d0e80/jobStates

Now it's passing in OCP 4.12, so LGTM

ramperher

LGTM

dcibot · 2024-12-05T17:35:30Z

from change #2549:

SUCCESS https://www.distributed-ci.io/jobs/357aec43-5f24-499c-a99e-c9aadaa08413/jobStates

sebrandon1 added the v1.6 BP Update label Nov 6, 2024

sebrandon1 self-assigned this Nov 6, 2024

sebrandon1 force-pushed the bundle_less_than_1000 branch from b5cac8e to 44b1993 Compare November 6, 2024 21:35

ramperher reviewed Nov 7, 2024

View reviewed changes

sebrandon1 added the work in progress label Nov 7, 2024

sebrandon1 force-pushed the bundle_less_than_1000 branch 2 times, most recently from 58619de to ee78a85 Compare November 7, 2024 17:22

sebrandon1 commented Nov 7, 2024

View reviewed changes

pkg/provider/catalogsources.go Outdated Show resolved Hide resolved

sebrandon1 force-pushed the bundle_less_than_1000 branch from ee78a85 to cad28de Compare November 7, 2024 17:32

sebrandon1 force-pushed the bundle_less_than_1000 branch from cad28de to 2bda85b Compare November 7, 2024 21:04

sebrandon1 removed the work in progress label Nov 7, 2024

sebrandon1 force-pushed the bundle_less_than_1000 branch 3 times, most recently from 85a95fc to ba205ea Compare November 7, 2024 22:14

sebrandon1 mentioned this pull request Nov 8, 2024

operator: add bundle-count test redhat-best-practices-for-k8s/certsuite-qe#983

Merged

sebrandon1 force-pushed the bundle_less_than_1000 branch from ba205ea to a35e0a4 Compare November 8, 2024 00:27

sebrandon1 force-pushed the bundle_less_than_1000 branch from a35e0a4 to e321320 Compare November 8, 2024 00:55

sebrandon1 force-pushed the bundle_less_than_1000 branch from e321320 to 7e57fde Compare November 11, 2024 14:59

sebrandon1 force-pushed the bundle_less_than_1000 branch 2 times, most recently from 3f9b195 to aa6e294 Compare December 2, 2024 22:10

greyerof reviewed Dec 3, 2024

View reviewed changes

tests/operator/suite.go Outdated Show resolved Hide resolved

sebrandon1 force-pushed the bundle_less_than_1000 branch from aa6e294 to d87d5ef Compare December 3, 2024 14:35

greyerof approved these changes Dec 3, 2024

View reviewed changes

sebrandon1 force-pushed the bundle_less_than_1000 branch from d87d5ef to 3175a0f Compare December 3, 2024 14:46

sebrandon1 force-pushed the bundle_less_than_1000 branch from d3df56b to 837bc3c Compare December 3, 2024 21:05

sebrandon1 removed the work in progress label Dec 3, 2024

sebrandon1 force-pushed the bundle_less_than_1000 branch from 837bc3c to 52cd072 Compare December 3, 2024 21:20

sebrandon1 added 5 commits December 5, 2024 08:30

use grpcurl for ocp 4.12 or below

cb86ef4

Remove packageManifest name filter

f7a5d33

Test grpcurl

d6edc27

Add skip for channel entry loop

d8b0307

sebrandon1 force-pushed the bundle_less_than_1000 branch from 52cd072 to d8b0307 Compare December 5, 2024 14:30

ramperher approved these changes Dec 5, 2024

View reviewed changes

sebrandon1 merged commit c90ecc2 into redhat-best-practices-for-k8s:main Dec 5, 2024
34 checks passed

sebrandon1 deleted the bundle_less_than_1000 branch December 5, 2024 16:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

operator: image bundle < 1000 references test #2549

operator: image bundle < 1000 references test #2549

sebrandon1 commented Nov 6, 2024

dcibot commented Nov 6, 2024

ramperher left a comment

dcibot commented Nov 7, 2024

ramperher commented Nov 7, 2024

sebrandon1 commented Nov 7, 2024

dcibot commented Nov 7, 2024

dcibot commented Nov 7, 2024

sebrandon1 commented Nov 7, 2024

dcibot commented Nov 7, 2024

dcibot commented Nov 8, 2024

dcibot commented Nov 8, 2024

dcibot commented Nov 8, 2024

ramperher commented Nov 8, 2024

dcibot commented Dec 2, 2024

dcibot commented Dec 3, 2024

sebrandon1 commented Dec 3, 2024

dcibot commented Dec 3, 2024

ramperher commented Dec 4, 2024

dcibot commented Dec 4, 2024

ramperher commented Dec 4, 2024 •

edited

Loading

sebrandon1 commented Dec 4, 2024 •

edited

Loading

dcibot commented Dec 5, 2024

dcibot commented Dec 5, 2024

ramperher commented Dec 5, 2024

ramperher left a comment

dcibot commented Dec 5, 2024

operator: image bundle < 1000 references test #2549

operator: image bundle < 1000 references test #2549

Conversation

sebrandon1 commented Nov 6, 2024

dcibot commented Nov 6, 2024

ramperher left a comment

Choose a reason for hiding this comment

dcibot commented Nov 7, 2024

ramperher commented Nov 7, 2024

sebrandon1 commented Nov 7, 2024

dcibot commented Nov 7, 2024

dcibot commented Nov 7, 2024

sebrandon1 commented Nov 7, 2024

dcibot commented Nov 7, 2024

dcibot commented Nov 8, 2024

dcibot commented Nov 8, 2024

dcibot commented Nov 8, 2024

ramperher commented Nov 8, 2024

dcibot commented Dec 2, 2024

dcibot commented Dec 3, 2024

sebrandon1 commented Dec 3, 2024

dcibot commented Dec 3, 2024

ramperher commented Dec 4, 2024

dcibot commented Dec 4, 2024

ramperher commented Dec 4, 2024 • edited Loading

sebrandon1 commented Dec 4, 2024 • edited Loading

dcibot commented Dec 5, 2024

dcibot commented Dec 5, 2024

ramperher commented Dec 5, 2024

ramperher left a comment

Choose a reason for hiding this comment

dcibot commented Dec 5, 2024

ramperher commented Dec 4, 2024 •

edited

Loading

sebrandon1 commented Dec 4, 2024 •

edited

Loading