-
Notifications
You must be signed in to change notification settings - Fork 459
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[chore] Add a smoke test that is run as part of the scorecard tests V2 #1335
base: main
Are you sure you want to change the base?
[chore] Add a smoke test that is run as part of the scorecard tests V2 #1335
Conversation
|
a74867c
to
5d0d2c6
Compare
5d0d2c6
to
70751fa
Compare
70751fa
to
9385461
Compare
bundle/tests/scorecard/config.yaml
Outdated
- image: quay.io/operator-framework/scorecard-test-kuttl:v2.0.0 | ||
labels: | ||
suite: kuttlsuite | ||
test: kuttltest1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How is the label used?
Should we use a more descriptive label?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Afaics it's just printed in the scorecard output, i.e.
{
"kind": "Test",
"apiVersion": "scorecard.operatorframework.io/v1alpha3",
"spec": {
"image": "quay.io/operator-framework/scorecard-test-kuttl:v2.0.0",
"labels": {
"suite": "kuttlsuite",
"test": "kuttltest1"
},
"storage": {
"spec": {
"mountPath": {}
}
}
},
"status": {
"results": [
{
"name": "smoketest",
"state": "pass",
"creationTimestamp": null
}
]
}
}
Currently all tests of this kuttl suite (bundle/tests/scorecard/kuttl
) are run in parallel as a single scorecard test. If we decide later that we want to split kuttl tests in multiple scorecard tests, we should add a descriptive label to each. For now I think it's fine, as in other projects we also run all kuttl tests in the same kuttl suite (i.e. tests/e2e
is a single kuttle suite).
52d0ecb
to
27fcd2d
Compare
Co-authored-by: Kevin Earls <[email protected]> Setup service account and e2e env for kuttl scorecard tests Signed-off-by: Andreas Gerstmayr <[email protected]>
…Makefile Signed-off-by: Andreas Gerstmayr <[email protected]>
Signed-off-by: Andreas Gerstmayr <[email protected]>
27fcd2d
to
f0661f2
Compare
/easycla |
Signed-off-by: Andreas Gerstmayr <[email protected]>
- op: add | ||
path: /stages/0/tests/- | ||
value: | ||
image: docker.io/andreasgerstmayr/scorecard-test-kuttl:1681909083 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is it using a custom image?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It needs kuttl 0.15.0 because this kuttl version contains fixes regarding test cleanup, which is especially important if we run the tests in a single namespace.
Once operator-framework/operator-sdk#6401 is merged and released, we can switch to the official one again.
@@ -45,6 +46,12 @@ stages: | |||
storage: | |||
spec: | |||
mountPath: {} | |||
- image: docker.io/andreasgerstmayr/scorecard-test-kuttl:1681909083 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is it using a custom image? Could we switch to the upstream one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above (this file is auto-generated from the file above).
Signed-off-by: Andreas Gerstmayr <[email protected]>
This is a continuation of PR #1252, rebased on latest main branch with the following updates: