-
Notifications
You must be signed in to change notification settings - Fork 222
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
.zuul, playbooks, test/system: Optimize the CI on Fedora nodes
The test suite has expanded to 415 system tests. These tests can be very I/O intensive, because many of them copy OCI images from the test suite's image cache directory to its local container/storage store, create containers, and then delete everything to run the next test with a clean slate. This makes the system tests slow. Unfortunately, Zuul's max-job-timeout setting defaults to an upper limit of 3 hours or 10800 seconds for jobs [1], and this is what Software Factory uses [2]. So, there comes a point beyond which the CI can't be prevented from timing out by increasing the timeout. One way of scaling past this maximum time limit is to run the tests in parallel across multiple nodes. This has been implemented by splitting the system tests into different groups, which are run separately by different nodes. First, the tests were grouped into those that test commands and options accepted by the toolbox(1) binary, and those that test the runtime environment within the Toolbx containers. The first group has more tests, but runs faster, because many of them test error handling and don't do much I/O. The runtime environment tests take especially long on Fedora Rawhide nodes, which are often slower than the stable Fedora nodes. Possibly because Rawhide uses Linux kernels that are built with debugging enabled, which makes it slower. Therefore, this group of tests were further split for Rawhide nodes by the Toolbx images they use. Apart from reducing the number of tests in each group, this should also reduce the amount of time spent in downloading the images. The split has been implemented with Bats' tagging system that is available from Bats 1.8.0 [3]. Fortunately, commit 87eaeea already added a dependency on Bats >= 1.10.0. So, there's nothing to worry about. At the moment, Bats doesn't expose the tags being used to run the test suite to setup_suite() and teardown_suite() [4]. Therefore, the TOOLBX_TEST_SYSTEM_TAGS environment variable was used to optimize the contents of setup_suite(). [1] https://zuul-ci.org/docs/zuul/latest/tenants.html [2] Commit 83f28c5 83f28c52e47c2d44 #1548 [3] https://bats-core.readthedocs.io/en/stable/writing-tests.html [4] bats-core/bats-core#1006 #1551
- Loading branch information
1 parent
e435704
commit 987f5e2
Showing
27 changed files
with
455 additions
and
53 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # | ||
| # Copyright © 2021 – 2024 Red Hat, Inc. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| --- | ||
| - hosts: all | ||
| tasks: | ||
| - include_tasks: build.yaml | ||
|
|
||
| - name: Run the commands-options system tests | ||
| command: bats --filter-tags commands-options ./test/system | ||
| environment: | ||
| PODMAN: '/usr/bin/podman' | ||
| TMPDIR: '/var/tmp' | ||
| TOOLBX: '/usr/local/bin/toolbox' | ||
| TOOLBX_TEST_SYSTEM_TAGS: 'arch-fedora,commands-options,custom-image,ubuntu' | ||
| args: | ||
| chdir: '{{ zuul.project.src_dir }}' |
30 changes: 30 additions & 0 deletions
30
playbooks/system-test-runtime-environment-arch-fedora.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # | ||
| # Copyright © 2021 – 2024 Red Hat, Inc. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| --- | ||
| - hosts: all | ||
| tasks: | ||
| - include_tasks: build.yaml | ||
|
|
||
| - name: Run the (arch-fedora,runtime-environment) system tests | ||
| command: bats --filter-tags arch-fedora,runtime-environment ./test/system | ||
| environment: | ||
| PODMAN: '/usr/bin/podman' | ||
| TMPDIR: '/var/tmp' | ||
| TOOLBX: '/usr/local/bin/toolbox' | ||
| TOOLBX_TEST_SYSTEM_TAGS: 'arch-fedora,runtime-environment' | ||
| args: | ||
| chdir: '{{ zuul.project.src_dir }}' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # | ||
| # Copyright © 2021 – 2024 Red Hat, Inc. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| --- | ||
| - hosts: all | ||
| tasks: | ||
| - include_tasks: build.yaml | ||
|
|
||
| - name: Run the (runtime-environment,ubuntu) system tests | ||
| command: bats --filter-tags runtime-environment,ubuntu ./test/system | ||
| environment: | ||
| PODMAN: '/usr/bin/podman' | ||
| TMPDIR: '/var/tmp' | ||
| TOOLBX: '/usr/local/bin/toolbox' | ||
| TOOLBX_TEST_SYSTEM_TAGS: 'runtime-environment,ubuntu' | ||
| args: | ||
| chdir: '{{ zuul.project.src_dir }}' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.