Skip to content

Commit

Permalink
playbooks/system-test, test/system: Avoid running out of storage space
Browse files Browse the repository at this point in the history
The system tests download several images when setting up the test suite,
and cache them for later use by the tests [1].  This saves time and
avoids hitting rate limits imposed by OCI registries by not downloading
the same images repeatedly for several tests, but at the cost of
increased use of storage space to cache the images.

The images are cached under BATS_TMPDIR.  It defaults to the TMPDIR
environment variable, and if that's not set then to /tmp [2].  Normally,
TMPDIR isn't set, and the images end up getting cached under /tmp.  Now,
/tmp is typically on tmpfs backed by RAM or swap, which means that it
should be used for smaller size-bounded files only, and /var/tmp should
be used for everything else [3].

The images are big enough that a collection of them can't be described
as smaller and size-bounded, and it led to:
  1..306
  # test suite: Set up
  # test suite: Tear down
  not ok 1 setup_suite
  # (from function `setup_suite' in test file ./setup_suite.bash, line
      55)
  #   `_pull_and_cache_distro_image fedora "$((system_version-1))" ||
      false' failed
  # Failed to cache image registry.fedoraproject.org/fedora-toolbox:40
      to /tmp/bats-run-IPz4Cn/image-cache/fedora-toolbox-40
  # time="2024-02-19T11:41:43Z" level=fatal msg="copying system image
      from manifest list: writing blob: write
      /tmp/bats-run-IPz4Cn/image-cache/fedora-toolbox-40/dir-put-blob607392514:
      no space left on device"
  # bats warning: Executed 1 instead of expected 306 tests

So, change the default location of the BATS_TMPDIR environment variable
to /var/tmp by setting TMPDIR.

[1] Commit 50683c9
    containers@50683c9d9a78adc9
    containers#375

[2] https://bats-core.readthedocs.io/en/stable/writing-tests.html

[3] https://systemd.io/TEMPORARY_DIRECTORIES/

containers#1462
  • Loading branch information
debarshiray committed Feb 29, 2024
1 parent e270078 commit 571dc97
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 0 deletions.
1 change: 1 addition & 0 deletions playbooks/system-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
command: bats --timing ./test/system
environment:
PODMAN: '/usr/bin/podman'
TMPDIR: '/var/tmp'
TOOLBX: '/usr/local/bin/toolbox'
args:
chdir: '{{ zuul.project.src_dir }}'
6 changes: 6 additions & 0 deletions test/system/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,12 @@ location then you can use the `PODMAN`, `SKOPEO` and `TOOLBX` environmental
variables to set the path to the binaries. So the command to invoke the test
suite could look something like this: `PODMAN=/usr/libexec/podman TOOLBX=./toolbox bats ./test/system/`.

It's recommended to set the [TMPDIR](https://systemd.io/TEMPORARY_DIRECTORIES/)
environment variable to `/var/tmp` when running the tests. Otherwise, the
images downloaded during the tests will be cached in `/tmp`, which is
typically on tmpfs backed by RAM or swap and should be used for smaller
size-bounded files only.

When running the tests, make sure the `test suite: [job]` jobs are successful.
These jobs set up the whole environment and are a strict requirement for other
jobs to run correctly.
Expand Down

0 comments on commit 571dc97

Please sign in to comment.