CI and f37 gating tests: failure in --runtime-flag test - maybe crun?

[edsantiago]

Impossible-to-debug failure in f37 gating tests:

not ok 696 run --runtime --runtime-flag
# (in test file /usr/share/buildah/test/system/run.bats, line 810)
#   `[ -n "$output" ]' failed
# /usr/share/buildah/test/system /var/str
# # [checking for: docker.io/library/alpine]
# # [restoring from cache: /var/tmp/buildah-image-cache.5515 / docker.io/library/alpine]
# Getting image source signatures
# Copying blob sha256:9d16cba9fb961d1aafec9542f2bf7cb64acfc55245f9e4eb5abecd4cdc38d749
# Copying config sha256:961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
# Writing manifest to image destination
# Storing signatures
# $ /usr/bin/buildah from --security-opt seccomp=/var/tmp/buildah_tests.6b1cxe/seccomp.json --quiet --pull=false --signature-policy /usr/share/buildah/test/system/policy.json alpine
# alpine-working-container
# $ /usr/bin/buildah run --runtime=crun --runtime-flag=debug alpine-working-container true
# /var/str

The broken test is being fixed in #4502; until that merges we can't actually know what happened. One possibility is that crun changed somehow, such that debug no longer produces debug output? The test failure suggests that this command:

$ buildah run --runtime=crun --runtime-flag=debug alpine-working-container true

...should produce output, but it isn't? Could this be a crun change? A change in how buildah passes the --runtime-flag option?

[rhatdan]

@flouthoc PTAL

[edsantiago]

This is now breaking all CI jobs [example]:

[+2905s] # #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
[+2905s] # #|     FAIL: Output from running 'true' with --runtime-flag=debug
[+2905s] # #| expected: != ''
[+2905s] # #|   actual:    ''
[+2905s] # #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[edsantiago]

@flouthoc @giuseppe @nalind PTAL high priority? All of CI is broken, and gating-tests will be too. I don't know if this is a breakage in crun, or buildah, or some other component. This should be very quick for one of you to figure out. TIA.

[Luap99]

I guess #4514 broke it here because it updated the VM images without running the full tests. Likely some change in the underlying oci runtime that is used.
cc @cevich

[Luap99]

Can be reproduced locally as well bats -f "run --runtime --runtime-flag" tests/
crun 1.7.2 fails, 1.6 works

[Luap99]

git bisect points to containers/crun@c4af47e
I have no idea how this is related, @giuseppe @flouthoc PTAL

[cevich]

FWIW, the images were build here containers/automation_images#245 (comment) (links to build logs)

[flouthoc]

Checking

[giuseppe]

that happens because newer crun versions use a cache for seccomp, so the warning happens only the first time it is built

[giuseppe]

one way to address it would be to use a new root directory:

diff --git a/tests/run.bats b/tests/run.bats
index 7f8d1d7d9..8c82b124f 100644
--- a/tests/run.bats
+++ b/tests/run.bats
@@ -824,7 +824,7 @@ _EOF
 
        if [ -n "$(command -v crun)" ]; then
                found_runtime=y
-               run_buildah run --runtime=crun --runtime-flag=debug $cid true
+               run_buildah run --runtime=crun --runtime-flag=root=${TEST_SCRATCH_DIR}/crun --runtime-flag=debug $cid true
                assert "$output" != "" "Output from running 'true' with --runtime-flag=debug"
        fi

[Luap99]

are there no debug logs in crun? runc prints a lot of verbose logs, maybe crun should do so too? At least --debug implies that it does not only print warnings.

[giuseppe]

not really, we added --debug just for CLI compatibility with runc

[Luap99]

fair enough, if your diff fixes the test please open a PR with it

[giuseppe]

I'd rather prefer we drop the check, it has an assumption on the behavior of the runtime that is not documented anywhere

[Luap99]

That is an option but how should --runtime-flag be tested then? I guess it could use the same thing like the bud test does?

buildah/tests/bud.bats

Lines 4754 to 4773 in 39dfa73

	if [ -n "$(command -v crun)" ]; then
	found_runtime=y
	# Use seccomp to make crun output a warning message because crun writes few logs.
	cat > ${TEST_SCRATCH_DIR}/seccomp.json << _EOF
	{
	"defaultAction": "SCMP_ACT_ALLOW",
	"syscalls": [
	{
	"name": "unknown",
	"action": "SCMP_ACT_KILL"
	}
	]
	}
	_EOF
	run_buildah build --runtime=crun --runtime-flag=debug --security-opt seccomp=${TEST_SCRATCH_DIR}/seccomp.json \
	-q -t alpine-bud-crun $WITH_POLICY_JSON --file ${mytmpdir}/Containerfile .
	expect_output --substring "unknown seccomp syscall"
	fi

[giuseppe]

proposed fix: #4523