go builds start failing with buildah 1.19.4

[mythi]

Description

Our CI builds kubernetes device plugins containers (Dockerfiles) using both docker and buildah. Buildah is installed from http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/.

The repo moved to buildah 1.19.4 over the weekend and our buildah images started failing.

Describe the results you received:
go commands error with:
build cache is required, but could not be located: GOCACHE is not defined and neither $XDG_CACHE_HOME nor $HOME are defined

For what is worth, I can add RUN go env debug prints and they work OK first and then suddenly start failing.

Describe the results you expected:
builds work OK. FWIW, Github Actions virtual env runners have buildah v1.19.2 and they work fine.

Output of rpm -q buildah or apt list buildah:

buildah/unknown,now 100:1.19.4-1 amd64 [installed]
buildah/unknown 100:1.19.4-1 arm64
buildah/unknown 100:1.19.4-1 armhf
buildah/unknown 100:1.19.4-1 ppc64el
buildah/unknown 100:1.19.4-1 s390x

Output of buildah version:

Version:         1.19.4
Go Version:      go1.15.2
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.10.1
Git Commit:      
Built:           Thu Jan  1 02:00:00 1970
OS/Arch:         linux/amd64

[rhatdan]

I am not sure what issue you are reporting? Are you reporting the Buildah fails to be able to build from Git?

[mythi]

I am not sure what issue you are reporting?

My buildah bud -f <Dockerile> builds start failing with 1.19.4. The error is that go suddenly fails to run, somehow the env gets destroyed between the layers

[rhatdan]

Can you give us a Dockerfile that shows the behaviour you are seeing?

[mythi]

@rhatdan The failing Dockerfiles are in our repository.

[rhatdan]

Everyone of them or one in particular?

[mythi]

Everyone of them, I have debugged with intel-gpu-plugin locally.

[TomSweeneyRedHat]

@umohnani8 can you take a quick peak at this?

[mythi]

NB: The latest master has a workaround so if you are testing with HEAD you don't see errors.
I added

RUN mkdir -p /go/cache
ENV GOCACHE=/go/cache

right before go install to make go work. If you're testing the failure, revert the workaround

[mythi]

fails with v1.19.6 too

[umohnani8]

So looked through the history and this is the PR that caused this https://github.com/containers/buildah/pull/2929/files. From some tests, looks like this is failing when the OCI Runtime is crun, which comes from the containers.conf file. When I revert the patch and use runc as the DefaultRuntime it works fine.
@giuseppe any ideas why it is failing with crun?

[mythi]

@umohnani8 Thanks! So for me this seems to be just a config issue (I can move back to runc).

[umohnani8]

@mythi yes, moving back to runc (setting the runtime field in containers.conf to runc) should fix your issue while we investigate why it blows up with crun.

[giuseppe]

I think this is the same issue as containers/podman#9378

crun behaves differently than runc (on purpose) and doesn't try to override HOME if it set to the empty value.

can you show the result of env in your image?

The best fix is to override HOME and set it to a reasonable value instead of depending on the runtime to set it.

[mythi]

crun behaves differently than runc (on purpose) and doesn't try to override HOME if it set to the empty value.

root@b16db14e2a9d/go # env|grep HOME
HOME=/
root@b16db14e2a9d/go # cat /etc/passwd
cat: /etc/passwd: No such file or directory

[mythi]

My fix is to:

-    buildah bud --pull-always -t ${IMG}:${TAG} ${BUILD_ARGS} -f ${DOCKERFILE} .
+    BUILDAH_RUNTIME=runc buildah bud --pull-always -t ${IMG}:${TAG} ${BUILD_ARGS} -f ${DOCKERFILE} .