podman build fails: Unable to copy tar file from request body

[roy-work]

Description

Running podman build ../../.. -f local.Dockerfile on a Dockerfile we have causes podman to hang for ~12 minutes before finally failing with:

Error: failed Request: Unable to copy tar file from request body /v3.3.1/libpod/build?dockerfile=%5B%22<snip>%22%5D&forcerm=1&isolation=1&jobs=1&layers=1&networkmode=0&nsoptions=%5B%7B%22Name%22%3A%22user%22%2C%22Host%22%3Atrue%2C%22Path%22%3A%22%22%7D%2C%7B%22Name%22%3A%22network%22%2C%22Host%22%3Atrue%2C%22Path%22%3A%22%22%7D%5D&outputformat=application%2Fvnd.oci.image.manifest.v1%2Bjson&pullpolicy=missing&rm=1&shmsize=67108864&t=

Steps to reproduce the issue:

1. podman build ../../.. -f local.Dockerfile

Describe the results you received:
The above error.

Describe the results you expected:
A build Docker image.

Output of rpm -q buildah or apt list buildah:
This is on macOS?

Output of buildah version:

» buildah version
zsh: command not found: buildah

Output of podman version if reporting a podman build issue:

» podman version
Client:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.17
Built:        Mon Aug 30 15:15:26 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Mon Aug 30 16:46:36 2021
OS/Arch:      linux/amd64

Output of cat /etc/*release:
macOS Big Sur Version 11.4; on an Intel.

Output of uname -a:

Darwin <snip> 20.5.0 Darwin Kernel Version 20.5.0: Sat May  8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64 x86_64

Output of cat /etc/containers/storage.conf:

cat: /etc/containers/storage.conf: No such file or directory

Other information

I'm not able to share local.Dockerfile, unfortunately, as it is proprietary. I suspect the issue here is that the context is … large. We normally build with BuildKit, which will only transmit the portions of the context necessary for the actual build, and that's much more manageable in our case. Before any .dockerignore, the context is ~81 GiB, although at least ~13 GiB should be covered under ignores.

Subsequent runs of podman build resulted in,

Error: mkdir /var/tmp/libpod_builder190006202: no space left on device

So it seems like it didn't even clean up after failing. (That appears to be the VM's device, not the hosts.)

Reset the VM from scratch. Shrunk my context to ~38 GiB before ignores (so ~25 GiB?), we'll see if that works (it did not work), but I'm not hopeful, since I think the VM is by default only 10 GiB? Going to probably need to figure out how to increase it's size…

[flouthoc]

@roy-work thanks for reporting this, similar issue is being tracked here containers/buildah#2850, do you think issue is duplicate ? If yes I'd suggested to lookout for updates on original issue as i am just picking it up.

Edit: RootCause of this is not same as containers/buildah#2850.

[rostrovsky]

I have similar issue although I have different scenario:

1. Container #1 podman with service started as sudo podman system service --log-level=debug --time=0 tcp:0.0.0.0:8080
2. Container #2 - docker client with DOCKER_HOST=tcp://<podman_container>:8080

docker pull ... works fine in Container #2, but building from Dockerfile fails with reason:

APIHandler(0fb04acc-86e2-45f5-b4aa-08a16ba2d96a) -- POST /v1.40/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&session=mm8c5i8pftmsrpdrbxcser2zo&shmsize=0&t=something&target=&ulimits=null&version=1 BEGIN
DEBU[0489] APIHandler(0fb04acc-86e2-45f5-b4aa-08a16ba2d96a) -- Header: User-Agent=[Docker-Client/19.03.14 (linux)]
DEBU[0489] APIHandler(0fb04acc-86e2-45f5-b4aa-08a16ba2d96a) -- Header: Content-Type=[application/x-tar]
DEBU[0489] APIHandler(0fb04acc-86e2-45f5-b4aa-08a16ba2d96a) -- Header: X-Registry-Config=<hidden>
DEBU[0489] IdleTracker 0xc0004b8098:active 1m+0h/5t connection(s)
INFO[0489] Failed Request: (404:Not Found) for POST:'/session'
DEBU[0489] IdleTracker 0xc0004b8098:idle 2m+0h/5t connection(s)
DEBU[0489] IdleTracker 0xc0004b8098:closed 2m+0h/5t connection(s)
INFO[0489] Request Failed(Internal Server Error): failed Request: Unable to copy tar file from request body /v1.40/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&session=mm8c5i8pftmsrpdrbxcser2zo&shmsize=0&t=something&target=&ulimits=null&version=1
DEBU[0489] APIHandler(0fb04acc-86e2-45f5-b4aa-08a16ba2d96a) -- POST /v1.40/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&session=mm8c5i8pftmsrpdrbxcser2zo&shmsize=0&t=something&target=&ulimits=null&version=1 END
DEBU[0489] IdleTracker 0xc0000105b8:closed 1m+0h/5t connection(s)

Please ignore the above, I have solved it

I was using old docker client image in Container #2. Issue disappeared once I have used docker:20.10.7 image.

[baude]

Are we good to go with 3.4?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I believe this is fixed in the current release, reopen if I am mistaken.

[fdcds]

I believe this is fixed in the current release, reopen if I am mistaken.

I am using Podman 3.4.4 with Docker 18.09.5 in a container and get:

Request Failed(Internal Server Error): failed Request: Unable to copy tar file from request body /v1.39/build?[...]

Updating to Docker 20.10.12 makes the build succeed. Will the problem be fixed in a future Podman release or is the solution to this issue to update the Docker client to one that uses API v1.40?