Dockerfile containing RUN with heredoc is accepted by docker build but not docker buildx build, where you get "error: failed to solve: rpc error: code = Unknown desc = unterminated heredoc"

[oliverlockwood]

My (obviously simplified for demonstration purposes) Dockerfile:

FROM debian:buster

RUN echo <<EOF \
    hello \
    mr \
    potato \
    EOF > /tmp/output && \
    echo 'done'

(although, based on comments in #2439, I have also tried line 3 as RUN echo <<-"EOF" \ without any difference in behaviour).
#2170 seems to suggest that heredoc in RUN commands is supported, though I couldn't see an example where the Herero

Running a normal docker build:

mycomputer$ docker build --no-cache -f Dockerfile --pull -t ogl-test .
Sending build context to Docker daemon  42.82MB
Step 1/2 : FROM debian:buster
buster: Pulling from library/debian
Digest: sha256:e5b41ae2b4cf0d04b80cd2f89724e9cfc09e334ac64f188b9808929c748af526
Status: Image is up to date for debian:buster
 ---> 354ff99d6bff
Step 2/2 : RUN echo <<-"EOF"     hello     mr     potato     EOF > /tmp/output &&     echo 'done'
 ---> Running in 615df9c4f39e
done
Removing intermediate container 615df9c4f39e
 ---> 1107a46f07b7
Successfully built 1107a46f07b7
Successfully tagged ogl-test:latest
mycomputer$ docker run -it ogl-test
root@9adca399ee6f:/# cat /tmp/output
hello mr potato EOF
root@9adca399ee6f:/# exit
exit

Running docker buildx build:

$ docker buildx build --no-cache -f Dockerfile --platform linux/amd64 --pull -t ogl-test .
[+] Building 0.0s (2/2) FINISHED
 => [internal] load .dockerignore                                                                                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                                                                           0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                      0.0s
 => => transferring dockerfile: 155B                                                                                                                                                                                                                      0.0s
Dockerfile:3
--------------------
   2 |
   3 | >>> RUN echo <<-"EOF" \
   4 | >>>     hello \
   5 | >>>     mr \
   6 | >>>     potato \
   7 | >>>     EOF > /tmp/output && \
   8 | >>>     echo 'done'
   9 | >>>
  10 |
--------------------
error: failed to solve: rpc error: code = Unknown desc = unterminated heredoc

Version information:

docker version
Client: Docker Engine - Community
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:18:20 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:16:15 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[jedevc]

Just to sanity check - are you doing both builds with buildkit? You should be able to confirm by comparing the results of DOCKER_BUILDKIT=1 docker build . and DOCKER_BUILDKIT=0 docker build .

Heredocs are only supported in buildkit, and probably won't be backported to the legacy builder.

Remember, that heredoc terminators need to be terminated without the indent. Your example would probably need to look more like (using cat instead of echo, to actually read the heredoc):

FROM debian:buster

RUN cat <<EOF > /tmp/output && echo 'done'
hello
mr
potato
EOF

[oliverlockwood]

@jedevc thanks for getting in touch. With your suggested Dockerfile, regardless of whether I use DOCKER_BUILDKIT=0 or DOCKER_BUILDKIT=1, it errors like this:

failed to solve with frontend dockerfile.v0: failed to create LLB definition: dockerfile parse error line 4: unknown instruction: HELLO

If I change this to include newline escapes, i.e.:

FROM debian:buster

RUN cat <<EOF > /tmp/output && echo 'done' \
hello \
mr \
potato \
EOF

then it builds, but in both cases the /tmp/output file is empty.

If I return to my original Dockerfile, then docker build behaves identically regardless of the setting of DOCKER_BUILDKIT.

I still cannot find an incantation which produces content in an output file (thereby simulating the actual commands I want to run) and works in docker buildx build, as per the original issue description.

Perhaps you can try it at your end? The idea behind my producing a simplified repro scenario was that it would be easy for you to reproduce the issue yourself 😃

[jedevc]

Hey! So glanced through your version information again - it doesn't look like you have a version that supports heredocs. It was released about a year ago, and heredocs weren't in the stable channel then.

Take a look through the examples presented in the documentation: you'll need to include the #syntax=docker/dockerfile:1.4 or some other version that includes heredocs support.

I can't reproduce the scenario you're describing on an up to date docker installation, have a try with the syntax directive 🎉

[oliverlockwood]

@jedevc Thanks for the feedback. I'll have a go at this. Stand by.

[jedevc]

@oliverlockwood closing this now as resolved - feel free to re-open if you're still having issues with this 😄