Heredoc notation doesn't work

[ElSamhaa]

Description

Heredoc notation is supported in Dockerfile syntax, however buildah doesn't handle it correctly.

Steps to reproduce the issue:
1.Create a Dockerfile with the following content:

FROM alpine

COPY <<eof /tmp/hello
hello there
eof

2.Run podman build -t tst .

Describe the results you received:

STEP 1: FROM alpine
STEP 2: COPY <<eof /tmp/hello
Error: error building at STEP "COPY <<eof /tmp/hello": checking on sources under "/home/myhome/Play/test/test": copier: stat: "/<<eof": no such file or directory

Describe the results you expected:
The image should build and have a file /tmp/hello that contain hello world.

Output of rpm -q buildah or apt list buildah:

-

Output of buildah version:

-

Output of podman version if reporting a podman build issue:

Version:      3.2.3
API Version:  3.2.3
Go Version:   go1.16.6
Built:        Mon Aug  2 21:39:21 2021
OS/Arch:      linux/amd64

Output of cat /etc/*release:

Fedora release 34 (Thirty Four)
NAME=Fedora
VERSION="34 (Workstation Edition)"
ID=fedora
VERSION_ID=34
VERSION_CODENAME=""
PLATFORM_ID="platform:f34"
PRETTY_NAME="Fedora 34 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:34"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f34/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=34
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=34
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation
Fedora release 34 (Thirty Four)
Fedora release 34 (Thirty Four)

Output of uname -a:

Linux fedora 5.13.12-200.fc34.x86_64 #1 SMP Wed Aug 18 13:27:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver, Must be set for proper operation.
driver = "overlay"

# Temporary storage location
runroot = "/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

# Storage path for rootless users
#
# rootless_storage_path = "$HOME/.local/share/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to the UIDs/GIDs as they should appear outside of the container,
# and the length of the range of UIDs/GIDs.  Additional mapped sets can be
# listed and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and then a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped in-container ID,
# until all of the entries have been used for maps.
#
# remap-user = "containers"
# remap-group = "containers"

# Root-auto-userns-user is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid and /etc/subgid file.  These ranges will be partitioned
# to containers configured to create automatically a user namespace.  Containers
# configured to automatically create a user namespace can still overlap with containers
# having an explicit mapping set.
# This setting is ignored when running as rootless.
# root-auto-userns-user = "storage"
#
# Auto-userns-min-size is the minimum size for a user namespace created automatically.
# auto-userns-min-size=1024
#
# Auto-userns-max-size is the minimum size for a user namespace created automatically.
# auto-userns-max-size=65536

[storage.options.overlay]
# ignore_chown_errors can be set to allow a non privileged user running with
# a single UID within a user namespace to run containers. The user can pull
# and use any image even those with multiple uids.  Note multiple UIDs will be
# squashed down to the default uid in the container.  These images will have no
# separation between the users in the container. Only supported for the overlay
# and vfs drivers.
#ignore_chown_errors = "false"

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,metacopy=on"

# Set to skip a PRIVATE bind mount on the storage home directory.
# skip_mount_home = "false"

# Size is used to set a maximum size of the container image.
# size = ""

# ForceMask specifies the permissions mask that is used for new files and
# directories.
#
# The values "shared" and "private" are accepted.
# Octal permission masks are also accepted.
#
#  "": No value specified.
#     All files/directories, get set with the permissions identified within the
#     image.
#  "private": it is equivalent to 0700.
#     All files/directories get set with 0700 permissions.  The owner has rwx
#     access to the files. No other users on the system can access the files.
#     This setting could be used with networked based homedirs.
#  "shared": it is equivalent to 0755.
#     The owner has rwx access to the files and everyone else can read, access
#     and execute them. This setting is useful for sharing containers storage
#     with other users.  For instance have a storage owned by root but shared
#     to rootless users as an additional store.
#     NOTE:  All files within the image are made readable and executable by any
#     user on the system. Even /etc/shadow within your image is now readable by
#     any user.
#
#   OCTAL: Users can experiment with other OCTAL Permissions.
#
#  Note: The force_mask Flag is an experimental feature, it could change in the
#  future.  When "force_mask" is set the original permission mask is stored in
#  the "user.containers.override_stat" xattr and the "mount_program" option must
#  be specified. Mount programs like "/usr/bin/fuse-overlayfs" present the
#  extended attribute permissions to processes within containers rather then the
#  "force_mask"  permissions.
#
# force_mask = ""

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# metadata_size is used to set the `pvcreate --metadatasize` options when
# creating thin devices. Default is 128k
# metadata_size = ""

# Size is used to set a maximum size of the container image.
# size = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I tried this with Docker, and it fails?

# systemctl start docker
# cat /tmp/test/Dockerfile
FROM alpine

COPY <<eof /tmp/hello
hello there
eof
# docker build /tmp/test/
Sending build context to Docker daemon  2.048kB
Error response from daemon: dockerfile parse error line 4: unknown instruction: HELLO

[nalind]

It's a labs (experimental?) feature.

[rhatdan]

Ok But this still breaks on docker buildx build at least in the version of Docker i have installed.

# docker buildx build /tmp/test/
[+] Building 0.2s (6/6) FINISHED                                                 
 => [internal] load build definition from Dockerfile                        0.1s
 => => transferring dockerfile: 150B                                        0.0s
 => [internal] load .dockerignore                                           0.1s
 => => transferring context: 2B                                             0.0s
 => [internal] load metadata for docker.io/library/alpine:latest            0.0s
 => [internal] load build context                                           0.0s
 => => transferring context: 2B                                             0.0s
 => [1/2] FROM docker.io/library/alpine                                     0.0s
 => ERROR [2/2] COPY <<eof /tmp/hello hello there eof                       0.0s
------
 > [2/2] COPY <<eof /tmp/hello hello there eof:
------
error: failed to solve: failed to compute cache key: "/hello" not found: not found

[rhatdan]

# docker buildx build /tmp/test/
[+] Building 0.1s (2/2) FINISHED                                                 
 => [internal] load build definition from Dockerfile                        0.0s
 => => transferring dockerfile: 89B                                         0.0s
 => [internal] load .dockerignore                                           0.0s
 => => transferring context: 2B                                             0.0s
error: failed to solve: failed to solve with frontend dockerfile.v0: failed to create LLB definition: dockerfile parse error line 4: unknown instruction: HELLO

[nalind]

My understanding is that this requires the correct # syntax=docker/dockerfile:1.3-labs at the top of the Dockerfile.

[rhatdan]

Yup that works.

# docker buildx build /tmp/test/
[+] Building 5.9s (11/11) FINISHED                                               
 => [internal] load build definition from Dockerfile                        0.0s
 => => transferring dockerfile: 184B                                        0.0s
 => [internal] load .dockerignore                                           0.0s
 => => transferring context: 2B                                             0.0s
 => resolve image config for docker.io/docker/dockerfile:1.3-labs           1.3s
 => docker-image://docker.io/docker/dockerfile:1.3-labs@sha256:03ca0e50aa4  4.3s
 => => resolve docker.io/docker/dockerfile:1.3-labs@sha256:03ca0e50aa4b6e7  0.0s
 => => sha256:ce897ecde42e8c45c0056748ec1a17a3edc5b7d3b091 9.67MB / 9.67MB  4.1s
 => => sha256:03ca0e50aa4b6e76365fa9a5607c3f988bc9284de6a8 2.00kB / 2.00kB  0.0s
 => => sha256:c6afe91d6f5e32ee187adf7516f1ced1d7d2f9621ff4e907 528B / 528B  0.0s
 => => sha256:456bbe8ddbe759d5109d4f5bfe8ce105c0086863298f 1.21kB / 1.21kB  0.0s
 => => extracting sha256:ce897ecde42e8c45c0056748ec1a17a3edc5b7d3b091dae35  0.1s
 => [internal] load .dockerignore                                           0.0s
 => [internal] load build definition from Dockerfile                        0.0s
 => [internal] load metadata for docker.io/library/alpine:latest            0.0s
 => CACHED [1/2] FROM docker.io/library/alpine                              0.0s
 => [internal] preparing inline document                                    0.0s
 => [2/2] COPY <<eof /tmp/hello                                             0.0s
 => exporting to image                                                      0.0s
 => => exporting layers                                                     0.0s
 => => writing image sha256:c14ea19390b6e4241141c42a5fb74abccf9c22af8f9a48  0.0s

Do you think we should support it?

[nalind]

Yeah, but I'm not working on the patches yet.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc PTAL.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc Did you ever get a chance to look at this?

[flouthoc]

I'll check this thanks.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc any progress?

[flouthoc]

Ackd. Not yet, will start working on this.

[guillaume-d]

It's a labs (experimental?) feature.

Meanwhile it's supported by default by BuildKit >= v0.10.1.
(That BuildKit version was the first one to include a Dockerfile frontend as the built-in one with a syntax >= 1.4 (1.4.1), which is required for here-documents.)

[mwoodpatrick]

Any ETA for when this might be resolved?

[rhatdan]

I am sure PRs would help move it along. :^) I don't believe anyone has looked at this yet.

[lizelive]

Also impacted by this issue

[asishrs]

Waiting for this.

[flouthoc]

I already have somewhat working code for this I'll open a draft PR soon.

[rhatdan]

Great, and for those complaining and stating that they are waiting... The best way to get a feature into an opensource project is to roll up ones sleeves and create a PR.

[kevin-medme]

+1 heredoc would be great.

If anyone lands here and finds themselves needing an alternate approach / workaround there are a few angles of attack. The following example is not as elegant as heredoc but I can confirm it's working with podman 4.3.1:

RUN echo '#!/usr/bin/env bash\n\
  # example \n\
  echo "hello world"' > /path/to/file.sh

[christhomas]

Is this feature still not released? I tried to use this today and it was after reading an article from july 2021

[hussam-qasem]

Greetings! On the Docker website, it's now under Here-Documents / Creating inline files, and requires adding this line at the top of the Dockerfile:

# syntax=docker/dockerfile:1

I just thought the syntax was cleaner than something like:

FROM alpine
RUN mkdir /app && echo -e "\
    echo hello \${FOO}\n\
" > /app/script.sh
RUN FOO=abc ash /app/script.sh

[rhatdan]

@flouthoc @nalind lets talk about this next week.

[andy-vs]

Is this available via the REST API as well?

# syntax=docker/dockerfile:1.4-labs
FROM alpine
RUN <<EOF
apt-get update --fix-missing
EOF

This generates an error for apt-get
Error: (HTTP code 400) unexpected - {"message":"dockerfile parse error line 10: unknown instruction: APT-GET"}

Note that this runs via. docker build .

[Erotemic]

I'm hoping that there is some movement on this soon. It would really be useful.

[flouthoc]

I should be ready with a PR soon, I have early infrastructure for this on the branches flouthoc/imagebuilder@000f471 , https://github.com/flouthoc/imagebuilder/tree/heredoc and https://github.com/flouthoc/buildah/tree/heredoc-experiment

[flouthoc]

Okay my branch https://github.com/flouthoc/buildah/tree/heredoc-experiment and most recent commit i.e flouthoc@270f3f9 has a working version with RUN in case if someone wants to try it out.

A tryout Containerfile could be

FROM ubuntu:20.04
RUN <<EOF
echo "Hello" >> /hello
echo "World!" >> /hello
EOF

RUN ls -a
RUN cat hello

I will open a PR in coming days once i gets a ack on design from maintainers.

Thanks

[EvanCarroll]

@flouthoc maybe the best way to get the ack is to open the PR?

[mafeifan]

DOCKER_BUILDKIT=1 docker build .
If you are running a version of Docker Engine version earlier than 23.0

https://docs.docker.com/build/buildkit/#getting-started

[rhatdan]

@flouthoc THis is rising in importance since Podman-desktop needs this.

[TomSweeneyRedHat]

And FWIW, Mark Russel and I were talking about this yesterday. It would be good if we can squeeze this into 4.8.

[flouthoc]

This feature is merged into upstream via #5092 and will be included in next release.

[TomSweeneyRedHat]

This is in Buildah v1.33.0 and above, FWIW