Change in the way quotes in env files are treated

[morty]

Description of the issue

The way docker-compose treats quoted values in env_files has changed. The documentation says:

The value of VAL is used as is and not modified at all. For example if the value is surrounded by quotes (as is often the case of shell variables), the quotes are included in the value passed to Compose.

But now if the value is quoted the container sees the unquoted value. This means that the same env file provided via docker run and docker-compose will give different environment variables inside the container.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Output of docker version

Client:
 Cloud integration: 1.0.17
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.16.4
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:22 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:58 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Steps to reproduce the issue

# cat test.env
TESTING="123"

# cat docker-compose.yml
version: "3.7"
services:
  test:
    image: alpine
    env_file:
      - test.env
    command: "env"

# docker-compose up

Observed result

Starting docker-env-issue_test_1 ... done
Attaching to docker-env-issue_test_1
test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_1  | HOSTNAME=91644b7b5129
test_1  | TESTING=123
test_1  | HOME=/root
docker-env-issue_test_1 exited with code 0

Expected result

Starting docker-env-issue_test_1 ... done
Attaching to docker-env-issue_test_1
test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_1  | HOSTNAME=91644b7b5129
test_1  | TESTING="123"
test_1  | HOME=/root
docker-env-issue_test_1 exited with code 0

Additional information

The commit which has caused the regression seems to be 6d2658e.

[ndeloof]

The way docker run uses env files as "raw values" is unfortunately not consistent with .env usages in Python/Ruby/xx. We hardly can reconcile them into an homogeneoux US.

[benatkin]

This changes from using a custom parser to using the dotenv library. I think the new behavior is better, but it appears to be labeled a "bug fix" in the changelog, and not called out as a breaking change. To be fair, it is a departure from .env, which is a standard, one that needs work, but still a standard.

@ndeloof are you using an old version of docker-compose? I read this issue wrong the first time. The "expected result" is that it carries the quotes to the environment variable, and the actual result is that it gets rid of the quotes, which is what I've come to expect. It does this by using the dotenv library.

[benatkin]

With Docker Compose 2.0 on the way, it's good that this change already been made, so 2.x can have one fewer breaking changes.

[benatkin]

The current docs are incorrect

Compose expects each line in an env file to be in VAR=VAL format. Lines beginning with # are treated as comments and are ignored. Blank lines are also ignored.

...should be changed to something like:

Compose uses python-dotenv to read the environment file. VAR="VAL", VAR='VAL', and VAR=VAL all resolve to the same. \n in double quotes ("\n") will be converted into a newline, but when \n is in single quotes ('\n') the literal characters \n will be preserved.

[ndeloof]

it is a departure from .env, which is a standard

Unfortunately, dotEnv is only a de facto standard, with no precise definition. For sample, Python dotEnv doesn't behave exactly the same as Ruby dotEnv.

Docs is indeed incorrect, I have a pending PR to make this change more visible: https://github.com/docker/docker.github.io/pull/13474/files#diff-8edc9c85338bc8e7c7a80ef3a0a01ec54cfad9e2ac61824e40c42c66604c3355

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.