Replicate environment variable from local environment fails with default .env -file

[hmontone]

Describe the bug
If environment variable value is defined in default .env -file and environment variable is replicated in compose.yml file, environment variable is not passed to container

To Reproduce
.env:

TEST=test

compose.yml:

services:
  test:
    image: busybox
    environment:
      - TEST
    command: printenv

Expected behavior
Environment variable TEST should be included in output

Actual behavior
Environment variable TEST is not included in output

Output

$ podman compose run --rm test
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
container=podman
TERM=xterm
HOME=/root
HOSTNAME=aa7b6faaf810

My environment:

$ podman compose run --rm test
container=podman
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/root
HOSTNAME=6da914b12a2e
$ podman compose version
podman-compose version 1.2.0
podman version 5.1.2
$ cat /etc/os-release
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo
$ pacman -Qi podman podman-compose
Name            : podman
Version         : 5.1.2-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/podman
Licenses        : Apache-2.0
Groups          : None
Provides        : None
Depends On      : catatonit  conmon  containers-common  oci-runtime  gcc-libs  glibc  iptables  device-mapper  libdevmapper.so  gpgme  libgpgme.so=11-64  libseccomp
                  libseccomp.so=2-64  passt
Optional Deps   : apparmor: for AppArmor support
                  btrfs-progs: support btrfs backend devices
                  cni-plugins: for an alternative container-network-stack implementation
                  fuse-overlayfs: for storage driver in rootless environment
                  slirp4netns: for alternative rootless network support
                  podman-compose: for docker-compose compatibility [installed]
                  podman-docker: for Docker-compatible CLI [installed]
Required By     : podman-compose  podman-docker
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 77.29 MiB
Packager        : David Runge <[email protected]>
Build Date      : Fri 12 Jul 2024 12:26:12 AM EEST
Install Date    : Sat 13 Jul 2024 07:33:05 PM EEST
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Name            : podman-compose
Version         : 1.2.0-1
Description     : A script to run docker-compose.yml using podman
Architecture    : any
URL             : https://github.com/containers/podman-compose
Licenses        : GPL-2.0-only
Groups          : None
Provides        : None
Depends On      : podman  python  python-dotenv  python-yaml
Optional Deps   : aardvark-dns: resolve hostnames of linked containers [installed]
Required By     : None
Optional For    : podman
Conflicts With  : None
Replaces        : None
Installed Size  : 486.06 KiB
Packager        : David Runge <[email protected]>
Build Date      : Fri 05 Jul 2024 11:00:08 AM EEST
Install Date    : Thu 11 Jul 2024 09:50:35 AM EEST
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

[hmontone]

Behaviour in Docker Compose (as described in documentation) is what I expected.

[hmontone]

It seems that if I interpolate value in compose.yml it picks the value correctly from .env:

$ cat .env
TEST=test
$ cat compose.yml
services:
  test:
    image: busybox
    environment:
      - TEST=$TEST
    command: printenv
$ podman compose run --rm test
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
container=podman
TERM=xterm
TEST=test
HOME=/root
HOSTNAME=cb5574ddcaef

On the other hand if I set environment variable as local environment variable then it is replicated correctly:

$ cat .env
TEST=test
$ cat compose.yml
services:
  test:
    image: busybox
    environment:
      - TEST
    command: printenv
$ TEST=local podman compose run --rm test
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
container=podman
TERM=xterm
TEST=local
HOME=/root
HOSTNAME=481b5794f566

[legobeat]

Might be the same issue as:

• env_file ignored for environment #891

Related:

• Environment handling needs integration tests #970
• Multiple env files as arguments or environment variable #813
• Incorrect implementation of include #999

[terminatorbs]

not sure if i'm doing this correctly, but this issue has bothered me and i've implemented a fix that seems to work great in my (admittedly short) tests:

main...terminatorbs:podman-compose:patch-1

sorry i'm not knowledgeable to add tests for this. also no worries if the change doesn't make it into a PR, i just wanted a fix for myself because for example Jitsi has like hundreds of env vars passed like this and i thought i'd rather fix it here than write hacky scripts that modify compose files to add interpolation to every environment: entry.

edit: the code change is in the commit, but here it is with 2 comments added that kinda explain what it does:

    env = norm_as_list(cnt.get("environment", {}))
    for e in env:
        if '=' in e:
            # value is already assigned, directly add to podman_args
            podman_args.extend(["-e", e])
        else:
            # if we find a key in our env, pass it and it's value to the container
            if e in compose.environ:
                podman_args.extend(["-e", f"{e}={compose.environ[e]}"])

this is (as of right now) from line 1100 in podman-compose.py, in the container_to_args() function.

edit 2:
if the change seems reasonable and someone thinks it might be good to merge, i am willing to invest the time to make a test or two for it and open a proper PR.