docker-compose up --force-recreate fails with 'Network [..] needs to be recreated - option "isolate" has changed'

[vnznznz]

/kind bug

Description

Since updating podman to 4.2.0 I'm unable to use docker-compose up -d --force-recreate when the containers are already running

Steps to reproduce the issue:

1. Have minimal docker-compose.yml e.g.

version: "3.8"
services:
  redis:
    image: "redis:alpine"
    ports:
      - "127.0.0.1:6379:6379"

2. first run of docker-compose up -d --force-recreate as there are no containers running

3. second run of docker-compose up -d --force-recreate fails with

ERROR: Network "recreatebug_default" needs to be recreated - option "isolate" has changed

Describe the results you received:
Unable to force recreate running containers defined in a docker-compose file.

Describe the results you expected:

Able to force recreate running containers (worked with podman 4.0.2)

Additional notes

The network must have been created by podman 4.2.0 to have

 "Options": {
        "isolate": "true"
    },

Networks created without this options don't cause this error.

Output of docker-compose version:

docker-compose version 1.29.2, build unknown
docker-py version: 5.0.3
CPython version: 3.10.6
OpenSSL version: OpenSSL 3.0.5 5 Jul 2022

Output of podman version:

Client:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.4
Built:        Thu Aug 11 16:42:17 2022
OS/Arch:      linux/amd64

Output of podman info:

host:
  arch: amd64
  buildahVersion: 1.27.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 98.14
    systemPercent: 0.42
    userPercent: 1.44
  cpus: 20
  distribution:
    distribution: fedora
    variant: workstation
    version: "36"
  eventLogger: journald
  hostname: base
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.19.4-200.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 6905892864
  memTotal: 33489911808
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.5-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 3h 36m 51.00s (Approximately 0.12 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - localhost
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/vinz/.config/containers/storage.conf
  containerStore:
    number: 14
    paused: 0
    running: 12
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/vinz/.local/share/containers/storage
  graphRootAllocated: 998483427328
  graphRootUsed: 462067011584
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1137
  runRoot: /run/user/1000/containers
  volumePath: /home/vinz/.local/share/containers/storage/volumes
version:
  APIVersion: 4.2.0
  Built: 1660228937
  BuiltTime: Thu Aug 11 16:42:17 2022
  GitCommit: ""
  GoVersion: go1.18.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.2.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.2.0-2.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[vrothberg]

Thanks for reaching out, @vinzBad!

@flouthoc PTAL

[jebw]

I'm hitting this too

I can reliably trigger with docker-compose run .... - first invocation works, second one will always triggr the error

[Luap99]

We translate the docker options to podman options in the network create compat endpoint, to fix this we also need to translate the podman options to docker options before displaying them in the network inspect/list endpoint.

[ricardoinacio]

I am having this too. The temporary workaround I'm using is to run

docker-compose down --remove-orphans && docker-compose up -d

So it will forcibly remove the existing docker-compose containers and network.

[akostadinov]

Anybody looked at this?

Could it be selinux related? Running my compose, I saw a selinux issue, see

https://bugzilla.redhat.com/show_bug.cgi?id=2125878

[ghost]

@akostadinov even when selinux is in permissive mode, it still has this problem. I thought it might be selinux related too, but the selinux issues i ran into were unrelated.

[djnotes]

I am having a similar issue when I run docker-compose up for the second time i.e. docker-compose up has been run at least once before and then stopped for some reason. So, I have to run docker-compose down first and then docker-compose up.

[useri@fedora proj]$ docker-compose up
ERROR: Network "proj_default" needs to be recreated - option "isolate" has changed

[yajo]

The workaround here is using this file and obeying the comments:

# We assume this exists in /some/path/example/docker-compose.yaml
version: "3.8"
services:
  redis:
    image: "redis:alpine"
    ports:
      - "127.0.0.1:6379:6379"

networks:
  default:
    # Comment lines below if using for the 1st time
    external: true
    name: example_default

This way, docker-compose won't try to check the value of the isolate option. But the 1st time you run the file, you should comment those last 2 lines to let docker-compose create the network (or precreate it with podman network create ...).

[Luap99]

I will fix it sometime next week

[yajo]

I just tested with podman 4.2.1 and it is still failing. On which version is this fix gonna get released?

[Luap99]

4.3

[akostadinov]

Confirmed that this was fixed with 4.3.0 in fedora. Just installed from updates-testing

sudo dnf update --refresh --enablerepo=updates-testing podman

[talha110]

Confirmed that this was fixed with 4.3.0 in fedora. Just installed from updates-testing

sudo dnf update --refresh --enablerepo=updates-testing podman

what about RHEL/CentOS?

[rhatdan]

4.3.0 should be release ot RHEL and Centos with 12 weeks.

[tdgroot]

I was having the same problem with docker-compose 1.9.x and Podman 4.3.1, but can verify that upgrading from docker-compose 1.9.x to 2.14.x solved this problem for me.