Healthcheck is ignored in compose files when adding labels to volumes

[dcermak]

/kind bug

Description

It appears that podman is ignoring the healtcheck configuration in a compose file if you add a :Z or a :z label to a volume.

Steps to reproduce the issue:

1. git clone https://github.com/os-autoinst/openQA.git && cd container/webui && sed -i 's|./workdir/db:/var/lib/postgresql/data|./workdir/db:/var/lib/postgresql/data:z|' docker-compose.yaml

2. docker-compose up

Describe the results you received:

$ docker-compose up
Creating webui_db_1            ... done
Creating webui_webui_db_init_1 ... done

ERROR: for webui  Service "db" is missing a healthcheck configuration
ERROR: Encountered errors while bringing up the project.

Note that not adding the :Z label is not a solution, because it only makes the docker compose fail later (and the db container is actually not working at that point as it can't write ./workdir/db/):

Creating webui_webui_db_init_1 ... done
Creating webui_db_1            ... done
Creating webui_webui_1         ... done
Creating webui_webui_2         ... done

ERROR: for scheduler  Service "webui" is missing a healthcheck configuration

ERROR: for livehandler  Service "webui" is missing a healthcheck configuration

ERROR: for gru  Service "webui" is missing a healthcheck configuration

ERROR: for websockets  Service "webui" is missing a healthcheck configuration
ERROR: Encountered errors while bringing up the project.

Describe the results you expected:

The compose should succeed.
Additional information you deem important (e.g. issue happens only occasionally):

• This is always reproducible.
• tried with both docker-compose 1.28.6 from Fedora's repositories and the latest release 1.29.2 installed via pip.

Output of podman version:

Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.16.3
Built:        Wed May 12 21:27:59 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.20.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.27-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 4
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: Cerberus
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    - container_id: 65537
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    - container_id: 65537
      host_id: 100000
      size: 65536
  kernel: 5.12.5-300.fc34.x86_64
  linkmode: dynamic
  memFree: 198840320
  memTotal: 8259194880
  ociRuntime:
    name: crun
    package: crun-0.19.1-2.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1
      commit: 1535fedf0b83fb898d449f9680000f729ba719f5
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 14715969536
  swapTotal: 16676544512
  uptime: 9h 52m 50.38s (Approximately 0.38 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/dan/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.3
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/dan/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 120
  runRoot: /run/user/1000/containers
  volumePath: /home/dan/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.2
  Built: 1620847679
  BuiltTime: Wed May 12 21:27:59 2021
  GitCommit: ""
  GoVersion: go1.16.3
  OsArch: linux/amd64
  Version: 3.1.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.1.2-3.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

I have followed this guide to setup podman with compose: https://www.redhat.com/sysadmin/podman-docker-compose

[rhatdan]

I think the SELinux labels is a red herring. Could you not add the :z and just run in permissive mode?

I think one of the issues would be if you are building the image on the fly, are we building in Docker format or OCI. OCI Images to not support healthchecks.

[dcermak]

I think the SELinux labels is a red herring. Could you not add the :z and just run in permissive mode?

I don't even have to change the selinux config to permissive: when I remove the :z flag then this happens:

Creating webui_webui_db_init_1 ... done
Creating webui_db_1            ... done
Creating webui_webui_1         ... done
Creating webui_webui_2         ... done

ERROR: for scheduler  Service "webui" is missing a healthcheck configuration

ERROR: for livehandler  Service "webui" is missing a healthcheck configuration

ERROR: for gru  Service "webui" is missing a healthcheck configuration

ERROR: for websockets  Service "webui" is missing a healthcheck configuration
ERROR: Encountered errors while bringing up the project.

I think one of the issues would be if you are building the image on the fly, are we building in Docker format or OCI. OCI Images to not support healthchecks.

The images get build by docker-compose on the fly. I assume that podman then uses the default (iirc OCI).

[vrothberg]

Sounds more like something in the compat REST api to me. I skimmed the docker-compose code quickly and the error is raised if 'State.Health.Status' is None in the container's inspect data.

[vrothberg]

Yes, I can reproduce.

# Run a container with a healthcheck.
$ podman-remote run --name 123 -d quay.io/libpod/alpine_healthcheck
# Inspect the remote container.
$ podman-remote inspect --format "{{ .State.Healthcheck.Status}} " 123
healthy       
# Now inspect the container via the compat API.                                                                                       
$ curl -XGET --unix-socket /run/user/1000/podman/podman.sock http:/v1.40/containers/123/json|jq .State 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                 
                                 Dload  Upload   Total   Spent    Left  Speed                                                   
100  4381    0  4381    0     0   713k      0 --:--:-- --:--:-- --:--:--  713k                                                  
{                                                                                                                               
  "Status": "running",                                                                                                          
  "Running": true,                                                                                                              
  "Paused": false,                                                                                                              
  "Restarting": false,                                                                                                          
  "OOMKilled": false,                                                                                                           
  "Dead": false,                                                                                                                
  "Pid": 525342,                                                                                                                
  "ExitCode": 0,                                                                                                                
  "Error": "",                                                                                                                  
  "StartedAt": "2021-06-09T11:37:13.618363453Z",                                                                                
  "FinishedAt": "0001-01-01T00:00:00Z"                                                                                          
}                                                                                                                               

The compat endpoint does not include the health check data. @cdoern, do you have cycles to tackle it?

Cc @jwhonce

[cdoern]

@vrothberg i'll fix the compat endpoint this morning, I'll take a look now!

[vrothberg]

Awesome, thank you, @cdoern.