Support Azure DevOps resource containers for builds

[jamjon3]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Azure DevOps on self-hosted agents (RHEL8/Podman) cannot load containers defined in 'resources'.

It fails on this workflow command in the "Initialize Containers" step

/usr/bin/docker info -f "{{range .Plugins.Network}}{{println .}}{{end}}"

This fails because there is no .Plugins.Network in podman info. The same on docker with docker info produces a section that is missing in podman info:

 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog

I'm using 'podman-docker' to look like 'docker' to the Azure DevOps agent.

Steps to reproduce the issue:

1. Add this section something like the following to azure-pipelines.yml

resources:
  containers:
  - container: win_rm
    image: nexus01.pd.pods.com:5443/pods-llc/swe/containers/base/devops_winrm_container:latest
    endpoint: pods_nexus_registry

2. Add in the 'jobs' section with a reference to the agent pool and container

    jobs:
      - job: BuildInsideContainer
        displayName: building inside a container
        pool: mypool
        container: win_rm

3. Run the pipeline (include some 'steps' of course). The "Initialize containers" step should pull the container and setup the 'steps' to run inside that specified container.

Describe the results you received:

Starting: Initialize containers
/usr/bin/docker version --format '{{.Server.APIVersion}}'
''3.2.3'
Docker daemon API version: ''3.2.3'
/usr/bin/docker version --format '{{.Client.APIVersion}}'
'3.2.3'
Docker client API version: '3.2.3'
/usr/bin/docker ps --all --quiet --no-trunc --filter "label=dc4b27"
/usr/bin/docker network prune --force --filter "label=dc4b27"
/usr/bin/docker login --username "***" --password-stdin ***
Login Succeeded!
/usr/bin/docker pull nexus01.pd.pods.com:5443/pods-llc/swe/containers/base/devops_winrm_container:latest
Trying to pull nexus01.pd.pods.com:5443/pods-llc/swe/containers/base/devops_winrm_container:latest...
Getting image source signatures
Copying blob sha256:d0badf5ab1aefb2806b494241481be6171425991987d6023c90074ea9404d6d8
Copying blob sha256:0a0b8f5ff20da9ce383904f041b750faee36dc6a258bc0100e7fcaa5d01b5101
Copying blob sha256:f9aac8178ace131a12b7f8e848ea3a2bb6b65eb6841946cf048588d161df2ff2
Copying blob sha256:fc725350b2637af7c79163bea9e3df54d78712803a01c97fb45333abd34807c0
Copying blob sha256:d251a2e2e8a37b8c79ad94dec69eaa86aeca6498ed8f3979e41005c309ddfa9a
Copying blob sha256:1b474f8e669eca50e71598ac473acae7d517247f94cee83b928c03bd53dc2ee0
Copying blob sha256:b77f066bf58c59e5edf0518c85b448a4c6b343b8b4e74c4fee6055a7942b01dd
Copying blob sha256:f443915232178bf37943f98091f60d27660b8fc7d29b31d741b92215bb87f930
Copying blob sha256:22a76ff78b8cd21fce32f3fa9e01428c7e94b795d0259564f5fa6eac6c02f163
Copying config sha256:606eda11e92bdc5b0e33f144415e9ff38c38aa12b0215c06fbeb91db85302a3c
Writing manifest to image destination
Storing signatures
606eda11e92bdc5b0e33f144415e9ff38c38aa12b0215c06fbeb91db85302a3c
/usr/bin/docker logout ***
Removed login credentials for nexus01.pd.pods.com:5443
/usr/bin/docker info -f "{{range .Plugins.Network}}{{println .}}{{end}}"
Error: template: info:1:16: executing "info" at <.Plugins.Network>: can't evaluate field Plugins in type *define.Info
##[error]Exit code 125 returned from process: file name '/usr/bin/docker', arguments 'info -f "{{range .Plugins.Network}}{{println .}}{{end}}"'.
Finishing: Initialize containers

Describe the results you expected:

With regular docker, docker info works fine:

Login Succeeded
/bin/docker pull nexus01.pd.pods.com:5443/pods-llc/swe/containers/base/devops_winrm_container:latest
latest: Pulling from pods-llc/swe/containers/base/devops_winrm_container
Digest: sha256:f11bfc11854f07c6f23b10de23fdb483c4b09e42fc7d522006ee8ea5e64cd210
Status: Image is up to date for nexus01.pd.pods.com:5443/pods-llc/swe/containers/base/devops_winrm_container:latest
nexus01.pd.pods.com:5443/pods-llc/swe/containers/base/devops_winrm_container:latest
/bin/docker logout ***
Removing login credentials for nexus01.pd.pods.com:5443
/bin/docker info -f "{{range .Plugins.Network}}{{println .}}{{end}}"
bridge
host
ipvlan
macvlan
null
overlay
/bin/docker network create --label 9f0d2f vsts_network_bf2a782037e240bbb012f1d5decbb5a6
7eaf4da58f61ff6a995593ed64f1e2b7d349eff90afed21bf86f74f5ba2b18dd
/bin/docker inspect --format="{{index .Config.Labels \"com.azure.dev.pipelines.agent.handler.node.path\"}}" nexus01.pd.pods.com:5443/pods-llc/swe/containers/base/devops_winrm_container:latest
/bin/docker create --name win_rm_nexus01pdpodscom5443podsllcswecontainersbasedevops_winrm_containerrelease130_f6c662 --label 9f0d2f --network vsts_network_bf2a782037e240bbb012f1d5decbb5a6  -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/usr/local/share/vsts-agent/61":"/__a/61" -v "/usr/local/share/vsts-agent/_temp":"/__a/_temp" -v "/usr/local/share/vsts-agent/_tasks":"/__a/_tasks" -v "/usr/local/share/vsts-agent/_tool":"/__t" -v "/usr/local/share/vsts-agent/externals":"/__a/externals":ro -v "/usr/local/share/vsts-agent/.taskkey":"/__a/.taskkey" nexus01.pd.pods.com:5443/pods-llc/swe/containers/base/devops_winrm_container:latest "/__a/externals/node/bin/node" -e "setInterval(function(){}, 24 * 60 * 60 * 1000);"
546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32
/bin/docker start 546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32
546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32
/bin/docker ps --all --filter id=546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32 --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32 Up Less than a second
/bin/docker exec  546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32 sh -c "command -v bash"
/usr/bin/bash
whoami 
DevOps1
id -u DevOps1
1000
Try to create a user with UID '1000' inside the container.
/bin/docker exec  546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32 bash -c "getent passwd 1000 | cut -d: -f1 "
/bin/docker exec  546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32 useradd -m -u 1000 DevOps1_azpcontainer
Grant user 'DevOps1_azpcontainer' SUDO privilege and allow it run any command without authentication.
/bin/docker exec  546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32 groupadd azure_pipelines_sudo
/bin/docker exec  546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32 usermod -a -G azure_pipelines_sudo DevOps1_azpcontainer
/bin/docker exec  546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32 su -c "echo '%azure_pipelines_sudo ALL=(ALL:ALL) NOPASSWD:ALL' >> /etc/sudoers"
Allow user 'DevOps1_azpcontainer' run any docker command without SUDO.
stat -c %g /var/run/docker.sock
992
/bin/docker exec  546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32 bash -c "cat /etc/group"
/bin/docker exec  546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32 groupadd -g 992 azure_pipelines_docker
/bin/docker exec  546fef1066d5156b1c7777129bfa3380704661f8a07fbf66e2aa74f99e5c6e32 usermod -a -G azure_pipelines_docker DevOps1_azpcontainer
Finishing: Initialize containers

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.2.3
API Version:  3.2.3
Go Version:   go1.15.7
Built:        Tue Jul 27 07:29:39 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.3
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-1.module+el8.4.0+11822+6cc1e7d7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: ae467a0c8001179d4d0adf4ada381108a893d7ec'
  cpus: 8
  distribution:
    distribution: '"rhel"'
    version: "8.4"
  eventLogger: file
  hostname: tpapdvlcibld06
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-305.12.1.el8_4.x86_64
  linkmode: dynamic
  memFree: 15081103360
  memTotal: 16600363008
  ociRuntime:
    name: runc
    package: runc-1.0.0-74.rc95.module+el8.4.0+11822+6cc1e7d7.x86_64
    path: /usr/bin/runc
    version: |-
      runc version spec: 1.0.2-dev
      go: go1.15.13
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 34355539968
  swapTotal: 34355539968
  uptime: 4h 39m 4.18s (Approximately 0.17 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.2.3
  Built: 1627370979
  BuiltTime: Tue Jul 27 07:29:39 2021
  GitCommit: ""
  GoVersion: go1.15.7
  OsArch: linux/amd64
  Version: 3.2.3

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No and yes. I'm using the distribution package for RHEL8 but I have checked the Podman Troubleshooting Guide.

Additional environment details (AWS, VirtualBox, physical, etc.):

VMWare virtual machine

[rhatdan]

@Luap99 PTAL

[Luap99]

Adding this field should be simple but I am wondering how to query the supported drivers.
For log we support journald, k8s-file and none.
Network should be bridge and macvlan but I think we should add this to new network Interface at some point.
@mheon I assume it is also possible to get a list of configured volume plugins?

[flouthoc]

@Luap99 volume plugins are masked under field driver afaik. Users manually configure plugins usually it defaults to local everywhere.

[mheon]

@Luap99 They're just stored in containers.conf - https://github.com/containers/common/blob/main/pkg/config/config.go#L396

The map is name of plugin (what you want) to path of plugin socket - so iterating through and taking all the names should be sufficient.

[jamjon3]

Thank you all and I'll watch the PR discussion and stay out of the way but I'm still active and watching the conversations.