Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

APIv2: missing NetworkSettings and failure to start a container #5580

Closed
max-arnold opened this issue Mar 21, 2020 · 55 comments
Closed

APIv2: missing NetworkSettings and failure to start a container #5580

max-arnold opened this issue Mar 21, 2020 · 55 comments
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@max-arnold
Copy link

max-arnold commented Mar 21, 2020

/kind bug

Iteration 2 of my APIv2 & docker-py compatibility tests (see #5553 for the first one).

1. I still had to patch the expected minimal version (from 1.9.0 to 1.8.0).

However, problems (2) and (3) from #5553 are fixed!

2. Attempt to run a container failed due to Salt code that expects NetworkSettings -> Networks to be a non-empty dictionary

Network settings:

Docker
sudo docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
0b0f5edd3b6b        bridge              bridge              local
abf694a88ec0        host                host                local
62f65a4a0015        none                null                local

docker network inspect 0b0f5edd3b6b
[
    {
        "Name": "bridge",
        "Id": "0b0f5edd3b6b1047f4bbabe5a037f2faccf7ff2d2aa35bc5a1374d81c1def154",
        "Created": "2020-03-21T05:49:03.575468367-07:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]
Podman
sudo podman network ls
NAME     VERSION   PLUGINS
podman   0.4.0     bridge,portmap,firewall,tuning

sudo podman network inspect podman
[
	{
		"cniVersion": "0.4.0",
		"name": "podman",
		"plugins": [
			{
				"bridge": "cni-podman0",
				"ipMasq": true,
				"ipam": {
					"ranges": [
						[
							{
								"gateway": "10.88.0.1",
								"subnet": "10.88.0.0/16"
							}
						]
					],
					"routes": [
						{
							"dst": "0.0.0.0/0"
						}
					],
					"type": "host-local"
				},
				"isGateway": true,
				"type": "bridge"
			},
			{
				"capabilities": {
					"portMappings": true
				},
				"type": "portmap"
			},
			{
				"type": "firewall"
			},
			{
				"type": "tuning"
			}
		]
	}
]

The data structures look quite different.

Below is the difference between API responses from Docker (19.03.8) and Podman (note the <-- comments):

Docker
REQ http+docker://localhost/v1.40/containers/create?name=ubuntu
{
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "AttachStdin": false,
    "AttachStdout": true,
    "AttachStderr": true,
    "Image": "ubuntu:18.04",
    "NetworkDisabled": false,
    "HostConfig": {
        "NetworkMode": "default"
    }
}
RES {"Id":"59ff788b92417444a765d34d9b6352252d4e1a71cebe0a7c0d9d94e7df0cbd8f","Warnings":[]}

REQ http+docker://localhost/v1.40/containers/json?limit=-1&all=1&size=0&trunc_cmd=0
RES [
    {
        "Id": "59ff788b92417444a765d34d9b6352252d4e1a71cebe0a7c0d9d94e7df0cbd8f",
        "Names": [
            "/ubuntu"
        ],
        "Image": "ubuntu:18.04",
        "ImageID": "sha256:4e5021d210f65ebe915670c7089120120bc0a303b90208592851708c1b8c04bd",
        "Command": "/bin/bash",
        "Created": 1584795153,
        "Ports": [],
        "Labels": {
        },
        "State": "created",
        "Status": "Created",
        "HostConfig": {
            "NetworkMode": "default"
        },
        "NetworkSettings": {               <-- Not empty
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        },
        "Mounts": []
    }
]


REQ http+docker://localhost/v1.40/containers/59ff788b92417444a765d34d9b6352252d4e1a71cebe0a7c0d9d94e7df0cbd8f/json
RES {
    "Id": "59ff788b92417444a765d34d9b6352252d4e1a71cebe0a7c0d9d94e7df0cbd8f",
    "Created": "2020-03-21T12:52:33.720720901Z",
    "Path": "/bin/bash",
    "Args": [],
    "State": {
        "Status": "created",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "0001-01-01T00:00:00Z",
        "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "sha256:4e5021d210f65ebe915670c7089120120bc0a303b90208592851708c1b8c04bd",
    "ResolvConfPath": "",
    "HostnamePath": "",
    "HostsPath": "",
    "LogPath": "",
    "Name": "/ubuntu",
    "RestartCount": 0,
    "Driver": "overlay2",
    "Platform": "linux",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": null,
        "ContainerIDFile": "",
        "LogConfig": {
            "Type": "json-file",
            "Config": {
            }
        },
        "NetworkMode": "default",
        "PortBindings": null,
        "RestartPolicy": {
            "Name": "",
            "MaximumRetryCount": 0
        },
        "AutoRemove": false,
        "VolumeDriver": "",
        "VolumesFrom": null,
        "CapAdd": null,
        "CapDrop": null,
        "Capabilities": null,
        "Dns": null,
        "DnsOptions": null,
        "DnsSearch": null,
        "ExtraHosts": null,
        "GroupAdd": null,
        "IpcMode": "private",
        "Cgroup": "",
        "Links": null,
        "OomScoreAdj": 0,
        "PidMode": "",
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyRootfs": false,
        "SecurityOpt": null,
        "UTSMode": "",
        "UsernsMode": "",
        "ShmSize": 67108864,
        "Runtime": "runc",
        "ConsoleSize": [
            0,
            0
        ],
        "Isolation": "",
        "CpuShares": 0,
        "Memory": 0,
        "NanoCpus": 0,
        "CgroupParent": "",
        "BlkioWeight": 0,
        "BlkioWeightDevice": null,
        "BlkioDeviceReadBps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteIOps": null,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "Devices": null,
        "DeviceCgroupRules": null,
        "DeviceRequests": null,
        "KernelMemory": 0,
        "KernelMemoryTCP": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "MemorySwappiness": null,
        "OomKillDisable": false,
        "PidsLimit": null,
        "Ulimits": null,
        "CpuCount": 0,
        "CpuPercent": 0,
        "IOMaximumIOps": 0,
        "IOMaximumBandwidth": 0,
        "MaskedPaths": [
            "/proc/asound",
            "/proc/acpi",
            "/proc/kcore",
            "/proc/keys",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/proc/scsi",
            "/sys/firmware"
        ],
        "ReadonlyPaths": [
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ]
    },
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/60e2b374a35c4bc6f26d8ab6d0de24efe0247cacb8fa357084d5d841ab60fca8-init/diff:/var/lib/docker/overlay2/5164a85b327d3504c94ff8c459ad5c65a9da0b430b948ebe788251c1002dbbc7/diff:/var/lib/docker/overlay2/4587dab6541269a14050d4ca804b710c70e4f70be4db3d53cd08cc0719169c86/diff:/var/lib/docker/overlay2/65ecf6089d50f2ef086fd4daa6c0ea2e73fcfdbdea83ad8dc9866992d64ec6ee/diff:/var/lib/docker/overlay2/dcc9eb484566cb1d83f0cca6b78df0bc30b125b7886410cabb4f6f8df348a8af/diff",
            "MergedDir": "/var/lib/docker/overlay2/60e2b374a35c4bc6f26d8ab6d0de24efe0247cacb8fa357084d5d841ab60fca8/merged",
            "UpperDir": "/var/lib/docker/overlay2/60e2b374a35c4bc6f26d8ab6d0de24efe0247cacb8fa357084d5d841ab60fca8/diff",
            "WorkDir": "/var/lib/docker/overlay2/60e2b374a35c4bc6f26d8ab6d0de24efe0247cacb8fa357084d5d841ab60fca8/work"
        },
        "Name": "overlay2"
    },
    "Mounts": [],
    "Config": {
        "Hostname": "59ff788b9241",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": true,
        "AttachStderr": true,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Cmd": [
            "/bin/bash"
        ],
        "Image": "ubuntu:18.04",
        "Volumes": null,
        "WorkingDir": "",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": {
        }
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {
        },
        "SandboxKey": "",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {               <-- Not empty
            "bridge": {
                "IPAMConfig": null,
                "Links": null,
                "Aliases": null,
                "NetworkID": "",
                "EndpointID": "",
                "Gateway": "",
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "",
                "DriverOpts": null
            }
        }
    }
}

Podman
REQ http+docker://localhost/v1.40/containers/create?name=ubuntu
{
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "AttachStdin": false,
    "AttachStdout": true,
    "AttachStderr": true,
    "Image": "ubuntu:18.04",
    "NetworkDisabled": false,
    "HostConfig": {
        "NetworkMode": "default"
    }
}
RES {"Id":"8b826a876b72af7fe58ffbeea7720989fcc7d004422a794ccd911eaefe9d0c3c","Warnings":[]}

REQ http+docker://localhost/v1.40/containers/json?limit=-1&all=1&size=0&trunc_cmd=0
RES [
    {
        "Id": "8b826a876b72af7fe58ffbeea7720989fcc7d004422a794ccd911eaefe9d0c3c",
        "Names": [
            "/ubuntu"
        ],
        "Image": "ubuntu:18.04",
        "ImageID": "4e5021d210f65ebe915670c7089120120bc0a303b90208592851708c1b8c04bd",
        "Command": "/bin/bash",
        "Created": 1584789004,
        "Ports": null,
        "Labels": {
        },
        "State": "created",
        "Status": "",
        "NetworkSettings": null,      <-- EMPTY
        "Mounts": null,
        "Name": "",
        "Config": null,
        "NetworkingConfig": null,
        "AdjustCPUShares": false
    }
]

REQ http+docker://localhost/v1.40/containers/190aa780e68a9c1ed8cb25d3f4da05d71dc0f558f7da61c00ff97d0bf7e413f5/json
RES {
    "Id": "190aa780e68a9c1ed8cb25d3f4da05d71dc0f558f7da61c00ff97d0bf7e413f5",
    "Created": "2020-03-21 06:47:05.766619228 -0700 PDT",
    "Path": "",
    "Args": null,
    "State": {
        "Status": "configured",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "0001-01-01T00:00:00Z",
        "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "ubuntu:18.04",
    "ResolvConfPath": "",
    "HostnamePath": "",
    "HostsPath": "",
    "LogPath": "/var/lib/containers/storage/overlay-containers/190aa780e68a9c1ed8cb25d3f4da05d71dc0f558f7da61c00ff97d0bf7e413f5/userdata/ctr.log",
    "Name": "/ubuntu",
    "RestartCount": 0,
    "Driver": "overlay",
    "Platform": "linux",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "ExecIDs": [],
    "HostConfig": {
        "Binds": [],
        "ContainerIDFile": "",
        "LogConfig": {
            "Type": "",
            "Config": null
        },
        "NetworkMode": "default",
        "PortBindings": {
        },
        "RestartPolicy": {
            "Name": "",
            "MaximumRetryCount": 0
        },
        "AutoRemove": false,
        "VolumeDriver": "",
        "VolumesFrom": null,
        "CapAdd": [],
        "CapDrop": [],
        "Capabilities": null,
        "CgroupnsMode": "",
        "Dns": [],
        "DnsOptions": [],
        "DnsSearch": [],
        "ExtraHosts": [],
        "GroupAdd": [],
        "IpcMode": "",
        "Cgroup": "",
        "Links": null,
        "OomScoreAdj": 0,
        "PidMode": "",
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyRootfs": false,
        "SecurityOpt": [],
        "UTSMode": "",
        "UsernsMode": "",
        "ShmSize": 0,
        "Runtime": "oci",
        "ConsoleSize": [
            0,
            0
        ],
        "Isolation": "",
        "CpuShares": 1024,
        "Memory": 0,
        "NanoCpus": 0,
        "CgroupParent": "",
        "BlkioWeight": 0,
        "BlkioWeightDevice": null,
        "BlkioDeviceReadBps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteIOps": null,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "Devices": [],
        "DeviceCgroupRules": null,
        "DeviceRequests": null,
        "KernelMemory": 0,
        "KernelMemoryTCP": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "MemorySwappiness": 0,
        "OomKillDisable": false,
        "PidsLimit": 0,
        "Ulimits": [
            {
                "Name": "RLIMIT_NOFILE",
                "Hard": 1048576,
                "Soft": 1048576
            },
            {
                "Name": "RLIMIT_NPROC",
                "Hard": 1048576,
                "Soft": 1048576
            }
        ],
        "CpuCount": 0,
        "CpuPercent": 0,
        "IOMaximumIOps": 0,
        "IOMaximumBandwidth": 0,
        "MaskedPaths": null,
        "ReadonlyPaths": null
    },
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/containers/storage/overlay/0b21f9cedf302a51263f78af136fcbdd1ed4ff9a909191bd3615d29a3de2bab7/diff:/var/lib/containers/storage/overlay/3c1fcef44107cd6175b87987b91eea89bf7eb2f01c3221da322b787f95c0808a/diff:/var/lib/containers/storage/overlay/c9d6762179607ce7078ee988fe4d5280771e5cf36324cfa8481630ad9808e4bf/diff:/var/lib/containers/storage/overlay/c8be1b8f4d60d99c281fc2db75e0f56df42a83ad2f0b091621ce19357e19d853/diff",
            "UpperDir": "/var/lib/containers/storage/overlay/651a258c669fcc542c498a3847ef995234cfc866e66e01b3fd9722702d896282/diff",
            "WorkDir": "/var/lib/containers/storage/overlay/651a258c669fcc542c498a3847ef995234cfc866e66e01b3fd9722702d896282/work"
        },
        "Name": "overlay"
    },
    "SizeRootFs": 0,
    "Mounts": [],
    "Config": {
        "Hostname": "190aa780e68a",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "TERM=xterm",
            "HOSTNAME="
        ],
        "Cmd": [
            "/bin/bash"
        ],
        "Image": "ubuntu:18.04",
        "Volumes": null,
        "WorkingDir": "/",
        "Entrypoint": [],
        "OnBuild": null,
        "Labels": {
        },
        "StopSignal": "\u000f",
        "StopTimeout": 10
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": null,
        "SandboxKey": "",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": null           <-- EMPTY 
    }
}

3. And finally (after I patched Salt to not crash on (2)), the container fails to start with OCI runtime error

REQ http+docker://localhost/v1.40/containers/ubuntu/start
RES {"cause":"OCI runtime error","message":"args must not be empty: OCI runtime error","response":500}
Docker
REQ http+docker://localhost/v1.40/containers/ubuntu/start
RES
REQ http+docker://localhost/v1.40/containers/ubuntu/json
RES {
    "Id": "59ff788b92417444a765d34d9b6352252d4e1a71cebe0a7c0d9d94e7df0cbd8f",
    "Created": "2020-03-21T12:52:33.720720901Z",
    "Path": "/bin/bash",
    "Args": [],
    "State": {
        "Status": "running",
        "Running": true,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 27755,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2020-03-21T12:52:35.747352749Z",
        "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "sha256:4e5021d210f65ebe915670c7089120120bc0a303b90208592851708c1b8c04bd",
    "ResolvConfPath": "/var/lib/docker/containers/59ff788b92417444a765d34d9b6352252d4e1a71cebe0a7c0d9d94e7df0cbd8f/resolv.conf",
    "HostnamePath": "/var/lib/docker/containers/59ff788b92417444a765d34d9b6352252d4e1a71cebe0a7c0d9d94e7df0cbd8f/hostname",
    "HostsPath": "/var/lib/docker/containers/59ff788b92417444a765d34d9b6352252d4e1a71cebe0a7c0d9d94e7df0cbd8f/hosts",
    "LogPath": "/var/lib/docker/containers/59ff788b92417444a765d34d9b6352252d4e1a71cebe0a7c0d9d94e7df0cbd8f/59ff788b92417444a765d34d9b6352252d4e1a71cebe0a7c0d9d94e7df0cbd8f-json.log",
    "Name": "/ubuntu",
    "RestartCount": 0,
    "Driver": "overlay2",
    "Platform": "linux",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "docker-default",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": null,
        "ContainerIDFile": "",
        "LogConfig": {
            "Type": "json-file",
            "Config": {
            }
        },
        "NetworkMode": "default",
        "PortBindings": null,
        "RestartPolicy": {
            "Name": "",
            "MaximumRetryCount": 0
        },
        "AutoRemove": false,
        "VolumeDriver": "",
        "VolumesFrom": null,
        "CapAdd": null,
        "CapDrop": null,
        "Capabilities": null,
        "Dns": null,
        "DnsOptions": null,
        "DnsSearch": null,
        "ExtraHosts": null,
        "GroupAdd": null,
        "IpcMode": "private",
        "Cgroup": "",
        "Links": null,
        "OomScoreAdj": 0,
        "PidMode": "",
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyRootfs": false,
        "SecurityOpt": null,
        "UTSMode": "",
        "UsernsMode": "",
        "ShmSize": 67108864,
        "Runtime": "runc",
        "ConsoleSize": [
            0,
            0
        ],
        "Isolation": "",
        "CpuShares": 0,
        "Memory": 0,
        "NanoCpus": 0,
        "CgroupParent": "",
        "BlkioWeight": 0,
        "BlkioWeightDevice": null,
        "BlkioDeviceReadBps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteIOps": null,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "Devices": null,
        "DeviceCgroupRules": null,
        "DeviceRequests": null,
        "KernelMemory": 0,
        "KernelMemoryTCP": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "MemorySwappiness": null,
        "OomKillDisable": false,
        "PidsLimit": null,
        "Ulimits": null,
        "CpuCount": 0,
        "CpuPercent": 0,
        "IOMaximumIOps": 0,
        "IOMaximumBandwidth": 0,
        "MaskedPaths": [
            "/proc/asound",
            "/proc/acpi",
            "/proc/kcore",
            "/proc/keys",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/proc/scsi",
            "/sys/firmware"
        ],
        "ReadonlyPaths": [
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ]
    },
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/60e2b374a35c4bc6f26d8ab6d0de24efe0247cacb8fa357084d5d841ab60fca8-init/diff:/var/lib/docker/overlay2/5164a85b327d3504c94ff8c459ad5c65a9da0b430b948ebe788251c1002dbbc7/diff:/var/lib/docker/overlay2/4587dab6541269a14050d4ca804b710c70e4f70be4db3d53cd08cc0719169c86/diff:/var/lib/docker/overlay2/65ecf6089d50f2ef086fd4daa6c0ea2e73fcfdbdea83ad8dc9866992d64ec6ee/diff:/var/lib/docker/overlay2/dcc9eb484566cb1d83f0cca6b78df0bc30b125b7886410cabb4f6f8df348a8af/diff",
            "MergedDir": "/var/lib/docker/overlay2/60e2b374a35c4bc6f26d8ab6d0de24efe0247cacb8fa357084d5d841ab60fca8/merged",
            "UpperDir": "/var/lib/docker/overlay2/60e2b374a35c4bc6f26d8ab6d0de24efe0247cacb8fa357084d5d841ab60fca8/diff",
            "WorkDir": "/var/lib/docker/overlay2/60e2b374a35c4bc6f26d8ab6d0de24efe0247cacb8fa357084d5d841ab60fca8/work"
        },
        "Name": "overlay2"
    },
    "Mounts": [],
    "Config": {
        "Hostname": "59ff788b9241",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": true,
        "AttachStderr": true,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Cmd": [
            "/bin/bash"
        ],
        "Image": "ubuntu:18.04",
        "Volumes": null,
        "WorkingDir": "",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": {
        }
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "75d19db2a37d0f52339e80cbae3519fc7369414f4ba5edab5f969f02f1d6d7f1",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {
        },
        "SandboxKey": "/var/run/docker/netns/75d19db2a37d",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "39245ba9b166c0e74df4e11e6acb5c5e4f37dfee069e9dffe2aea9bd9f66b184",
        "Gateway": "172.17.0.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "172.17.0.2",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "MacAddress": "02:42:ac:11:00:02",
        "Networks": {
            "bridge": {
                "IPAMConfig": null,
                "Links": null,
                "Aliases": null,
                "NetworkID": "0b0f5edd3b6b1047f4bbabe5a037f2faccf7ff2d2aa35bc5a1374d81c1def154",
                "EndpointID": "39245ba9b166c0e74df4e11e6acb5c5e4f37dfee069e9dffe2aea9bd9f66b184",
                "Gateway": "172.17.0.1",
                "IPAddress": "172.17.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:42:ac:11:00:02",
                "DriverOpts": null
            }
        }
    }
}

Output of podman version:

Version:            1.8.2
RemoteAPI Version:  1
Go Version:         go1.10.1
OS/Arch:            linux/amd64

Output of podman info --debug:

podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.10.1
  podman version: 1.8.2
host:
  BuildahVersion: 1.14.3
  CgroupVersion: v1
  Conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.14, commit: '
  Distribution:
    distribution: ubuntu
    version: "18.04"
  MemFree: 101597184
  MemTotal: 499093504
  OCIRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  SwapFree: 926937088
  SwapTotal: 1027600384
  arch: amd64
  cpus: 1
  eventlogger: journald
  hostname: minion1
  kernel: 4.15.0-72-generic
  os: linux
  rootless: false
  uptime: 3h 46m 56.31s (Approximately 0.12 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes
@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 21, 2020
@max-arnold
Copy link
Author

max-arnold commented Apr 12, 2020

Actually we do not care about the output compatibility with docker network ls. There are three main problems:

  1. Docker version number. I can fix it on Salt side, but I got the impression that you guys have a patch for it (you mentioned it in the previous issue: APIv2 incompatibilities with docker-py #5553 (comment)). Please tell me if I'm wrong and I'll remove the corresponding note from this issue.

  2. NetworkSettings->Networks APIv2 response is empty. This data structure is heavily used by Salt to set up a container network (it compares a lot of nested keys to ensure the network state). This is hard to fix on Salt side without losing Docker compatibility.

  3. And finally, the container fails to start with OCI runtime error. I tried to dig into the podman source code to understand why, but unfortunately this area is beyond my current skills.

Overall (with a couple of workarounds), Salt is able to successfully pull the container but can't start it yet

If you need an easier way to reproduce the problems, I can write a simple Python script (it is hard to do using curl due to dynamic container IDs)

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@max-arnold
Copy link
Author

Up

@rhatdan
Copy link
Member

rhatdan commented May 13, 2020

I think you are indicating that this is still an issue.

@max-arnold
Copy link
Author

Yes, just want to make sure that the issue won’t be closed automatically due to inactivity

@mheon
Copy link
Member

mheon commented May 13, 2020

We were discussing this a bit yesterday among the team. It's definitely on our radar, and will probably come up sooner rather than later as part of our work on the Podman HTTP API.

@rhatdan
Copy link
Member

rhatdan commented May 13, 2020

@max-arnold BTW, we currently do not close issues automatically. The reminder is just a kick in the butt for us to relook at the issue.

@rhatdan
Copy link
Member

rhatdan commented Jun 9, 2020

@max-arnold Could you try this against master to tell us if it still exists.

@max-arnold
Copy link
Author

Thanks for the update! Unfortunately, due to personal reasons, I’ll be able to check this only in 2-4 weeks (I’m not at home and have no laptop with me).

I’ll get back to you on that as soon as I can.

@rhatdan
Copy link
Member

rhatdan commented Jun 10, 2020

@ashley-cui Mind taking a look to see if this issue still exists?

@max-arnold
Copy link
Author

@rhatdan Are the 2.0.0 packages for Ubuntu delayed? I only see 1.9.3 here https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/

@rhatdan
Copy link
Member

rhatdan commented Jun 22, 2020

Not that I know of, but these are handled manually by @lsm5 I believe.

@lsm5
Copy link
Member

lsm5 commented Jun 22, 2020

@rhatdan Are the 2.0.0 packages for Ubuntu delayed? I only see 1.9.3 here https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/

We had disabled autobuilds for podman until v2.0.0. I'm in the middle of building them now and I'll enable autobuilds after that. Expect to be done before noon.

@lsm5
Copy link
Member

lsm5 commented Jun 22, 2020

@max-arnold should be available now.

@max-arnold
Copy link
Author

max-arnold commented Jun 29, 2020

I tried the 2.0.0 release briefly and it doesn't work with SaltStack due to an incorrect date format: #6796

On the positive side, I no longer need to patch salt for minimum expected Docker version (1.9.0), since Podman is 2.0.0

I'll try again once that issue is fixed.

@zhangguanzhang
Copy link
Collaborator

#6835
this will fix the error which:

args must not be empty

If this be merged, maybe you could try start containers to test

@github-actions
Copy link

github-actions bot commented Aug 2, 2020

A friendly reminder that this issue had no activity for 30 days.

@zhangguanzhang
Copy link
Collaborator

@ashley-cui Mind taking a look to see if this issue still exists?

@max-arnold
Copy link
Author

I’ll check it

@max-arnold
Copy link
Author

I still can't get past the incorrect date format issue.

Feel free to ping me directly once it is fixed and the updated package is avaliable here: http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/

@max-arnold
Copy link
Author

max-arnold commented Aug 9, 2020

I managed to compile podman 2.1.0-dev (libpod 3173a18, conmon 5a6b2aca34915436e71cb200d6ef01844cbd66c9, runc a340fa9b561432b14c0dabcb5b53c834a15a87ce, containernetworking/plugins bd589992fbe0740418fcc7df7572bd98f4d759c0).

Below are the results I get when creating a container in both Docker and Podman.

Docker
% sudo curl --unix-socket /run/docker.sock -H "Content-Type: application/json" --data '{"Tty": false, "OpenStdin": false, "StdinOnce": false, "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Cmd": ["bash"], "Image": "ubuntu:18.04", "NetworkDisabled": false, "HostConfig": {"NetworkMode": "default", "AutoRemove": true}}' 'http://d/v1.40/containers/create?name=ubuntu'

{"Id":"8ee46ed5032c9ebbddfdca56521c4101b5d4d27ddf8b5a468533066e6f318487","Warnings":[]}

% sudo curl --unix-socket /run/docker.sock http://d/v1.35/containers/8ee46ed5032c9ebbddfdca56521c4101b5d4d27ddf8b5a468533066e6f318487/json

{
   "Id":"8ee46ed5032c9ebbddfdca56521c4101b5d4d27ddf8b5a468533066e6f318487",
   "Created":"2020-08-09T12:15:09.713675077Z",
   "Path":"bash",
   "Args":[

   ],
   "State":{
      "Status":"created",
      "Running":false,
      "Paused":false,
      "Restarting":false,
      "OOMKilled":false,
      "Dead":false,
      "Pid":0,
      "ExitCode":0,
      "Error":"",
      "StartedAt":"0001-01-01T00:00:00Z",
      "FinishedAt":"0001-01-01T00:00:00Z"
   },
   "Image":"sha256:2eb2d388e1a255c98029f40d6d7f8029fb13f1030abc8f11ccacbca686a8dc12",
   "ResolvConfPath":"",
   "HostnamePath":"",
   "HostsPath":"",
   "LogPath":"",
   "Name":"/ubuntu",
   "RestartCount":0,
   "Driver":"overlay2",
   "Platform":"linux",
   "MountLabel":"",
   "ProcessLabel":"",
   "AppArmorProfile":"",
   "ExecIDs":null,
   "HostConfig":{
      "Binds":null,
      "ContainerIDFile":"",
      "LogConfig":{
         "Type":"json-file",
         "Config":{

         }
      },
      "NetworkMode":"default",
      "PortBindings":null,
      "RestartPolicy":{
         "Name":"",
         "MaximumRetryCount":0
      },
      "AutoRemove":true,
      "VolumeDriver":"",
      "VolumesFrom":null,
      "CapAdd":null,
      "CapDrop":null,
      "Capabilities":null,
      "Dns":null,
      "DnsOptions":null,
      "DnsSearch":null,
      "ExtraHosts":null,
      "GroupAdd":null,
      "IpcMode":"private",
      "Cgroup":"",
      "Links":null,
      "OomScoreAdj":0,
      "PidMode":"",
      "Privileged":false,
      "PublishAllPorts":false,
      "ReadonlyRootfs":false,
      "SecurityOpt":null,
      "UTSMode":"",
      "UsernsMode":"",
      "ShmSize":67108864,
      "Runtime":"runc",
      "ConsoleSize":[
         0,
         0
      ],
      "Isolation":"",
      "CpuShares":0,
      "Memory":0,
      "NanoCpus":0,
      "CgroupParent":"",
      "BlkioWeight":0,
      "BlkioWeightDevice":null,
      "BlkioDeviceReadBps":null,
      "BlkioDeviceWriteBps":null,
      "BlkioDeviceReadIOps":null,
      "BlkioDeviceWriteIOps":null,
      "CpuPeriod":0,
      "CpuQuota":0,
      "CpuRealtimePeriod":0,
      "CpuRealtimeRuntime":0,
      "CpusetCpus":"",
      "CpusetMems":"",
      "Devices":null,
      "DeviceCgroupRules":null,
      "DeviceRequests":null,
      "KernelMemory":0,
      "KernelMemoryTCP":0,
      "MemoryReservation":0,
      "MemorySwap":0,
      "MemorySwappiness":null,
      "OomKillDisable":false,
      "PidsLimit":null,
      "Ulimits":null,
      "CpuCount":0,
      "CpuPercent":0,
      "IOMaximumIOps":0,
      "IOMaximumBandwidth":0,
      "MaskedPaths":[
         "/proc/asound",
         "/proc/acpi",
         "/proc/kcore",
         "/proc/keys",
         "/proc/latency_stats",
         "/proc/timer_list",
         "/proc/timer_stats",
         "/proc/sched_debug",
         "/proc/scsi",
         "/sys/firmware"
      ],
      "ReadonlyPaths":[
         "/proc/bus",
         "/proc/fs",
         "/proc/irq",
         "/proc/sys",
         "/proc/sysrq-trigger"
      ]
   },
   "GraphDriver":{
      "Data":{
         "LowerDir":"/var/lib/docker/overlay2/dbd34c269fc293d1287830eecd155ff1364560d9aa09b59fd8bdf8172e4c0e31-init/diff:/var/lib/docker/overlay2/c5d154fbe09727d00303094bc5ca68c8bed5c8ad281b7ea580cb5a30a1ad8466/diff:/var/lib/docker/overlay2/1f1b314fff38b0c46a397178603e626fea2dabae88cd9d6af43c432139db33d5/diff:/var/lib/docker/overlay2/fa4568c371e4832c41f7c787b9a9aa3305de9ccaa2296eecbb283280a87f6fe6/diff:/var/lib/docker/overlay2/2a79fa161523805f6c896ce24deb5a91dbe914798ac672cf7f5389ae31336d4f/diff",
         "MergedDir":"/var/lib/docker/overlay2/dbd34c269fc293d1287830eecd155ff1364560d9aa09b59fd8bdf8172e4c0e31/merged",
         "UpperDir":"/var/lib/docker/overlay2/dbd34c269fc293d1287830eecd155ff1364560d9aa09b59fd8bdf8172e4c0e31/diff",
         "WorkDir":"/var/lib/docker/overlay2/dbd34c269fc293d1287830eecd155ff1364560d9aa09b59fd8bdf8172e4c0e31/work"
      },
      "Name":"overlay2"
   },
   "Mounts":[

   ],
   "Config":{
      "Hostname":"8ee46ed5032c",
      "Domainname":"",
      "User":"",
      "AttachStdin":false,
      "AttachStdout":true,
      "AttachStderr":true,
      "Tty":false,
      "OpenStdin":false,
      "StdinOnce":false,
      "Env":[
         "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      ],
      "Cmd":[
         "bash"
      ],
      "Image":"ubuntu:18.04",
      "Volumes":null,
      "WorkingDir":"",
      "Entrypoint":null,
      "OnBuild":null,
      "Labels":{

      }
   },
   "NetworkSettings":{
      "Bridge":"",
      "SandboxID":"",
      "HairpinMode":false,
      "LinkLocalIPv6Address":"",
      "LinkLocalIPv6PrefixLen":0,
      "Ports":{

      },
      "SandboxKey":"",
      "SecondaryIPAddresses":null,
      "SecondaryIPv6Addresses":null,
      "EndpointID":"",
      "Gateway":"",
      "GlobalIPv6Address":"",
      "GlobalIPv6PrefixLen":0,
      "IPAddress":"",
      "IPPrefixLen":0,
      "IPv6Gateway":"",
      "MacAddress":"",
      "Networks":{
         "bridge":{
            "IPAMConfig":null,
            "Links":null,
            "Aliases":null,
            "NetworkID":"",
            "EndpointID":"",
            "Gateway":"",
            "IPAddress":"",
            "IPPrefixLen":0,
            "IPv6Gateway":"",
            "GlobalIPv6Address":"",
            "GlobalIPv6PrefixLen":0,
            "MacAddress":"",
            "DriverOpts":null
         }
      }
   }
}
Podman
% sudo curl --unix-socket /run/podman/podman.sock -H "Content-Type: application/json" --data '{"Tty": false, "OpenStdin": false, "StdinOnce": false, "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Cmd": ["bash"], "Image": "ubuntu:18.04", "NetworkDisabled": false, "HostConfig": {"NetworkMode": "default", "AutoRemove": true}}' 'http://d/v1.40/containers/create?name=ubuntu'
{"Id":"a565948e2b6177d01d32d7cf02bf6347fd92a17bc05790690bd24d7ba695751a","Warnings":[]}

% sudo curl --unix-socket /run/podman/podman.sock http://d/v1.35/containers/a565948e2b6177d01d32d7cf02bf6347fd92a17bc05790690bd24d7ba695751a/json

{
   "Id":"a565948e2b6177d01d32d7cf02bf6347fd92a17bc05790690bd24d7ba695751a",
   "Created":"2020-08-09 05:11:47.969774599 -0700 PDT",
   "Path":"",
   "Args":null,
   "State":{
      "Status":"configured",
      "Running":false,
      "Paused":false,
      "Restarting":false,
      "OOMKilled":false,
      "Dead":false,
      "Pid":0,
      "ExitCode":0,
      "Error":"",
      "StartedAt":"0001-01-01T00:00:00Z",
      "FinishedAt":"0001-01-01T00:00:00Z"
   },
   "Image":"ubuntu:18.04",
   "ResolvConfPath":"",
   "HostnamePath":"",
   "HostsPath":"",
   "LogPath":"/var/lib/containers/storage/overlay-containers/a565948e2b6177d01d32d7cf02bf6347fd92a17bc05790690bd24d7ba695751a/userdata/ctr.log",
   "Name":"/ubuntu",
   "RestartCount":0,
   "Driver":"overlay",
   "Platform":"linux",
   "MountLabel":"",
   "ProcessLabel":"",
   "AppArmorProfile":"",
   "ExecIDs":[

   ],
   "HostConfig":{
      "Binds":[

      ],
      "ContainerIDFile":"",
      "LogConfig":{
         "Type":"",
         "Config":null
      },
      "NetworkMode":"",
      "PortBindings":{

      },
      "RestartPolicy":{
         "Name":"",
         "MaximumRetryCount":0
      },
      "AutoRemove":true,
      "VolumeDriver":"",
      "VolumesFrom":null,
      "CapAdd":[

      ],
      "CapDrop":[

      ],
      "Capabilities":null,
      "CgroupnsMode":"",
      "Dns":[

      ],
      "DnsOptions":[

      ],
      "DnsSearch":[

      ],
      "ExtraHosts":[

      ],
      "GroupAdd":[

      ],
      "IpcMode":"private",
      "Cgroup":"",
      "Links":null,
      "OomScoreAdj":0,
      "PidMode":"private",
      "Privileged":false,
      "PublishAllPorts":false,
      "ReadonlyRootfs":false,
      "SecurityOpt":[

      ],
      "UTSMode":"private",
      "UsernsMode":"",
      "ShmSize":0,
      "Runtime":"oci",
      "ConsoleSize":[
         0,
         0
      ],
      "Isolation":"",
      "CpuShares":0,
      "Memory":0,
      "NanoCpus":0,
      "CgroupParent":"",
      "BlkioWeight":0,
      "BlkioWeightDevice":null,
      "BlkioDeviceReadBps":null,
      "BlkioDeviceWriteBps":null,
      "BlkioDeviceReadIOps":null,
      "BlkioDeviceWriteIOps":null,
      "CpuPeriod":0,
      "CpuQuota":0,
      "CpuRealtimePeriod":0,
      "CpuRealtimeRuntime":0,
      "CpusetCpus":"",
      "CpusetMems":"",
      "Devices":[

      ],
      "DeviceCgroupRules":null,
      "DeviceRequests":null,
      "KernelMemory":0,
      "KernelMemoryTCP":0,
      "MemoryReservation":0,
      "MemorySwap":0,
      "MemorySwappiness":0,
      "OomKillDisable":false,
      "PidsLimit":0,
      "Ulimits":[
         {
            "Name":"RLIMIT_NOFILE",
            "Hard":1048576,
            "Soft":1048576
         },
         {
            "Name":"RLIMIT_NPROC",
            "Hard":4194304,
            "Soft":4194304
         }
      ],
      "CpuCount":0,
      "CpuPercent":0,
      "IOMaximumIOps":0,
      "IOMaximumBandwidth":0,
      "MaskedPaths":null,
      "ReadonlyPaths":null
   },
   "GraphDriver":{
      "Data":{
         "LowerDir":"/var/lib/containers/storage/overlay/802cad39c47beadbe4c98febc97d1c1b878718513bd5d8872e8590685910fe40/diff:/var/lib/containers/storage/overlay/429ac67f5820c7cf30c1966115900b18607ee53772918d113fd7613bd7eaf861/diff:/var/lib/containers/storage/overlay/8a1f2c24e0215398919ade6736616c413010aedef99dead4a06de6c18c3fa69a/diff:/var/lib/containers/storage/overlay/7ef3687765828a9cb2645925f27febbac21a5adece69e8437c26184a897b6ec7/diff",
         "UpperDir":"/var/lib/containers/storage/overlay/d0afacaf914f3df42da116f32bb162757d3e4547a0e180628e9d7180c8d84c38/diff",
         "WorkDir":"/var/lib/containers/storage/overlay/d0afacaf914f3df42da116f32bb162757d3e4547a0e180628e9d7180c8d84c38/work"
      },
      "Name":"overlay"
   },
   "SizeRootFs":0,
   "Mounts":[

   ],
   "Config":{
      "Hostname":"a565948e2b61",
      "Domainname":"",
      "User":"",
      "AttachStdin":false,
      "AttachStdout":false,
      "AttachStderr":false,
      "Tty":false,
      "OpenStdin":false,
      "StdinOnce":false,
      "Env":[
         "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
         "TERM=xterm",
         "HOSTNAME=",
         "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
         "TERM=xterm"
      ],
      "Cmd":[
         "bash"
      ],
      "Image":"ubuntu:18.04",
      "Volumes":null,
      "WorkingDir":"/",
      "Entrypoint":[

      ],
      "OnBuild":null,
      "Labels":{

      },
      "StopSignal":"\u000f",
      "StopTimeout":10
   },
   "NetworkSettings":{
      "Bridge":"",
      "SandboxID":"",
      "HairpinMode":false,
      "LinkLocalIPv6Address":"",
      "LinkLocalIPv6PrefixLen":0,
      "Ports":{

      },
      "SandboxKey":"",
      "SecondaryIPAddresses":null,
      "SecondaryIPv6Addresses":null,
      "EndpointID":"",
      "Gateway":"",
      "GlobalIPv6Address":"",
      "GlobalIPv6PrefixLen":0,
      "IPAddress":"",
      "IPPrefixLen":0,
      "IPv6Gateway":"",
      "MacAddress":"",
      "Networks":null
   }
}

The key network configuration differences are highlighted in the followind diff:

--- docker.json	2020-08-09 19:26:24.000000000 +0700
+++ podman.json	2020-08-09 19:26:39.000000000 +0700
@@ -1,39 +1,23 @@
    "NetworkSettings":{
       "Bridge":"",
       "SandboxID":"",
       "HairpinMode":false,
       "LinkLocalIPv6Address":"",
       "LinkLocalIPv6PrefixLen":0,
       "Ports":{
 
       },
       "SandboxKey":"",
       "SecondaryIPAddresses":null,
       "SecondaryIPv6Addresses":null,
       "EndpointID":"",
       "Gateway":"",
       "GlobalIPv6Address":"",
       "GlobalIPv6PrefixLen":0,
       "IPAddress":"",
       "IPPrefixLen":0,
       "IPv6Gateway":"",
       "MacAddress":"",
-      "Networks":{
-         "bridge":{
-            "IPAMConfig":null,
-            "Links":null,
-            "Aliases":null,
-            "NetworkID":"",
-            "EndpointID":"",
-            "Gateway":"",
-            "IPAddress":"",
-            "IPPrefixLen":0,
-            "IPv6Gateway":"",
-            "GlobalIPv6Address":"",
-            "GlobalIPv6PrefixLen":0,
-            "MacAddress":"",
-            "DriverOpts":null
-         }
-      }
+      "Networks":null
    }
 }

SaltStack just fails to introspect the NetworkSettings->Networks value because it is null and not a dictionary.

A quick scan of the Docker API shows that Networks should be an object: https://docs.docker.com/engine/api/v1.40/#operation/ContainerInspect

curl -s https://docs.docker.com/engine/api/v1.40.yaml | grep -A 3 NetworkSettings

@max-arnold
Copy link
Author

@zhangguanzhang The args must not be empty error seems to be fixed!

@rhatdan
Copy link
Member

rhatdan commented Sep 11, 2020

@max-arnold @zhangguanzhang Does this mean we can close this issue?

@max-arnold
Copy link
Author

@rhatdan The main issue (missing Networks key) is still present, see #5580 (comment)

@rsommer
Copy link
Contributor

rsommer commented Nov 20, 2020

I think this should be fixed thanks to @baude - can't locate the PR (it fixed a few other things at the same time) but he duplicated the Libpod side of things, which already matched Docker.

I could not find a matching PR and testing with the current HEAD/2.2.0-rc1 still yields the Networks-key as null.

$ podman version
Version:      2.2.0-rc1
API Version:  2.1.0
Go Version:   go1.15.5
Git Commit:   02843f881f9271440e7eaad8db231ddf20e33e51
Built:        Fri Nov 20 08:57:52 2020
OS/Arch:      linux/amd64

@mheon mheon reopened this Nov 20, 2020
@rhatdan
Copy link
Member

rhatdan commented Dec 24, 2020

@max-arnold @rsommer @baude is this fixed in the main branch now?

@baude
Copy link
Member

baude commented Dec 24, 2020

it is not ... but i can fix this. i need info from @mheon . @mheon what are you excluding the default network from inspect?

if len(networks) > 0 && !isDefault {

@mheon
Copy link
Member

mheon commented Dec 24, 2020

This matches Docker behaviour - network structs are only included if and only if they were explicitly requested by the user (otherwise, the network information is included in the base network info struct). The default network was not added by a user, so we don't generate a network struct for it.

@baude
Copy link
Member

baude commented Dec 24, 2020

@mheon, the problem is that it is organized differently for docker too. there is no general network settings with network underneath it. just networks ... which does include the default. ill cobble a pr to see if we can do this.

@baude
Copy link
Member

baude commented Dec 24, 2020

eh, this is more difficult. will need to discuss when we are back to work.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Jan 26, 2021

@baude Reminder this is something we might need?

@baude
Copy link
Member

baude commented Jan 26, 2021

I think #9109 may actually fix this?

@mheon
Copy link
Member

mheon commented Jan 26, 2021

I don't think I'm fixing compat inspect, though

@mheon
Copy link
Member

mheon commented Jan 26, 2021

Though, this pre-dates APIv2, so we can probably close this one after #9109 merges

@baude
Copy link
Member

baude commented Jan 26, 2021

well, i want to see your pr get in ... from there i can see what is and isnt

@baude
Copy link
Member

baude commented Jan 27, 2021

@max-arnold are you in a position to try the latest code in the master branch and indicate if there is still a problem? extra credit if you can get onto freenode #podman and hit me up when you do the test.

@max-arnold
Copy link
Author

I’ll take a look

@max-arnold
Copy link
Author

max-arnold commented Feb 4, 2021

Did another round of testing. Here is a json response diff between podman and docker:

GET http://localhost/v1.35/containers/5d30a00aa13ee9335cf7d1b7bf6f983364ad1f78ed9f6de3711c3e33f4d3e832/json

diff -u podman.json docker.json
--- podman.json	2021-02-04 12:51:01.000000000 +0700
+++ docker.json	2021-02-04 12:51:17.000000000 +0700
@@ -1,10 +1,8 @@
 {
-  "Id": "5d30a00aa13ee9335cf7d1b7bf6f983364ad1f78ed9f6de3711c3e33f4d3e832",
-  "Created": "2021-02-03T21:42:38.14509732-08:00",
+  "Id": "847867ecd11546f048c3145ac6fd869ee2b0dc8098c17e53ac3d0c01013edc66",
+  "Created": "2021-02-04T05:45:52.423553663Z",
   "Path": "bash",
-  "Args": [
-    "bash"
-  ],
+  "Args": [],
   "State": {
     "Status": "created",
     "Running": false,
@@ -18,28 +16,28 @@
     "StartedAt": "0001-01-01T00:00:00Z",
     "FinishedAt": "0001-01-01T00:00:00Z"
   },
-  "Image": "docker.io/library/ubuntu:18.04",
+  "Image": "sha256:c090eaba6b9480dcd1daf063e4efe271711745c21a73c53bdc94c8d9610d2fd9",
   "ResolvConfPath": "",
   "HostnamePath": "",
   "HostsPath": "",
-  "LogPath": "/var/lib/containers/storage/overlay-containers/5d30a00aa13ee9335cf7d1b7bf6f983364ad1f78ed9f6de3711c3e33f4d3e832/userdata/ctr.log",
+  "LogPath": "",
   "Name": "/ubuntu",
   "RestartCount": 0,
-  "Driver": "overlay",
+  "Driver": "overlay2",
   "Platform": "linux",
   "MountLabel": "",
   "ProcessLabel": "",
-  "AppArmorProfile": "containers-default-0.33.1",
-  "ExecIDs": [],
+  "AppArmorProfile": "",
+  "ExecIDs": null,
   "HostConfig": {
-    "Binds": [],
+    "Binds": null,
     "ContainerIDFile": "",
     "LogConfig": {
       "Type": "json-file",
-      "Config": null
+      "Config": {}
     },
-    "NetworkMode": "bridge",
-    "PortBindings": {},
+    "NetworkMode": "default",
+    "PortBindings": null,
     "RestartPolicy": {
       "Name": "",
       "MaximumRetryCount": 0
@@ -47,27 +45,27 @@
     "AutoRemove": false,
     "VolumeDriver": "",
     "VolumesFrom": null,
-    "CapAdd": [],
-    "CapDrop": [],
-    "CgroupnsMode": "",
-    "Dns": [],
-    "DnsOptions": [],
-    "DnsSearch": [],
-    "ExtraHosts": [],
-    "GroupAdd": [],
+    "CapAdd": null,
+    "CapDrop": null,
+    "CgroupnsMode": "host",
+    "Dns": null,
+    "DnsOptions": null,
+    "DnsSearch": null,
+    "ExtraHosts": null,
+    "GroupAdd": null,
     "IpcMode": "private",
     "Cgroup": "",
     "Links": null,
     "OomScoreAdj": 0,
-    "PidMode": "private",
+    "PidMode": "",
     "Privileged": false,
     "PublishAllPorts": false,
     "ReadonlyRootfs": false,
-    "SecurityOpt": [],
-    "UTSMode": "private",
+    "SecurityOpt": null,
+    "UTSMode": "",
     "UsernsMode": "",
-    "ShmSize": 65536000,
-    "Runtime": "oci",
+    "ShmSize": 67108864,
+    "Runtime": "runc",
     "ConsoleSize": [
       0,
       0
@@ -89,71 +87,73 @@
     "CpuRealtimeRuntime": 0,
     "CpusetCpus": "",
     "CpusetMems": "",
-    "Devices": [],
+    "Devices": null,
     "DeviceCgroupRules": null,
     "DeviceRequests": null,
     "KernelMemory": 0,
     "KernelMemoryTCP": 0,
     "MemoryReservation": 0,
     "MemorySwap": 0,
-    "MemorySwappiness": 0,
+    "MemorySwappiness": null,
     "OomKillDisable": false,
-    "PidsLimit": 2048,
-    "Ulimits": [
-      {
-        "Name": "RLIMIT_NOFILE",
-        "Hard": 1048576,
-        "Soft": 1048576
-      },
-      {
-        "Name": "RLIMIT_NPROC",
-        "Hard": 4194304,
-        "Soft": 4194304
-      }
-    ],
+    "PidsLimit": null,
+    "Ulimits": null,
     "CpuCount": 0,
     "CpuPercent": 0,
     "IOMaximumIOps": 0,
     "IOMaximumBandwidth": 0,
-    "MaskedPaths": null,
-    "ReadonlyPaths": null
+    "MaskedPaths": [
+      "/proc/asound",
+      "/proc/acpi",
+      "/proc/kcore",
+      "/proc/keys",
+      "/proc/latency_stats",
+      "/proc/timer_list",
+      "/proc/timer_stats",
+      "/proc/sched_debug",
+      "/proc/scsi",
+      "/sys/firmware"
+    ],
+    "ReadonlyPaths": [
+      "/proc/bus",
+      "/proc/fs",
+      "/proc/irq",
+      "/proc/sys",
+      "/proc/sysrq-trigger"
+    ]
   },
   "GraphDriver": {
     "Data": {
-      "LowerDir": "/var/lib/containers/storage/overlay/b4348bfd84050f05a39ebe017b137d3bde5ce307cc5dd819495c4f362587c4e3/diff:/var/lib/containers/storage/overlay/9dd0b45ebe39e477b2a869802e9b771d991f7341cb79e3446c9ca8231fc35d3f/diff:/var/lib/containers/storage/overlay/c95d2191d7773c6e29188f92922bc9547e1f0b6130e85dfc2f5e4eae13137c7c/diff",
-      "UpperDir": "/var/lib/containers/storage/overlay/3f8be611c10079bf6284c1c8d8e30ffff7dea7bfd2026cd8220e58b2856932a2/diff",
-      "WorkDir": "/var/lib/containers/storage/overlay/3f8be611c10079bf6284c1c8d8e30ffff7dea7bfd2026cd8220e58b2856932a2/work"
+      "LowerDir": "/var/lib/docker/overlay2/45722d27de80b9cb28e6efe7d84711a2e197a4c9e0127bb6afc150f53902a2d3-init/diff:/var/lib/docker/overlay2/bf8a4fa8f68f9e1843e1b1cdce1a62aad8ddcdef43ad860aaaedbe3e292041d5/diff:/var/lib/docker/overlay2/56a4d0bfdfc12c7842ff5a7df19a130189d9d2a8d0b8ef4bcdfc2ea2aa74722a/diff:/var/lib/docker/overlay2/8bb4e26228489ed028c33618f945b0eeb07b3e543a55067b23568f47fb3646fa/diff",
+      "MergedDir": "/var/lib/docker/overlay2/45722d27de80b9cb28e6efe7d84711a2e197a4c9e0127bb6afc150f53902a2d3/merged",
+      "UpperDir": "/var/lib/docker/overlay2/45722d27de80b9cb28e6efe7d84711a2e197a4c9e0127bb6afc150f53902a2d3/diff",
+      "WorkDir": "/var/lib/docker/overlay2/45722d27de80b9cb28e6efe7d84711a2e197a4c9e0127bb6afc150f53902a2d3/work"
     },
-    "Name": "overlay"
+    "Name": "overlay2"
   },
-  "SizeRootFs": 0,
   "Mounts": [],
   "Config": {
-    "Hostname": "5d30a00aa13e",
+    "Hostname": "847867ecd115",
     "Domainname": "",
     "User": "",
     "AttachStdin": false,
-    "AttachStdout": false,
-    "AttachStderr": false,
+    "AttachStdout": true,
+    "AttachStderr": true,
     "Tty": false,
     "OpenStdin": false,
     "StdinOnce": false,
     "Env": [
-      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
-      "TERM=xterm",
-      "container=podman"
+      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
     ],
     "Cmd": [
       "bash"
     ],
     "Image": "docker.io/library/ubuntu:18.04",
     "Volumes": null,
-    "WorkingDir": "/",
-    "Entrypoint": [],
+    "WorkingDir": "",
+    "Entrypoint": null,
     "OnBuild": null,
-    "Labels": {},
-    "StopSignal": "15",
-    "StopTimeout": 0
+    "Labels": {}
   },
   "NetworkSettings": {
     "Bridge": "",
@@ -174,11 +174,11 @@
     "IPv6Gateway": "",
     "MacAddress": "",
     "Networks": {
-      "default": {
+      "bridge": {
         "IPAMConfig": null,
         "Links": null,
         "Aliases": null,
-        "NetworkID": "default",
+        "NetworkID": "",
         "EndpointID": "",
         "Gateway": "",
         "IPAddress": "",

The NetworkSettings part (at the end of the diff) is almost identical, and that is good!

Then Salt tries to start the container:

POST http://localhost/v1.35/containers/5d30a00aa13ee9335cf7d1b7bf6f983364ad1f78ed9f6de3711c3e33f4d3e832/start

And this is the response:

{
  "cause": "CNI network \"default\" not found",
  "message": "error configuring network namespace for container 5d30a00aa13ee9335cf7d1b7bf6f983364ad1f78ed9f6de3711c3e33f4d3e832: CNI network \"default\" not found",
  "response": 500
}

libpod: 4e1bcf3
conmon: c3f31c0671f6c84f5e02ec710385e1e8f2936066
runc: 19437f3a55eb28411a4f48efdd494b1e3f49e055
plugins: 74a6b28a2c2796a2638ba61a85bb620640bfeb31
Docker version 20.10.3, build 48d30b5

I'll ping you on the IRC soon.

@baude
Copy link
Member

baude commented Feb 4, 2021

I suspect this issue is because the default network for docker is actually called default and on podman it is called podman. If you want the default network, you could try not sending it (to Podman)

@mheon
Copy link
Member

mheon commented Feb 4, 2021

@baude I'm fairly certain default is not a network name, but rather a reserved word Docker uses to refer to whatever the current default network is.

@mheon
Copy link
Member

mheon commented Feb 4, 2021

We patched this elsewhere (I remember Dan working on the PR for it)

@max-arnold
Copy link
Author

I suspect this issue is because the default network for docker is actually called default and on podman it is called podman. If you want the default network, you could try not sending it (to Podman)

According to the diff, the default network for docker is called bridge, and the network id is empty (for podman they are both set to default):

--- podman.json	2021-02-04 12:51:01.000000000 +0700
+++ docker.json	2021-02-04 12:51:17.000000000 +0700
@@ -174,11 +174,11 @@
     "IPv6Gateway": "",
     "MacAddress": "",
     "Networks": {
-      "default": {
+      "bridge": {
         "IPAMConfig": null,
         "Links": null,
         "Aliases": null,
-        "NetworkID": "default",
+        "NetworkID": "",
         "EndpointID": "",
         "Gateway": "",
         "IPAddress": "",

This is the whole API exchange between Salt and Podman:

GET http+docker://localhost/v1.35/version
GET http+docker://localhost/v1.35/images/docker.io/library/ubuntu:18.04/json
GET http+docker://localhost/v1.35/images/json?only_ids=0&all=1
POST http+docker://localhost/v1.35/images/create?tag=18.04&fromImage=docker.io%2Flibrary%2Fubuntu
GET http+docker://localhost/v1.35/containers/ubuntu/json
POST http+docker://localhost/v1.35/containers/create?name=ubuntu 
    DATA: {"Tty": false, "OpenStdin": false, "StdinOnce": false, "AttachStdin": false,
                "AttachStdout": true, "AttachStderr": true, "Cmd": ["bash"],
                "Image": "docker.io/library/ubuntu:18.04", "NetworkDisabled": false, 
                "HostConfig": {"NetworkMode": "default"}}
GET http+docker://localhost/v1.35/containers/5d30a00aa13ee9335cf7d1b7bf6f983364ad1f78ed9f6de3711c3e33f4d3e832/json
POST http+docker://localhost/v1.35/containers/5d30a00aa13ee9335cf7d1b7bf6f983364ad1f78ed9f6de3711c3e33f4d3e832/start

I did a little bit of digging to see where the "HostConfig": {"NetworkMode": "default"} part comes from. Apparently, it is a default value in docker-py: https://github.com/docker/docker-py/blob/4.4.1/docker/types/containers.py#L341

The Docker API documentation contains the following text:

NetworkMode | string

Network mode to use for this container. Supported standard values are:
bridge, host, none, and container:<name\|id>.
Any other value is taken as a custom network's name to which this container should connect to.

However, there is a docker-py issue that contains more details about the default mode: docker/docker-py#986 (comment)

To summarize:

  1. Docker accepts the default mode and uses bridge as a result
  2. Podman also accepts the default mode but then tries to actually use it and fails

This difference is also visible in the json response diff above.

@max-arnold
Copy link
Author

I did a little bit more digging, and while Salt can't pass an empty NetworkMode (because of the default fallback value in docker-py), it can provide an explicit network_mode keyword:

sudo salt-call docker.run_container registry.fedoraproject.org/f29/httpd name=httpd bg=true replace=true network_mode=podman port_bindings=8080:8080/tcp

This avoids the 500 error when starting the container:

GET http+docker://localhost/v1.35/version
GET http+docker://localhost/v1.35/images/registry.fedoraproject.org/f29/httpd/json
GET http+docker://localhost/v1.35/containers/httpd/json
POST http+docker://localhost/v1.35/containers/create?name=httpd
     {"ExposedPorts": {"8080/tcp": {}}, "Tty": false, "OpenStdin": false, "StdinOnce": false,
      "AttachStdin": false, "AttachStdout": true, "AttachStderr": true,
      "Image": "registry.fedoraproject.org/f29/httpd", "NetworkDisabled": false,
      "HostConfig": {"NetworkMode": "podman", "PortBindings": 
      {"8080/tcp": [{"HostIp": "", "HostPort": "8080"}]}}}
GET http+docker://localhost/v1.35/containers/9edf973ab566ae9cae353869391ec4b51634739c1df2818050d8d51a8ac3d5a7/json
POST http+docker://localhost/v1.35/containers/9edf973ab566ae9cae353869391ec4b51634739c1df2818050d8d51a8ac3d5a7/start
GET http+docker://localhost/v1.35/containers/9edf973ab566ae9cae353869391ec4b51634739c1df2818050d8d51a8ac3d5a7/json
GET http+docker://localhost/v1.35/containers/json?limit=-1&all=1&size=0&trunc_cmd=0
GET http+docker://localhost/v1.35/containers/9edf973ab566ae9cae353869391ec4b51634739c1df2818050d8d51a8ac3d5a7/json
GET http+docker://localhost/v1.35/containers/json?limit=-1&all=1&size=0&trunc_cmd=0
GET http+docker://localhost/v1.35/containers/9edf973ab566ae9cae353869391ec4b51634739c1df2818050d8d51a8ac3d5a7/json

And finally:

curl -s http://localhost:8080 | head

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
	<head>
		<title>Test Page for the Apache HTTP Server on Fedora</title>
		<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
		<style type="text/css">
			/*<![CDATA[*/
			body {
				background-color: #fff;

So I can call it a success! 🎉

@baude
Copy link
Member

baude commented Feb 5, 2021

if you good, mind closing this?

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

No branches or pull requests

9 participants