Docker API Compatibility.

[computator]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

The podman compatibility API seems to have several formatting and compatibility issues vs the real docker API. I encountered issues myself running Podman under Nomad with the Docker API. This seems to have been fixed in the latest build according to #8235, likely because of #7860, but that still leaves some other issues I see when comparing the differences in API output manually.

I only compared the inspect container endpoint, but I expect that others have similar issues with how many I encountered here.

Steps to reproduce the issue:

1. Run a container on podman, then repeat the steps on docker:
   podman|docker run -d --rm gcr.io/google_containers/pause-amd64:3.2

2. Inspect the created containers via the respective APIs (Using jq to format the JSON):

• Podman:
  curl -s --unix-socket /run/podman/podman.sock http://d/containers/{CONTAINER_ID}/json | jq .
• Docker:
  curl -s --unix-socket /var/run/docker.sock http://d/containers/{CONTAINER_ID}/json | jq .

3. Compare the printed output

Describe the results you received:
Several fields were mismatched in syntax. A few I noticed:

• Created format: "Created": "2020-11-12 22:26:13.620613454 +0000 UTC" vs "Created": "2020-11-12T22:26:35.43720122Z" (Fixed in aforementioned Docker Compatibility API Timestamp Format #7860)
• Empty args format: "Args": null vs "Args": []
• Image specifier format: "Image": "gcr.io/google_containers/pause-amd64:3.2" vs "Image": "sha256:80d28bedfe5dec59da9ebf8e6260224ac9008ab5c11dbbe16ee3ba3e4439ac2c" (The image format in Config.Image matches though)
• ExecIDs: "ExecIDs": [] vs "ExecIDs": null (Swapped order vs Args above)
• Several other instances of empty Arrays or Objects vs null
• Several 0s or empty strings vs other specified options or seeming defaults.

Describe the results you expected:
I expected everything to match in syntax, and most things to match in value except for obvious differences such as host paths and container IDs. I expected everything to match the Docker API spec https://docs.docker.com/engine/api/v1.40/#operation/ContainerInspect

Output of podman version:

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Additional environment details (AWS, VirtualBox, physical, etc.):
Docker version info:

Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:02:52 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:01:20 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[afbjorklund]

I think nomad has a driver for podman ? https://github.com/hashicorp/nomad-driver-podman

[computator]

It does yes, but at the moment it still uses the deprecated Varlink API and it also has a lot less features. They are working on updating to the new API (hashicorp/nomad-driver-podman#51). I should be able to use either driver though, and wanted to use the docker one for now because of it's expanded featureset. Hopefully in the future nomad-driver-podman will support additional podman specific features that the docker driver does not!

[afbjorklund]

I don't think the nomad driver has to worry about any of the remote APIs, varlink or the new REST one ? Unlike the docker driver or the containerd driver, the podman driver could just talk directly to libpod locally and work more like the exec driver does. But like you say you could use either the docker API or the podman API to talk to a podman service, but it wouldn't be "daemonless"...

https://www.nomadproject.io/docs/drivers/podman

The Podman task driver plugin for Nomad uses the Pod Manager (podman) daemonless container runtime for executing Nomad tasks. Podman supports OCI containers and its command line tool is meant to be a drop-in replacement for Docker's.

[computator]

In theory yes, but that's not the way they decided to implement it per these:

• [Feature Request] Support Podman on HashiCorp Nomad #3387 (comment)
• [Feature Request] Support podman for running containers hashicorp/nomad#5312 (comment)
• How to use libpod as library in my own program? #3445

But like you say you could use either the docker API or the podman API to talk to a podman service, but it wouldn't be "daemonless"...

What I was saying is that there are two options currently implemented that should work right now: the docker driver (via Podman APIv2 docker compatibility), or the podman driver (via Podman varlink API, and soon via the native APIv2).

[afbjorklund]

Interesting, that means that libpod is not usable and it was easier to use the daemon (err, "systemd socket-activated service")

More like a "podman-remote" driver

Thanks for the links. I guess a third option would be to call the podman CLI, similar to how the singularity driver works...

Too bad about the library approach.

[rhatdan]

@jwhonce PTAL

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@computator @afbjorklund Is this still an issue, lots of fixes have gone into the API.

[afbjorklund]

@computator @afbjorklund Is this still an issue, lots of fixes have gone into the API.

I'm not actually using Nomad with Podman, just wondered why "docker" API and not "podman" API...

But I think it would be a good thing to support (running with Nomad), as an alternative to Docker Swarm.

#3387 #3445

Seems like the nomad driver supports podman v2 now though: hashicorp/nomad-driver-podman#37

[rhatdan]

Ok reopen if this is currently broken.

[computator]

This issue is about API compatibility, not Nomad. Nomad just happens to be my usecase. If it doesn't match it will inevitably break other things as well. I haven't had a chance to check whether Podman's API fully follows the Docker API schema yet, but I'll check that out later today if I can.

[rhatdan]

Great thanks.

[computator]

Yes there seems to still be a lot of minor schema differences. I repeated the process from my first post and created a diff between them. I will include a modified version of the diff that I have updated to remove expected differences, leaving only format differences or data differences that it looks like API consumers might depend on. I am sure there are additional schema differences beyond this depending on the container options or environment. This is also only a single endpoint and I expect there are probably differences on other endpoints too based on the differences in this one.

A difference of particular note that may or may not be indicative of another issue is the Args key. In my previous test with this procedure Args was set to null, a minor schema difference vs Docker's [], but has now changed to be ["/pause"]. This appears to now be in the correct format (probably by accident), but I am unsure why it is no longer empty like before.

Updated podman version:

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.2
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

--- <podman>
+++ <docker>
@@ -2,9 +2,7 @@
   "Id": "1af3dc25809e988e01c2fd943cad05cad2ecf159d55e2d76e70aadc1d859ccf2",
   "Created": "2020-12-16T21:59:28.379904918-08:00",
   "Path": "/pause",
-  "Args": [
-    "/pause"
-  ],
+  "Args": [],
   "State": {
     "Status": "running",
     "Running": true,
@@ -18,7 +16,7 @@
     "StartedAt": "2020-12-16T21:59:28.717352768-08:00",
     "FinishedAt": "0001-01-01T00:00:00Z"
   },
-  "Image": "gcr.io/google_containers/pause-amd64:3.2",
+  "Image": "sha256:80d28bedfe5dec59da9ebf8e6260224ac9008ab5c11dbbe16ee3ba3e4439ac2c",
   "ResolvConfPath": "/var/run/containers/storage/overlay-containers/1af3dc25809e988e01c2fd943cad05cad2ecf159d55e2d76e70aadc1d859ccf2/userdata/resolv.conf",
   "HostnamePath": "/var/run/containers/storage/overlay-containers/1af3dc25809e988e01c2fd943cad05cad2ecf159d55e2d76e70aadc1d859ccf2/userdata/hostname",
   "HostsPath": "/var/run/containers/storage/overlay-containers/1af3dc25809e988e01c2fd943cad05cad2ecf159d55e2d76e70aadc1d859ccf2/userdata/hosts",
@@ -30,31 +28,31 @@
   "MountLabel": "",
   "ProcessLabel": "",
   "AppArmorProfile": "containers-default-0.29.0",
-  "ExecIDs": [],
+  "ExecIDs": null,
   "HostConfig": {
-    "Binds": [],
+    "Binds": null,
     "ContainerIDFile": "",
     "LogConfig": {
-      "Type": "k8s-file",
-      "Config": null
+      "Type": "json-file",
+      "Config": {}
     },
     "NetworkMode": "bridge",
     "PortBindings": {},
     "RestartPolicy": {
-      "Name": "",
+      "Name": "no",
       "MaximumRetryCount": 0
     },
     "AutoRemove": true,
     "VolumeDriver": "",
     "VolumesFrom": null,
-    "CapAdd": [],
-    "CapDrop": [],
-    "CgroupnsMode": "",
+    "CapAdd": null,
+    "CapDrop": null,
+    "Capabilities": null,
     "Dns": [],
     "DnsOptions": [],
     "DnsSearch": [],
-    "ExtraHosts": [],
-    "GroupAdd": [],
+    "ExtraHosts": null,
+    "GroupAdd": null,
     "IpcMode": "private",
     "Cgroup": "",
     "Links": null,
@@ -63,11 +61,11 @@
     "Privileged": false,
     "PublishAllPorts": false,
     "ReadonlyRootfs": false,
-    "SecurityOpt": [],
+    "SecurityOpt": null,
     "UTSMode": "private",
     "UsernsMode": "",
     "ShmSize": 65536000,
-    "Runtime": "oci",
+    "Runtime": "runc",
     "ConsoleSize": [
       0,
       0
@@ -78,7 +76,7 @@
     "NanoCpus": 0,
     "CgroupParent": "",
     "BlkioWeight": 0,
-    "BlkioWeightDevice": null,
+    "BlkioWeightDevice": [],
     "BlkioDeviceReadBps": null,
     "BlkioDeviceWriteBps": null,
     "BlkioDeviceReadIOps": null,
@@ -96,7 +94,7 @@
     "KernelMemoryTCP": 0,
     "MemoryReservation": 0,
     "MemorySwap": 0,
-    "MemorySwappiness": 0,
+    "MemorySwappiness": null,
     "OomKillDisable": false,
     "PidsLimit": 2048,
     "Ulimits": [
@@ -145,7 +143,7 @@
       "HOSTNAME=1af3dc25809e",
       "HOME="
     ],
-    "Cmd": [],
+    "Cmd": null,
     "Image": "gcr.io/google_containers/pause-amd64:3.2",
     "Volumes": null,
     "WorkingDir": "/",
@@ -175,6 +173,22 @@
     "IPPrefixLen": 16,
     "IPv6Gateway": "",
     "MacAddress": "ae:c8:77:ae:b6:7a",
-    "Networks": null
+    "Networks": {
+      "bridge": {
+        "IPAMConfig": null,
+        "Links": null,
+        "Aliases": null,
+        "NetworkID": "3a30429450c6f046eb60570cc21315388c2e301e0c2207c97defa79ac2c99189",
+        "EndpointID": "16d55b9e2b8d5079c2a399baa4db5a3283dec3155e4464bfc93919726ed5560e",
+        "Gateway": "172.17.0.1",
+        "IPAddress": "172.17.0.2",
+        "IPPrefixLen": 16,
+        "IPv6Gateway": "",
+        "GlobalIPv6Address": "",
+        "GlobalIPv6PrefixLen": 0,
+        "MacAddress": "02:42:ac:11:00:02",
+        "DriverOpts": null
+      }
+    }
   }
 }

[rhatdan]

Could you open another issue with these. Otherwise they will get lost.