introduce compose support in podman

[typekpb]

/kind feature

Description

Introduce compose as part of podman CLI functionality

Steps to reproduce the issue:

1. running:

podman compose
Error: unrecognized command `podman compose`
Try 'podman --help' for more information.

Describe the results you received:
similar to docker compose command I'd expect compose file to be interpreted

docker compose

Usage:  docker compose [OPTIONS] COMMAND

Docker Compose

Options:
      --ansi string                Control when to print ANSI control characters ("never"|"always"|"auto") (default "auto")
      --env-file string            Specify an alternate environment file.
  -f, --file stringArray           Compose configuration files
      --profile stringArray        Specify a profile to enable
      --project-directory string   Specify an alternate working directory
                                   (default: the path of the Compose file)
  -p, --project-name string        Project name

Commands:
  build       Build or rebuild services
  convert     Converts the compose file to platform's canonical format
  cp          Copy files/folders between a service container and the local filesystem
  create      Creates containers for a service.
  down        Stop and remove containers, networks
  events      Receive real time events from containers.
  exec        Execute a command in a running container.
  images      List images used by the created containers
  kill        Force stop service containers.
  logs        View output from containers
  ls          List running compose projects
  pause       pause services
  port        Print the public port for a port binding.
  ps          List containers
  pull        Pull service images
  push        Push service images
  restart     Restart containers
  rm          Removes stopped service containers
  run         Run a one-off command on a service.
  start       Start services
  stop        Stop services
  top         Display the running processes
  unpause     unpause services
  up          Create and start containers

Run 'docker compose COMMAND --help' for more information on a command.

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.17
Built:        Mon Aug 30 21:15:26 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Mon Aug 30 22:46:14 2021
OS/Arch:      linux/amd64

Package info (e.g. output of rpm -q podman or apt list podman):
podman installed via brew on MacOS

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):
MacOS

[tinywrkb]

Please consider adding features from podman-compose. I'm interested in annotations, though I'm aware that it's supported by containers.conf.
Example usage:

services:
  SERVICE_NAME:
    annotations: 
     - "run.oci.keep_original_groups=1"

Maybe add systemd service files? I've been using these with podman-compose.

[mheon]

Given that docker compose is now integrated directly into Docker as a CLI plugin, we are investigating implementing that interface ourselves to continue to use the new version of upstream Compose against Podman. We don't feel we have the manpower to implement our own version of docker compose at this time, so our efforts will continue to be on ensuring docker compose runs smoothly and can be easily used with Podman. That said, if someone were to implement native compose support directly into Podman, we'd be happy to review and merge that.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[baude]

@rhatdan I think doing this will confuse our users r/n with all the options out there -- 'podman compose'; and we are also trying to push people to play kube. wdyt?

[sebdanielsson]

I think that supporting the compose spec would make it conciderably easier for more users to make the switch from Docker to Podman. Compose is awesome👍

[rhatdan]

Well docker-compose works with Podman socket now.
The question is whether or not we take compose and make it a plugin under podman compose command.

[mathstuf]

I think the ability to convert a compose file into a pod (with the features it supports such as systemd units) would be handy at least (even if the cli is not there). Probably a different issue?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Aposhian]

I plan to start working on this in my fork.

[muayyad-alsadi]

in docker you place binaries with exec flag in /usr/libexec/docker/cli-plugins/ or ~/.docker/cli-plugins named docker-COMMAND

we need to place podman-compose in /usr/libexec/podman/cli-plugins/

[rhatdan]

Only issue, is you would need to add support for CLI plugins to Podman then.

[muayyad-alsadi]

@rhatdan I've created an issue for that

[Aposhian]

Related: #13425

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[sebdanielsson]

/remove stale

[rhatdan]

@ashley-cui Intern?

[ashley-cui]

@karthikelango137

@rhatdan, can you add him to Containers so we can assign him stuff?

[karthikelango137]

got it

[alexandrevicenzi]

Podman can play Kubernetes files and there's a compose package/plugin to run Docker Compose.

Could the interface for both be consolidated? For example:

podman play kube and podman play compose or something similar.

#12475 already suggested improving the current play syntax.

[vrothberg]

@ashley-cui @rhatdan what is the task in this issue? I see a number of ideas floating around but think we need to iron out what exactly should be done.

[rhatdan]

I think we go back to the original which looks for docker CLI Plugins and executes them as part of the podman command.
Getting podman --help
To list them
podman PLUGIN --help
to execute the PLUGIN with an options after the PLUGIN passed to the plugin.

[johnthagen]

Is there a central location where it's documented how to use Docker Compose v2 with podman? On one hand, issues like

• support CLI plugins #13461 (comment)

make it seem like it's not supported, but then issues like:

• Compose V2 v2.2.3+: Cannot run docker-compose if a container uses build: #13889 (comment)

and blog posts such as

• https://www.redhat.com/sysadmin/podman-compose-docker-compose

The Docker Compose project had also not been idle. In 2021, the Docker Compose project unveiled a complete rewrite of the tool in Go, called Docker Compose v2.0 (older, v1.x versions are often called docker-compose, while v2.x are called docker compose). The rewrite required additional effort to work with Podman, arriving in Podman v4.1 in 2022 (though support for the Buildkit API is still pending).

make it sound that Docker Compose v2 is on some level supported. Is there any documentation on how someone can get started with this? Is it only on macOS? Can an Ubuntu machine be set up to work with it? I'd like to try to start tinkering with whatever level of support exists, but can't seem to find a central location where this is documented.

[rhatdan]

Currently Podman does not work with CLI Plugins, This has nothing to do with docker compose. You can run a docker-compose client against Podman Service and it works. CLI Plugins are being worked on but slowly.

[mheon]

Compose v2 should work with Podman in the same manner as Compose v1 (install podman-docker or set DOCKER_HOST to point to your Podman socket), with the added requirement that the DOCKER_BUILDKIT=0 environment variable must also be set.

[mheon]

There are no support restrictions, it will work anywhere Podman runs

[johnthagen]

I was able to make some progress on making end-to-end instructions for Docker Compose v2 with podman on Ubuntu 22.04.

Podman

https://podman.io/getting-started/installation

sudo apt-get -y update
sudo apt-get -y install podman

$ podman --version
podman version 3.4.4

Note: This version is likely too old. 4.1 looks to be the minimal version.

sudo apt install podman-docker

Docker Compose v2

https://docs.docker.com/compose/install/compose-plugin/#install-the-plugin-manually

sudo apt install curl

DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
mkdir -p $DOCKER_CONFIG/cli-plugins
curl -SL https://github.com/docker/compose/releases/download/v2.6.1/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose

chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose

Add DockerHub as default registry

Add the following line to /etc/containers/registries.conf:

unqualified-search-registries = ["docker.io"]

Pulling

At this point, running without sudo doesn't yet work?

~/.docker/cli-plugins/docker-compose pull
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": dial unix /var/run/docker.sock: connect: permission denied

But running with sudo works for pulling

sudo ~/.docker/cli-plugins/docker-compose pull
[+] Running 8/9
 ⠿ backend Skipped                                                                                                    0.0s
 ⠿ redis Pulled                                                                                                       2.9s
   ⠿ 8927eef71ab5 Already exists                                                                                      0.0s
   ⠿ e90389148883 Already exists                                                                                      0.0s
   ⠿ 953bc66946b3 Already exists                                                                                      0.0s
   ⠿ 2408cc74d12b Already exists                                                                                      0.0s
   ⠿ c6c08b6ea4d5 Already exists                                                                                      0.0s
   ⠿ cf69f9aa4ebe Already exists                                                                                      0.0s
   ⠋ 4733b14e6f4c Download complete  

Building

Building does not quite work:

DOCKER_BUILDKIT=0 sudo ~/.docker/cli-plugins/docker-compose build
[+] Building 0.0s (0/0)                                                                                                    
listing workers for Build: failed to list workers: Unavailable: connection error: desc = "transport: Error while dialing unable to upgrade to h2c, received 404"

Questions

• Can compose be made to work without sudo?
• Can the latest podman be installed in Ubuntu 22.04 easily? I worry I may be testing on too old of Podman (3.4.4), even though 22.04 is fairly new.
  • For example, Docker Engine supports a third-party repo that you can point to that has the latest Docker engine easily apt installable: https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository
  • I've seen references to Kubic repos, but then also a notice "Kubic packages have been discontinued for Ubuntu 22.04 LTS"
  • From this blog post: "The rewrite required additional effort to work with Podman, arriving in Podman v4.1 in 2022"
• Any ideas why build is failing? (Edit: likely due to not using Podman 4.1)

If I can get help making this all work, I'd be happy to try to contribute the results into a "Porting from Docker Engine" doc page / tutorial or similar. I think having this laid out as a tutorial top to bottom would really help new users to Podman like myself.

Related

• Ubuntu 22 apt-get install podman is only version 3.4.4 #14065 (comment)

[mheon]

For running without sudo, you need to export the DOCKER_HOST environment variable to point at the user in question's Podman API socket (as opposed to the root socket, which only root can access).

For the second question, I'm afraid I can't answer, that's a packaging issue and I'm not really familiar with Ubuntu.

For the third question, you need to set an environment variable (DOCKER_BUILDKIT=0) to force the old build API.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mathstuf]

Bump because it's still relevant.

[callahad]

As a stopgap, would it be reasonable to update the podman-docker compatibility shim to explicitly forward docker compose invocations to podman-compose? Something like:

diff --git a/docker b/docker
index efc363f..1087d03 100755
--- a/docker
+++ b/docker
@@ -1,4 +1,10 @@
 #!/bin/sh
 [ -e /etc/containers/nodocker ] || \
 echo "Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg." >&2
-exec /usr/bin/podman "$@"
+
+if [ "$1" = "compose" ]; then
+    shift
+    exec /usr/bin/podman-compose "$@"
+else
+    exec /usr/bin/podman "$@"
+fi

I don't how much the Docker CLI has changed between legacy docker-compose and the Compose plug-in and how much compatibility / maintenance risk the above suggestion would entail.

[rhatdan]

podman-compose and docker-compose are two different things, that both use the compose YAML files to create containers.

podman-compose is a wrapper around the podman command and executes podman commands under the hood. While docker-compose talks to the podman service socket and talks the Docker API to it and the podman service creates containers.

We have talked about using the compose plugin of docker to be executed by podman compose as a compatibility issue, but so far no one has taken this up.

[lstolcman]

Bump, simple compose.yaml does not work when podman (4.5.1) is combined with docker-compose (2.19.0):

services:
  base:
    image: alpine:latest
    command: sh -c "echo ok"

See details here #16092 (comment)

[vrothberg]

#19297 added a new compose command to Podman. Please checkout the description in #19297 for details.