Some incompatibility between Docker Compose v2 and Mutagen

[evox95]

Platform

Mutagen version

0.12.0-beta8

Docker version

Client:
Cloud integration: 1.0.17
Version: 20.10.8
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:55:20 2021
OS/Arch: darwin/amd64
Context: default
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.8
API version: 1.41 (minimum version 1.12)
Go version: go1.16.6
Git commit: 75249d8
Built: Fri Jul 30 19:52:31 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b63
docker-init:
Version: 0.19.0
GitCommit: de40ad0

Problem

I am using the x-mutagen extension for docker-compose.yml. Everything was working fine until a certain Docker update. Running the mutagen compose -f docker-mutagen.yml up command started returning an error when trying to connect to a Docker volume:

[+] Running 1/0
⠿ Container myapp-mutagen-1 Running 0.0s
Resuming existing forwarding sessions
Resuming existing synchronization sessions
Creating synchronization session "app-sync"
Connecting to agent (POSIX)...
Error: unable to reconcile Mutagen sessions: unable to create synchronization session (app-sync): unable to connect to beta: unable to connect to endpoint: unable to dial agent endpoint: unable to create agent command: unable to probe container: container probing failed under POSIX hypothesis (exit status 1) and Windows hypothesis (exit status 1)

I reinstalled Docker and Mutagen, but it didn't help.

Resolution

I had to disable the 'Use Docker Compose V2' option in the Docker settings. After a reboot, everything started working again. There must be some incompatibility between Docker Compose v2 and Mutagen.

Mutagen is a great piece of software without which my work with Docker on macOS would be nearly impossible.

[xenoscopic]

Hi @evox95, this is a known issue, unfortunately without any workarounds at the moment. I've been meaning to update the docs, but haven't gotten around to it just yet.

Now that Compose V2 is GA, I'm working on support for it in a separate repository here. I'm working to get that shipped ASAP, probably in a beta form sometime next week. It should be significantly more robust than the current Mutagen Compose support since it will embed Compose V2 directly.

I'll leave this issue open for now as a tracking issue, as I don't think we have one yet.

[xenoscopic]

Mutagen Compose (i.e. Compose V2 support) is now available for testing:

https://github.com/mutagen-io/mutagen-compose/releases/tag/v0.13.0-beta2

This is a complete rewrite that uses the new Go-based Docker Compose V2 implementation directly (rather than emulating and wrapping the Python-based docker-compose command). I'm hoping this will replace Mutagen's internal Compose V1 integration (i.e. the mutagen compose command) as the recommended option within the next week, depending on the initial feedback. Compose V1 integration will be removed when Mutagen v0.13 is released.

Since this will now be an external project, I'd like to keep Compose-related issues in the Mutagen Compose issue tracker.

Here are the key details:

• Initial draft docs are here
  • Be sure to read the BuildKit note if using a remote Docker engine
• The design and configuration are the same as the Compose V1 integration
• Mutagen Compose is a standalone Compose implementation, so you don't need Docker Compose to use it (though you'll still need the docker command available for Mutagen to use)
• Mutagen Compose is a standalone command (mutagen-compose). The mutagen compose subcommand still points to the Compose V1 integration (and will probably become an alias for mutagen-compose once Compose V1 support goes away).
• You'll need a Mutagen version that matches the Mutagen Compose version (currently v0.13.0-beta2)
  • The best option at the moment is just to use Homebrew: brew install mutagen-io/mutagen/mutagen-beta mutagen-io/mutagen/mutagen-compose-beta. The versions will be kept in-sync in Mutagen release channels. Note that Homebrew is a bit goofy, so if you already have the mutagen-beta formula installed this way, then do brew update && brew install mutagen-compose-beta, otherwise it'll try to download casks.
  • You can also just download the binaries from the releases page
• Compose V1 integration is still present in Mutagen (and will be throughout v0.13.0-beta releases)
• Mutagen Compose removes most of the limitations of the Compose V1 integration (e.g. run should work)
• Mutagen Compose brings two new optimizations (also inherited by the Compose V1 integration in v0.13.0-beta2):
  • If a synchronization session targets an empty volume root and a non-default owner, group, or directory mode setting has been specified, then Mutagen will set the permissions of the volume root to match. This is a heuristic to work around the fact that Docker volumes don't have an easy way to set permissions upon creation.
  • If no file staging mode is explicitly set in the container, then Mutagen will stage files into a temporary directory in their target volumes, avoiding the need for file copy operations when applying changes. If this behavior is undesirable (e.g. due to conflicts with filesystem watchers), then you can choose to stage in the sidecar container filesystem itself by selecting the mutagen staging mode.

Hopefully that takes care of the Compose V2 support from the Mutagen side. Now it's just a matter of testing, feedback, and releasing a final version alongside Mutagen v0.13.