invalid mount config for type "volume"

[yuyaogawa]

I get an error below when creating container. I am new to docker. Do you have any ideas to solve the error?

$ docker buildx inspect --bootstrap
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 4.4s done
#1 creating container buildx_buildkit_mybuilder0 0.1s done
#1 ERROR: Error response from daemon: invalid mount config for type "volume": invalid mount path: 'C:/ProgramData/buildkitd/.buildstate' mount path must be absolute
------
 > [internal] booting buildkit:
------
Name:   mybuilder
Driver: docker-container

Nodes:
Name:     mybuilder0
Endpoint: npipe:////./pipe/docker_engine
Error:    Error response from daemon: invalid mount config for type "volume": invalid mount path: 'C:/ProgramData/buildkitd/.buildstate' mount path must be absolute

Docker :

Client:
 Cloud integration: 1.0.17
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:58:50 2021
 OS/Arch:           windows/amd64
 Context:           desktop-linux
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:10 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

OS : Windows 10 Home

[spiro-c]

+1
I struggle for two days but if i make the builder from wsl2 the error is not present
from windows power shell

❯ docker --version
Docker version 20.10.8, build 3967b7d
❯ docker buildx inspect --bootstrap
[+] Building 0.0s (1/1) FINISHED
 => ERROR [internal] booting buildkit                                                             3.3s
 => => pulling image moby/buildkit:buildx-stable-1                                                3.3s
 => => creating container buildx_buildkit_mybuilder0                                              0.0s
------
 > [internal] booting buildkit:
------
Name:   mybuilder
Driver: docker-container

Nodes:
Name:     mybuilder0
Endpoint: npipe:////./pipe/docker_engine
Error:    Error response from daemon: invalid mount config for type "volume": invalid mount path: 'C:/ProgramData/buildkitd/.buildstate' mount path must be absolute

From Ubuntu WSL2

❯ uname -a
Linux DESKTOP-HS5PI9S 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
❯ docker --version
Docker version 20.10.8, build 3967b7d
❯ docker buildx inspect --bootstrap
[+] Building 4.5s (1/1) FINISHED
 => [internal] booting buildkit                                                                     4.4s
 => => pulling image moby/buildkit:buildx-stable-1                                                  3.7s
 => => creating container buildx_buildkit_wslbuildr0                                                0.7s
Name:   wslbuildr
Driver: docker-container

Nodes:
Name:      wslbuildr0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

[kmpm]

I can only confirm and also checked that then folder C:\ProgramData\buildkitd doesn't even exist.
Works from WSL2, Fails from PowerShell.

[tonistiigi]

@thaJeztah maybe you know why it would behave like this in windows client and what is the correct mount config in this cases. There are no bind mounts so there should never be any windows paths anywhere.

[thaJeztah]

Ah, yes, I think I see what's happening. Buildx uses a volume to store the cache for the "container" driver. For that, I see it's using some BuildKit defaults;

buildx/driver/docker-container/driver.go

Lines 110 to 116 in a8a3b17

	Mounts: []mount.Mount{
	{
	Type: mount.TypeVolume,
	Source: d.Name + volumeStateSuffix,
	Target: appdefaults.Root,
	},
	},

Those paths are platform-specific, i.e., buildkitd on Windows (once functional) would be using the Windows location;

buildx/vendor/github.com/moby/buildkit/util/appdefaults/appdefaults_windows.go

Lines 12 to 15 in a8a3b17

	var (
	Root = filepath.Join(os.Getenv("ProgramData"), "buildkitd", ".buildstate")
	ConfigDir = filepath.Join(os.Getenv("ProgramData"), "buildkitd")
	)

But buildx is running on the client and the docker-container driver creates a Linux container, so as a result, if the buildx client is running on Windows, it will pick the Windows app settings, an it will try to create a Linux container that has a volume mounted at C:/ProgramData/buildkitd/.buildstate, which, In Linux terms, is not a valid "absolute path" (doesn't start with /).

I think the correct thing to do would be to;

• use a fixed path for the destination (e.g. /buildkit-buildstate/), which would probably work both for "Linux" and "Windows" containers (that is, once we have a Windows native buildkitd that we can run in a container)
• configure buildkitd running inside the container to use that path

Or to use separate variables for Linux and Windows appdefaults, so that Buildx can pick the one needed for the container platform (currently: "Linux" only)

[thaJeztah]

@crazy-max ^^ probably introduced in #672

[thaJeztah]

I also notice that Windows uses a .buildstate subdirectory for this, whereas Linux uses /var/lib/buildkit. Should both use a subdirectory for this? (to be somewhat more aligned)

Ironically the Windows default is using a dot (.) directory, which seems a very Linux thing to do (to make it a hidden directory).

[thaJeztah]

Or to use separate variables for Linux and Windows appdefaults, so that Buildx can pick the one needed for the container platform (currently: "Linux" only)

I guess this would not work, because on Windows, the path depends on the %PROGRAMDATA% env-var, which should be resolved to %PROGRAMDATA% on the "destination end" (in this case, the container), so cannot be resolved the client side (unless we hard-code the env-var for the container as well).

[thaJeztah]

Let me open a quick PR