podman does not support Windows Subsystem for Linux lxfs with overlay driver

[sunnychanwork]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. Install ubuntu on Windows Subsystem for Linux

2. Run "podman info --log-leve=debug"

Describe the results you received:

DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /var/run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay test mount with multiple lowers failed mountfrom re-exec error: exit status 1: output: invalid argument
DEBU[0000] overlay test mount with a single lower failed mountfrom re-exec error: exit status 1: output: invalid argument
ERRO[0000] 'overlay' is not supported over at "/var/lib/containers/storage/overlay"
ERRO[0000] could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Windows Subsystem for Linux's lxfs does support overlay:

root@HOST:~# mount | grep overlay
none on /tmp/overlay type overlay (rw,relatime,lowerdir=/tmp/lower,upperdir=/tmp/upper,workdir=/tmp/workdir)
root@HOST:~# uname -a
Linux HOST 4.4.0-17134-Microsoft #706-Microsoft Mon Apr 01 18:13:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

Output of podman version:

Version: 1.3.0-dev
RemoteAPI Version: 1
Go Version: go1.10.4
OS/Arch: linux/amd64

Output of podman info --debug:

As above

Additional environment details (AWS, VirtualBox, physical, etc.):
Using Windows Subsystem for Linux (17134)

[rhatdan]

I have no idea if this would work period, but container/storage underneath podman is attempting to setup Overlay on this underlying file system and it finds that Overlay is not supported by the kernel on that file system there is little we can do.

You could try to use the VFS storage driver to see if this works.

[mheon]

My suspicion is that Windows doesn't implement a lot of the namespace-related functionality of the Linux kernel, so I suspect that we won't actually be able to launch containers, even if we can get storage configured.

[sunnychanwork]

@rhatdan Overlay fs does work as I showed in the example shell output. However, I wanted to know what's mount options you use to check such that the overlay is not available? At least

Docker daemon does work and run containers under WSL (see microsoft/WSL#2291 (comment)) except some networking feature due to missing iptables calls so I think it has all the namespace-related functionality available. It would be very useful for Enterprise developer where they only have Windows available.

[mheon]

I stand corrected. That's actually damn impressive.

It's possible that we have a kernel version check in c/storage somewhere that's tripping because of the output of WSL?

[sunnychanwork]

This is what I see in the podman info debug output:

DEBU[0000] overlay test mount with multiple lowers failed mountfrom re-exec error: exit status 1: output: invalid argument

DEBU[0000] overlay test mount with a single lower failed mountfrom re-exec error: exit status 1: output: invalid argument

Is there anyway to know what "mount" command it is trying to execute?

[sunnychanwork]

I have scanned the code for the error message and it looks like it is in the storage component - so I have create an issue there for tracking.

[giuseppe]

root@HOST:~# mount | grep overlay
none on /tmp/overlay type overlay (rw,relatime,lowerdir=/tmp/lower,upperdir=/tmp/upper,workdir=/tmp/workdir)

the overlay you've showed here is using only one lower layer. Could you try to mount overlay using multiple lower layers (e.g. lowerdir=/tmp/lower1:/tmp/lower2)?

[giuseppe]

@sunnychanwork had a chance to try my suggestion?

[sunnychanwork]

Yes it does work:
none on /tmp/overlay type overlay (rw,relatime,lowerdir=/tmp/lower:/tmp/lower2,upperdir=/tmp/upper,workdir=/tmp/workdir)
root@xxx:/tmp/overlay# ls /tmp/lower file.lower root@xxx:/tmp/overlay# ls /tmp/lower2 file.lower2 root@xxx:/tmp/overlay# ls /tmp/overlay file.lower file.lower2 root@xxx:/tmp/overlay# uname -a Linux xxx 4.4.0-17134-Microsoft #706-Microsoft Mon Apr 01 18:13:00 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

[nalind]

This should be fixed in libpod 1.3.0 or later, by way of its inclusion of containers/storage 1.12.5 or later, which fixed containers/storage#327.

[rhatdan]

@sunnychanwork Are you still looking at this? I was wondering if current master is any better. We are working on cutting a podman 1.5.0 right now, it would be nice to see if it works on MS.

[sunnychanwork]

@rhatdan Microsoft since had a change of heart and announced WSL2 which runs Linux under a VM in Windows - so I didn't spend any more time looking at this for now

[kunalkushwaha]

I took look at this issue.

The filesystem issue seems to be fixed. Though error are for journald & network.
With wsl2, we will have native kernel running, I think these issues will be fixed with that.

We can close this issue.

$ sudo -E podman --log-level warn run -it --net=none  docker.io/library/alpine sh
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
ERRO[0000] unable to write pod event: "could not initialize socket to journald"
WARN[0000] Failed to add conmon to systemd sandbox cgroup: dial unix /run/systemd/private: connect: no such file or directory
Error: systemd cgroup flag passed, but systemd support for managing cgroups is not available: OCI runtime error
WARN[0000] unable to find /etc/containers/registries.conf. some podman (image shortnames) commands may be limited

~ ⌚ 17:12:37
$ sudo -E podman --log-level warn run -it   docker.io/library/alpine sh
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
ERRO[0000] unable to write pod event: "could not initialize socket to journald"
Error: error creating network namespace for container 0f8583c60cc0aa17fbac217a2154a21eec2d5b305384efe433300201f841409a: failed to create namespace:
 permission denied
WARN[0000] unable to find /etc/containers/registries.conf. some podman (image shortnames) commands may be limited

[sunnychanwork]

Just FYI, WSL1 doesn't work with systemd and therefore jorunald is not installed by default.