fs: enable to select the correct opaque xattr

[ktock]

Currently, stargz-snapshotter's filesystem adds both "trusted.overlay.opaque" and "user.overlay.opaque" to each opaque directory. But overlayfs uses either "trusted.overlay.opaque"(userxattr=false) or "user.overlay.opaque"(userxattr=true) for the opaque xattr and prevents it to be copied up to the upper. Thus the other (i.e. "user.overlay.opaque" when userxattr=false or "trusted.overlay.opaque" when userxattr=true) is treated as a normal xattr and can be copied up.

This behaviour of copying up of the xattr is problematic for BuildKit's overlayfs differ because it mistakenly treats a dir with the copied up opaque xattr as an opaque dir and creates unnecessary whiteout files. (this causes moby/buildkit#2720.)

This commit fixes this issue by allowing the caller of fs.NewFilesystem (e.g. BuildKit) to choose whether of "user.overlay.opaque" or "trusted.overlay.opaque" (or the both) to use for the opaque dir. If the caller uses userxattr option for overlayfs, it should specify "user.overlay.opaque". If not, should specify "trusted.overlay.opaque".

This issue can be reproduced with the example provided by moby/buildkit#2720.

# mkdir -p /tmp/ctx && cat <<EOF > /tmp/ctx/Dockerfile
FROM registry2-buildkit:5000/python:3.9-esgz as base
RUN pip --help
RUN echo hello_world
RUN ls /usr/local/lib/python3.9/site-packages/pip/_internal/cli/main.py
EOF
# nerdctl build --cache-to type=inline --output type=image,name=registry2-buildkit:5000/repo:something,push=true /tmp/ctx

(clear cache here)

# sed -i 's/hello_world/hello_world_/' /tmp/ctx/Dockerfile
# nerdctl build --cache-from registry2-buildkit:5000/repo:something /tmp/ctx

The last build fails:

[+] Building 3.3s (9/9) FINISHED                                                                                                                                                                                                                                                         
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                0.1s
 => => transferring dockerfile: 201B                                                                                                                                                                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                   0.1s
 => => transferring context: 2B                                                                                                                                                                                                                                                     0.0s
 => [internal] load metadata for registry2-buildkit:5000/python:3.9-esgz                                                                                                                                                                                                            0.1s
 => [auth] sharing credentials for registry2-buildkit:5000                                                                                                                                                                                                                          0.0s
 => importing cache manifest from registry2-buildkit:5000/repo:something                                                                                                                                                                                                            0.0s
 => [1/4] FROM registry2-buildkit:5000/python:3.9-esgz@sha256:41b443dadc9900a26845b974e83a33666cd9de9088304a108a5b448776322007                                                                                                                                                      0.0s
 => => resolve registry2-buildkit:5000/python:3.9-esgz@sha256:41b443dadc9900a26845b974e83a33666cd9de9088304a108a5b448776322007                                                                                                                                                      0.0s
 => [2/4] RUN pip --help                                                                                                                                                                                                                                                            0.7s
 => => extracting sha256:bdfd511c91ed6482cc7c41d00e906b26162c4f433ef4aa5d031da646cd924a1a                                                                                                                                                                                           0.1s
 => [3/4] RUN echo hello_world_                                                                                                                                                                                                                                                     0.7s
 => ERROR [4/4] RUN ls /usr/local/lib/python3.9/site-packages/pip/_internal/cli/main.py                                                                                                                                                                                             0.8s
                                                                                                                                                                                                                                                                                         
------
 > [4/4] RUN ls /usr/local/lib/python3.9/site-packages/pip/_internal/cli/main.py:
#0 0.698 ls: cannot access '/usr/local/lib/python3.9/site-packages/pip/_internal/cli/main.py': No such file or directory
------
Dockerfile:4
--------------------
   2 |     RUN pip --help
   3 |     RUN echo hello_world_
   4 | >>> RUN ls /usr/local/lib/python3.9/site-packages/pip/_internal/cli/main.py
   5 |     
--------------------
error: failed to solve: process "/bin/sh -c ls /usr/local/lib/python3.9/site-packages/pip/_internal/cli/main.py" did not complete successfully: exit code: 2

The exported cache contains an unnecessary whitout for usr/local/lib/python3.9/site-packages/pip/_internal/cli/main.py.

# crane blob registry2-buildkit:5000/repo:something@sha256:48cb6bf9990853b88d3a7fd3debd1a6399b4f276b48ed13a5a079137c3cb74b7 | tar -z --list
...
usr/local/lib/python3.9/site-packages/pip/_internal/cli/.wh.main.py
...