"Error: lock … is not a read-only lock" failure with additionalimagestores set

[srcshelton]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Possibly related to #7309, if I have additionalstorage set in storage.conf I always get, e.g.:

$ sudo podman image pull --root /space/podman/images helloworld:latest 
Error: lock "/space/podman/images/overlay-images/images.lock" is not a read-only lock

(and the same with --storage-opt=overlay.mountopt=nodev also specified, as suggested in the above issue)

I've tried creating images.lock (and layers.lock) as empty files, and also copying over the (~64byte?) files from the default storage location.

If I temporarily edit containers.conf to swap the graphRoot and additionalstorage paths and then re-run the failing command (which is actually:

sudo podman container commit sys-build sys-build:latest

) and remove the --root option then the operation succeeds, but in the original default storage location rather than the original additional storage location.

What is the correct process to follow if I wish to maintain multiple storage locations to differentiate between (effectively) ephemeral images and images which should have longer life-spans?

Output of podman version:

Version:      3.1.0-rc2
API Version:  3.1.0-rc2
Go Version:   go1.16.2
Git Commit:   1b56ea2d9df82cbba2679f646c077881fefb49d6
Built:        Sat Mar 27 00:25:10 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.8
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: app-emulation/conmon-2.0.27
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: 65fad4bfcb250df0435ea668017e643e7f462155'
  cpus: 8
  distribution:
    distribution: gentoo
    version: unknown
  eventLogger: file
  hostname: dellr330
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.11.10-gentoo
  linkmode: dynamic
  memFree: 2663763968
  memTotal: 33390899200
  ociRuntime:
    name: crun
    package: app-emulation/crun-0.18
    path: /usr/bin/crun
    version: |-
      crun version 0.18
      commit: 808420efe3dc2b44d6db9f1a3fac8361dde42a95
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 25769787392
  swapTotal: 25769787392
  uptime: 8h 29m 51.14s (Approximately 0.33 days)
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: []
    Prefix: localhost:5000
  search:
  - docker.io
  - docker.pkg.github.com
  - quay.io
  - public.ecr.aws
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /space/podman/images
    overlay.mountopt: nodev
  graphRoot: /space/podman/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 107
  runRoot: /var/run/podman
  volumePath: /space/podman/volumes
version:
  APIVersion: 3.1.0-rc2
  Built: 1616804710
  BuiltTime: Sat Mar 27 00:25:10 2021
  GitCommit: 1b56ea2d9df82cbba2679f646c077881fefb49d6
  GoVersion: go1.16.2
  OsArch: linux/amd64
  Version: 3.1.0-rc2

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

[vrothberg]

@rhatdan PTAL

[srcshelton]

I've realised that if I pull in image into the primary storage location, terminate all container-related processes, replace the entire additional storage location tree with a copy of the primary one, and then attempt to launch containers then I can start containers from either location successfully.

However, attempting to pull into the additional storage location or to commit a container to this location as an image fails with the above locking error.

(I realise that the additional location is marked as 'read only' in, for example, podman image ls output - but I assume that there is supposed to be a way to populate images in the first place, short of having a separate system where the additional storage directory is its primary storage directory?)

[rhatdan]

I think you need to tell podman to not use the additional store if you are updating it directly. Could you attempt to override the mountoptions when you are updating?

[srcshelton]

I'm not entirely sure how I'd do that 😯... something like:

podman image pull --storage-opt=overlay.imagestore=/space/podman/images --root /space/podman/images <image>

?

If it helps, I was looking at https://www.redhat.com/sysadmin/image-stores-podman... has this intentionally changed since the article was written?

Update:

$ sudo podman image pull --storage-opt=overlay.imagestore=/space/podman/images --root /space/podman/images alpine:latest
Error: lock "/space/podman/images/overlay-images/images.lock" is not a read-only lock

... doesn't seem to help :(

[srcshelton]

(I'll admit, I'm not sure what the mountopt options are or what they do, but for the sake of completeness I also tried:

sudo podman image pull --storage-opt=overlay.mountopt=nodev --root /space/podman/images alpine:latest
sudo podman image pull --storage-opt=overlay.mountopt=dev --root /space/podman/images alpine:latest

... with the same result.)

[rhatdan]

I am working on multiple fixes for you.

containers/storage#867

With this change you would be able to do.

STORAGE_OPTS= ./bin/podman --root /var/lib/shared pull alpine
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob ca3cd42a7c95 [--------------------------------------] 0.0b / 0.0b
Copying config 49f356fa45 done
Writing manifest to image destination
Storing signatures
49f356fa4513676c5e22e3a8404aad6c7262cc7aaed15341458265320786c58c

[srcshelton]

I've applied the patch above, and whilst it helps I think there are edge-cases which aren't yet addressed…

It feels as if it would be helpful, as per my comment on #9911, if use of the --root option also implied the --storage-opt parameter as appropriate.

I'm not sure if there's a command-line option (via storage-opt) to add entries to additionalimagestores for the command being executed - the use-case here being that if the main image store is removed by --root=/elsewhere --storage-opt="" then it may need to be re-added as a (non-primary) image-store so that existing images may still be referred to.

In my particular case, I have a multi-stage build where an image is built from a Containerfile, this image is then run and the CMD script makes changes to the container filesystem, and then podman commit is used to save the resultant filesystem to the ultimate output image.

There doesn't appear to be a way even after the above patch to take a container run with one root and commit it to an image in a different root. A workaround for this might be to run the build-container in the additionalimagestores root, but this also fails and it appears to be because buildah/podman build isn't examining image locations from additionalimagestores when a FROM directive is encountered (regardless of --root/--storage-opt arguments).

The eventual aim here is to have an amount of RAM carved-out as a block device, with the filesystem on this ramdisk used as the default graphroot to store container layers and "ephemeral" images. At the same time, the traditional disk-based filesystem would be configured as an additionalimagestores path to provide for "permanent" image storage which must survive a reboot, but may still want to be regenerated on a weekly to monthly cadence.

With the above patch, podman --storage-opt="" --root /space/podman/images pull docker.io/alpine:latest does indeed show the alpine image in the R/O store - but building and committing is still an issue.

Also, image operations would be more obvious if they applied across all currently specified additionalimagestores:

$ sudo podman image ls alpine
REPOSITORY                TAG     IMAGE ID      CREATED     SIZE     R/O
docker.io/library/alpine  latest  49f356fa4513  7 days ago  5.88 MB  true
$ sudo podman image rm 49f356fa4513
Error: 1 error occurred:
        * identifier is not an image
$ sudo podman --storage-opt= --root /space/podman/images image rm 49f356fa4513
Untagged: docker.io/library/alpine:latest
Deleted: 49f356fa4513676c5e22e3a8404aad6c7262cc7aaed15341458265320786c58c

… when referring to the ID of the image specifically, I'd expect that the extra paths would also be searched and the first of these two removal commands to have succeeded.

[srcshelton]

(It looks as if Skopeo also hasn't been updated to handle additionalimagestores? The documented Image Names options only allow for containers-storage: without a way to specify which store - which appears to mean that an image can't be created in one store and then moved to another. Possibly oci:path referring to the image directory created by podman might do the trick, but would probably need containers.conf to be rewritten to make the additional store the primary store, unless Skopeo also supports --root/--storage-opt?)

[srcshelton]

This may well be my mistake, but I've just noticed that I in my current working directory I have:

$ sudo tree '"'
"
└── space
    └── podman
        └── images"
            ├── libpod
            │   └── bolt_state.db
            ├── mounts
            ├── overlay
            │   └── l
            ├── overlay-containers
            │   └── containers.lock
            ├── overlay-images
            │   └── images.lock
            ├── storage.lock
            ├── tmp
            └── userns.lock

10 directories, 5 files

… which suggests that something somewhere has misinterpreted a path enclosed in quotes as starting and ending with literal quote characters.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@srcshelton Any further update on this? Do you still believe this is a Podman issue?

[srcshelton]

As I've detailed in my three preceding comments, this never worked for me :(

I was able to run containers from images in a separate store, but was never able to pull images to any additional store through manipulation of command-line arguments (--storage-opt and --root) alone, or successfully build containers from a Containerfile containing a FROM directive referencing an image in any store other than the default.

I'm awaiting feedback on what else I might try, or for further patches to enhance or fix this feature.

[rhatdan]

Have you tried with the latest podman?
podman -v
podman version 3.2.0-rc1

# podman info | grep shared
overlay.imagestore: /var/lib/shared
# podman --storage-opt= --root /var/lib/shared images
REPOSITORY                TAG     IMAGE ID      CREATED      SIZE
docker.io/library/alpine  latest  6dbb9cc54074  4 weeks ago  5.88 MB
# podman images
REPOSITORY                TAG       IMAGE ID      CREATED      SIZE     R/O
quay.io/libpod/testimage  20210427  aadc32e2a626  2 weeks ago  7.75 MB  false
docker.io/library/alpine  latest    6dbb9cc54074  4 weeks ago  5.88 MB  true
# podman --storage-opt= --root /var/lib/shared pull fedora
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob 7679c09af385 done  
Copying config 3567369c67 done  
Writing manifest to image destination
Storing signatures
3567369c671193f96f057f76e3e136ecbd3fdc7065019cc3dd6ed5a96894f128
 podman images
REPOSITORY                         TAG       IMAGE ID      CREATED      SIZE     R/O
registry.fedoraproject.org/fedora  latest    3567369c6711  8 hours ago  184 MB   true
quay.io/libpod/testimage           20210427  aadc32e2a626  2 weeks ago  7.75 MB  false
docker.io/library/alpine           latest    6dbb9cc54074  4 weeks ago  5.88 MB  true
# podman version
Version:      3.2.0-rc1
API Version:  3.2.0-rc1
Go Version:   go1.16.3
Built:        Wed May  5 17:03:46 2021
OS/Arch:      linux/amd64

[srcshelton]

So, definitely an improvement:

I can:

• pull into an additionalimagestores location with podman --storage-opt= --root <additionalstore> pull <image>;
• build an image with a Containerfile referencing an image in an <additionalimagestores> location without any additional options.

I doesn't appear possible to:

• build an image into an <additionalimagestores> location if it relies on any image in the graphroot location, as specifying --storage-opt= --root <additionalimagestore> loses the original graphroot.

My suggestions would be:

• If there's no stand-alone use for --root alone, make --root imply --storage-opt= --root;
• add a --swap-root (or similar) option which would make the current graphroot an additionalimagestores location, and make the commandline-specified root the graphroot for this invocation - this would allow building from images in the main graphroot location whilst saving the results to the specified additionalimagestore directory.

(Alternatively, is there any syntax which allows '--storage-opt' to clear the current storage configuration but then provide a new additionalimagestores array so that the original graphroot could be supplied this way?)

See #10393.

Do we have a solution to the original issue, above, of:

[…] a way […] to take a container run with one root and commit it to an image in a different root.

e.g. how can an image and container running from the main graphroot be written to an additionalimagestores location using podman container commit?

[srcshelton]

It would also be useful to reconsider additionalimagestores as additional locations, rather than read-only locations.

So, for example, if I do a podman image ls and see a unique IMAGE ID for an image in an additional image store, then I feel I should be able to podman image rm <ID> (as the ID is unique) without having to specify --root and --storage-opt=.

See #10394.

(And possibly rename R/O in image ls output to Additional, or similar?)

See #10391.

[srcshelton]

It'd also be good if podman image prune and podman system prune could (with an appropriate option: --all-image-stores?) process any additional defined stores in addition to the main graphroot.

See #10392.

[rhatdan]

You are making too many suggestions in the same issue...

Anyways, I am thinking --root implying --storage-opt="" makes some sense, since this was fairly difficult to discover.

The original concept for --additional-stores was to serve them off of a network based file system, and allowing clients to modify them makes no sense, since they were likely to be share by other clients, and having images disappear would cause breakages in the other clients.

[srcshelton]

You are making too many suggestions in the same issue...

Happy to break things out into separate issues if it'd help!

[srcshelton]

The original concept for --additional-stores was to serve them off of a network based file system, and allowing clients to modify them makes no sense, since they were likely to be share by other clients, and having images disappear would cause breakages in the other clients.

… and I realise that this is a different use-case for the same feature - the question is whether it's a use-case you'd care to support.

Obviously my suggestion would be it's so close to also meeting the (possibly unique?) use-case I have - but then, I'm biased ;)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[m8ram]

Ran into the same issue.
My use case: several users on a system need access to the same set of images. (The AWS CLI container and others)
To preserve disk space I want to store them centrally.

Running as root I get the error described here when not using the --storage-opt option:

podman --root /build/shared-containers/ images
Error: error opening database /build/shared-containers/libpod/bolt_state.db: open /build/shared-containers/libpod/bolt_state.db: permission denied

Adding the option resolves the error for root:

# podman --storage-opt= --root /build/shared-containers/ images
REPOSITORY                TAG         IMAGE ID      CREATED     SIZE
docker.io/amazon/aws-cli  latest      c49b990ce098  3 days ago  362 MB

However for a regular user the error is still thrown:

$ podman --storage-opt= --root /build/shared-containers/ images
Error: error opening database /build/shared-containers/libpod/bolt_state.db: open /build/shared-containers/libpod/bolt_state.db: permission denied
$ podman run --rm --storage-opt= --root /build/shared-containers/ docker.io/amazon/aws-cli version
Error: error opening database /build/shared-containers/libpod/bolt_state.db: open /build/shared-containers/libpod/bolt_state.db: permission denied
$ ls -lZ /build/shared-containers/libpod/bolt_state.db
-rwxr-xr-x. 1 root root unconfined_u:object_r:default_t:s0 32768 Sep 20 15:42 /build/shared-containers/libpod/bolt_state.db

I'm using podman 3.2.3 (RHEL 8.4)

$ podman version
Version:      3.2.3
API Version:  3.2.3
Go Version:   go1.15.7
Built:        Tue Jul 27 09:29:39 2021
OS/Arch:      linux/amd64

[giuseppe]

My use case: several users on a system need access to the same set of images. (The AWS CLI container and others)
To preserve disk space I want to store them centrally.

this won't work, the unprivileged users won't be able to read any file in the shared store.

One way to make it work is to specify force_mask=shared in the storage options when creating the shared storage, then using fuse-overlayfs at runtime to convert the information stored in the user.containers.override_stat xattr to present the correct UID/GID/mode to the container.

Anyway this looks like a different issue, that would be better to track separately.

Since the original issue seems to be solved and the conversation moved to a different feature/issue, I am closing it. Let's discuss the new feature, if still needed, in a new issue

[rhatdan]

Giuseppe we need to work on a blog on how to set this up.