database configuration mismatch with '--tmpdir=/tmp/user/$USER/libpod/tmp'

[mrinaldhillon]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Not able to change path to the tmp directory for libpod state content

Steps to reproduce the issue:

1. podman --tmpdir=/tmp/user/$USER/libpod/tmp info --debug

Describe the results you received:
podman --tmpdir=/tmp/user/$USER/libpod/tmp info --debug
Error: could not get runtime: database libpod temporary files directory (tmpdir) "/run/user/1000/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/tmp/user/mdhillon/libpod/tmp": database configuration mismatch

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):
Same issue is reproducible on podman v1.8.1
Output of podman version:

podman version 1.8.0

Output of podman info --debug:

podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.13.6
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.1-1.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.1, commit: 5e0eadedda9508810235ab878174dca1183f4013'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 3387842560
  MemTotal: 16763703296
  OCIRuntime:
    name: crun
    package: crun-0.12.2.1-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.12.2.1
      commit: cd7cea7114db5f6aa35fbb69fa307c19c2728a31
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 33764929536
  SwapTotal: 33780920320
  arch: amd64
  cpus: 16
  eventlogger: journald
  hostname: localhost.localdomain
  kernel: 5.3.7-301.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 543h 54m 19.25s (Approximately 22.62 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /home/mdhillon/.config/containers/storage.conf
  ContainerStore:
    number: 94
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.6.4-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.6.4
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/mdhillon/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 154
  RunRoot: /run/user/1000
  VolumePath: /home/mdhillon/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.0-2.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
NAME=Fedora
VERSION="31 (Workstation Edition)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora 31 (Workstation Edition)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f31/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation

[mheon]

The reason for this error is that, if we move part of Libpod's state but not all of it (the static content - database, container images, etc) we can end up with undesired behavior. We'd end up with content from running containers in the old temporary directory missing, for example. Furthermore, Libpod detects reboots by checking a file in the temporary directory, so assigning a new directory would invoke our reboot handling, which would wipe the state on the assumption that all running containers cannot be running anymore because the system rebooted.

Can I ask what you're trying to do here? There are ways around this, but knowing what's needed would help understand how.

[mrinaldhillon]

Unprivileged users need to run rootless container in kubernetes pods. Users don't have access permissions to /run/user/$(id -u) path due to security policy. So we need to change this to /tmp/user/$(id -u) .

[mheon]

@rhatdan Do we have anything in containers.conf to fix this?

For now, I recommend that you remove the libpod database and files at ~/.local/shared/containers/storage/libpod (which will also remove all containers for the user - unfortunate, but necessary to ensure no breakage). Then, run one Podman command with --tmp-dir set - this will write the new location in the database when it is recreated, and all subsequent Podman calls will use the new path.

You may also wish to set --runroot - I think that may also live in /run/user by default

[mrinaldhillon]

Still the same issue: could you please test on your setup

Error: could not get runtime: database libpod temporary files directory (tmpdir) "/run/user/1000/libpod/tmp" does not match our libpod temporary files directory (tmpdir) "/tmp/user/mdhillon/libpod/tmp": database configuration mismatch

[mheon]

Something must be recreating the database before the Podman command. Can you verify that ~/.local/share/containers/storage/libpod/bolt_state.db exists? I wonder if your Podman isn't using another directory for the database

[mrinaldhillon]

Your are correct I am using /b/workspace/.local/share/containers/storage/libpod :) . tmpdir issue got fixed by cleaning up this path. But now I am stuck on this #5488