storage.conf mishandling with zfs storage driver #20324

ajakk · 2023-08-20T02:01:35Z

Issue Description

Using this storage.conf:

[storage]
driver = "zfs"
graphroot = "/var/lib/containers/storage"
#runroot = "/var/lib/containers/runroot"

[storage.options.zfs]
fsname = "zpool/podman"

There seems to be no default for runroot:

~/podman # ./bin/podman system info
ERRO[0000] runroot must be set

Uncommenting the runroot option does enable me to use that command, but shows runroot and graphroot as being the current working directory (output heavily abbreviated, full output below):

sirius ~/podman # ./bin/podman system info
store:
  configFile: /etc/containers/storage.conf
  graphDriverName: zfs
  graphOptions:
    zfs.fsname: zpool/podman
  graphRoot: /root/podman
  runRoot: /root/podman
  volumePath: /var/lib/containers/storage/volumes

Steps to reproduce the issue

Clone podman. reproduced with 20f28e5 and Gentoo's 4.5.0
make
Use bin/podman

Describe the results you received

Podman litters the current working directory with zfs-related directories and files:

~/podman # git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        defaultNetworkBackend
        storage.lock
        userns.lock
        zfs-containers/
        zfs-images/
        zfs-layers/

nothing added to commit but untracked files present (use "git add" to track)

Describe the results you expected

Podman should respect the configuration file, and provide a sane default for the runroot.

podman info output

~/podman # ./bin/podman info
host:
  arch: amd64
  buildahVersion: 1.32.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: app-containers/conmon-2.1.7
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.7, commit: v2.1.7'
  cpuUtilization:
    idlePercent: 99.55
    systemPercent: 0.26
    userPercent: 0.19
  cpus: 24
  databaseBackend: boltdb
  distribution:
    distribution: gentoo
    version: "2.14"
  eventLogger: journald
  freeLocks: 2045
  hostname: sirius
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.1.41-gentoo-dist-hardened
  linkmode: dynamic
  logDriver: journald
  memFree: 17448341504
  memTotal: 33621889024
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
    package: app-containers/cni-plugins-1.2.0
    path: /opt/cni/bin
  ociRuntime:
    name: crun
    package: app-containers/crun-1.8.1
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: app-containers/slirp4netns-1.2.0
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.4
  swapFree: 17179865088
  swapTotal: 17179865088
  uptime: 148h 41m 49.00s (Approximately 6.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
  - registry.fedoraproject.org
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: zfs
  graphOptions:
    zfs.fsname: zpool/podman
  graphRoot: /root/podman
  graphRootAllocated: 964842221568
  graphRootUsed: 350250168320
  graphStatus:
    Compression: lz4
    Parent Dataset: zpool/podman
    Parent Quota: "no"
    Space Available: "81018269696"
    Space Used By Parent: "6990344192"
    Zpool: zpool
    Zpool Health: ONLINE
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /root/podman
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.0-dev
  Built: 1692496059
  BuiltTime: Sat Aug 19 18:47:39 2023
  GitCommit: 20f28e538d21aef62eb8159e6689e4e71ade0b87
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.0-dev

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

The text was updated successfully, but these errors were encountered:

Luap99 · 2023-08-21T08:53:20Z

@giuseppe PTAL

giuseppe · 2023-08-21T12:39:53Z

from the title, it seems the issue happens only with zfs. Can you confirm that?

ajakk · 2023-08-24T03:35:36Z

ZFS is the only storage backend I've reproduced with, but I haven't tried with more than that and the default.

giuseppe · 2023-08-24T09:35:18Z

I've never tried the ZFS backend myself and not sure in what state it is.

Do you have any particular reason for using it instead of overlay?

ajakk · 2023-08-24T15:19:41Z

Mostly because it's easier to manage ZFS datasets rather than a directory on my root filesystem. And I get the niceties that ZFS enables, like snapshots.

github-actions · 2023-09-24T00:06:57Z

A friendly reminder that this issue had no activity for 30 days.

ajakk · 2023-09-24T17:12:08Z

A friendly reminder that this issue had no activity for 30 days.

And yet, no sign of a fix...

giuseppe · 2023-09-24T18:10:13Z

nobody from the core team is working on the zfs backend, since overlay is what we support and suggest people to use.

There won't likely be any update unless someone steps up and look into it

rhatdan · 2023-09-24T23:08:20Z

Most likely a container/storage bug so transferring over there. If you are interested in opening a PR to fix, it would be welcome.

shlande · 2023-10-03T08:07:43Z

Also encountered this problem. it seems like a podman bug or just a wrong configuration.

   363:		// Grab config from the database so we can reset some defaults
   364:		dbConfig, err := runtime.state.GetDBConfig()
=> 365:		if err != nil {
   366:			if runtime.doReset {
   367:				// We can at least delete the DB and the static files
   368:				// directory.
   369:				// Can't safely touch anything else because we aren't
   370:				// sure of other directories.
(dlv) print dbConfig
*github.com/containers/podman/v4/libpod.DBConfig {
	LibpodRoot: "/var/lib/containers/storage/libpod",
	LibpodTmp: "/run/libpod",
	StorageRoot: ".",
	StorageTmp: ".",
	GraphDriver: "zfs",
	VolumePath: "/var/lib/containers/storage/volumes",}

Podman runtime uses boltDB to store some states, which include those wrong storageRoot and storageTmp. These args then are passed to the zfs driver and lead to this strange behavior.

In my case, after deleting the blotDB file (/var/lib/containers/storage/libpod/bolt_state.db), everything is back to normal.

~~Having no idea how these blotDB records are created~~

reproduced it in version 4.4.1. Podman leaves a "." record in bolt_state.db when runRoot is empty, which can affect even upgraded.

rhatdan · 2023-10-10T22:58:24Z

@mheon PTAL

mheon · 2023-10-10T23:41:31Z

Bolt here is placing the configured root and runroot into the DB, without doing any validation on them; so I suspect that here, it's only showing that we have bad values for storage config. @shlande What does your storage.conf look like?

shlande · 2023-10-13T16:10:52Z

same as ajakk, version 4.4.1

[storage]
driver = "zfs"
graphroot = "/var/lib/containers/storage"
#runroot = "/var/lib/containers/runroot"

[storage.options.zfs]
fsname = "storage"

in higher version podman may complain this config.

mheon · 2023-10-13T17:21:42Z

I think this is probably a ZFS specific issue, and probably in c/storage because of that. Graphroot is explicitly specified, and somehow has been reset to CWD by the time Podman has finished initializing.

ajakk · 2023-10-14T17:34:11Z

Yes, I've only seen this with the ZFS storage driver.

virtorgan · 2023-12-09T19:29:40Z

Just installed podman on a fresh debian 12 :
apt install podman result in :
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.

then looked at log messages from services: systemctl status podman
Error: 'overlay' is not supported over zfs, a mount_program is required:

then looked for /etc/containers/storage.conf : not existing
same for ~/.config/containers

Does it mean Podman is not compatible with zfs & btrfs file systems ?

mheon · 2023-12-10T00:41:38Z

It does not. I use Podman over btrfs every day using the overlay driver. You are not using the ZFS graph driver, so your issue is unrelated to this one. Please open a new bug and fill out the full bug template.

tazmo · 2023-12-11T17:20:20Z

With the recent addition of overlayfs support in openzfs-2.2.0, any chance this could be addressed now?

mheon · 2023-12-11T19:58:07Z

No, this is specific to the ZFS graphdriver which does not use overlayfs

h0tw1r3 · 2023-12-24T08:18:57Z

@tazmo easy to do already
Create a mount helper script /usr/local/bin/overmount:

#!/bin/sh
/bin/mount -t overlay overlay $*

Then add to /etc/containers/storage.conf:

[storage.options]
mount_program = "/usr/local/bin/overmount"

DFINITYManu · 2024-08-27T11:00:34Z

That suggestion doesn't work @h0tw1r3 :

 user   master ⚑ 4  ~  src  ic  SIGINT  sudo podman image exists ...
[sudo] password for user: 
WARN[0000] Storage configuration is unset - using hardcoded default graph root "/var/lib/containers/storage" 
WARN[0000] Storage configuration is unset - using hardcoded default graph root "/var/lib/containers/storage" 
WARN[0000] Storage configuration is unset - using hardcoded default graph root "/var/lib/containers/storage" 
Error: Unknown option zfs.mount_program

giuseppe · 2024-08-27T11:07:21Z

mount_program is specific to overlay. It has no effect with other drivers

ajakk added the kind/bug Categorizes issue or PR as related to a bug. label Aug 20, 2023

giuseppe mentioned this issue Sep 6, 2023

podman system reset deletes user directory?!? #19870

Closed

github-actions bot added the stale-issue label Sep 24, 2023

rhatdan transferred this issue from containers/podman Sep 24, 2023

rhatdan removed the stale-issue label Oct 10, 2023

rhatdan transferred this issue from containers/storage Oct 10, 2023

tteck mentioned this issue Dec 8, 2023

ERRO[0000] [graphdriver] prior storage driver overlay failed: 'overlay' is not supported over zfs, a mount_program is required tteck/Proxmox#2167

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

storage.conf mishandling with zfs storage driver #20324

storage.conf mishandling with zfs storage driver #20324

ajakk commented Aug 20, 2023 •

edited

Loading

Luap99 commented Aug 21, 2023

giuseppe commented Aug 21, 2023

ajakk commented Aug 24, 2023

giuseppe commented Aug 24, 2023

ajakk commented Aug 24, 2023

github-actions bot commented Sep 24, 2023

ajakk commented Sep 24, 2023 •

edited

Loading

giuseppe commented Sep 24, 2023

rhatdan commented Sep 24, 2023

shlande commented Oct 3, 2023 •

edited

Loading

rhatdan commented Oct 10, 2023

mheon commented Oct 10, 2023

shlande commented Oct 13, 2023

mheon commented Oct 13, 2023

ajakk commented Oct 14, 2023

virtorgan commented Dec 9, 2023

mheon commented Dec 10, 2023

tazmo commented Dec 11, 2023

mheon commented Dec 11, 2023

h0tw1r3 commented Dec 24, 2023

DFINITYManu commented Aug 27, 2024

giuseppe commented Aug 27, 2024

storage.conf mishandling with zfs storage driver #20324

storage.conf mishandling with zfs storage driver #20324

Comments

ajakk commented Aug 20, 2023 • edited Loading

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

Luap99 commented Aug 21, 2023

giuseppe commented Aug 21, 2023

ajakk commented Aug 24, 2023

giuseppe commented Aug 24, 2023

ajakk commented Aug 24, 2023

github-actions bot commented Sep 24, 2023

ajakk commented Sep 24, 2023 • edited Loading

giuseppe commented Sep 24, 2023

rhatdan commented Sep 24, 2023

shlande commented Oct 3, 2023 • edited Loading

rhatdan commented Oct 10, 2023

mheon commented Oct 10, 2023

shlande commented Oct 13, 2023

mheon commented Oct 13, 2023

ajakk commented Oct 14, 2023

virtorgan commented Dec 9, 2023

mheon commented Dec 10, 2023

tazmo commented Dec 11, 2023

mheon commented Dec 11, 2023

h0tw1r3 commented Dec 24, 2023

DFINITYManu commented Aug 27, 2024

giuseppe commented Aug 27, 2024

ajakk commented Aug 20, 2023 •

edited

Loading

ajakk commented Sep 24, 2023 •

edited

Loading

shlande commented Oct 3, 2023 •

edited

Loading