systemd zfs.target service doesn't start on Debian Stretch with custom kmod packages

[jwittlincohen]

Distribution Name | Debian
Distribution Version | Stretch (9.1)
Linux Kernel | 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux
Architecture | AMD64
ZFS Version | 0.7.0-1
SPL Version | 0.7.0-1

Describe the problem you're observing

I followed the instructions on the ZFS Custom Packages Wiki to build custom kmod packages of ZFS/SPL 0.7.0 for Debian Stretch. I am using ZFS for my root pool and was able to get a working initramfs by following the Debian on Root ZFS guide which instructs the creation of a file at /usr/share/initramfs-tools/conf.d/zfs (this file and its content is included in Debian's binary packages). While the system mounts my / partition, systemd fails to start the "zfs.target" service, which prevents the automatic mounting of other partitions, such as /home. I see the following error messages from dmesg and systemctl:

root@debian-vm:~# dmesg | grep zfs
[    3.850446] systemd[1]: zfs.target: Cannot add dependency job, ignoring: Unit zfs.target is masked.

root@debian-vm:~# systemctl status zfs.target
● zfs.target
   Loaded: masked (/dev/null; bad)
   Active: inactive (dead)

In contrast, here is the systemctl output on a Debian Stretch system with Debian's packaged DKMS binaries:

jason@jason-desktop:~$ sudo systemctl status zfs.target
● zfs.target - ZFS startup target
   Loaded: loaded (/lib/systemd/system/zfs.target; enabled; vendor preset: enabled)
   Active: active since Mon 2017-07-31 19:01:32 EDT; 19min ago

Describe how to reproduce the problem

1. Install Debian Stretch on a VM or on physical hardware (I have tested with both) using the Debian Jessie ZFS on Root Guide. Instructions must be altered slightly for Debian Stretch. This guide will setup a working Debian Stretch system with a ZFS root pool using Debian's packages for ZFS/SPL 0.6.5.9.

2. Uninstall all Debian SPL and ZFS packages

3. Follow the instructions from the wiki to build custom SPL and ZFS 0.7.0 packages. Install resulting packages

4. To ensure a working initramfs create the following file. This step is not required for the binaries included with Stretch, but is for the custom kmod packages. Update the initramfs:

# vi /usr/share/initramfs-tools/conf.d/zfs
for x in $(cat /proc/cmdline)
do
    case $x in
        root=ZFS=*)
            BOOT=zfs
            ;;
    esac
done

5. Reboot. Only the / partition will be mounted. /home and /home/user will not be mounted.

[Conan-Kudo]

@jwittlincohen This makes it sound like that zfs.target is actually masked by the Debian system, and the Debian packages themselves unmask it during installation. Can you check to see if any zfs unit files show up on the system and if they're symlinks to /dev/null? What your report says indicates that the zfs.target is specifically blocked by default.

[jwittlincohen]

@Conan-Kudo Thanks for the suggestion. On my Debian Stretch system with the Debian-packaged ZFS binaries, /lib/systemd/system contains several zfs unit files including zfs.target. However, on my VM with the custom kmod packages of ZFS 0.7.1, /lib/systemd/system contains no zfs.target or any unit files with "ZFS" in the name.

[Conan-Kudo]

@jwittlincohen Can you check to see if they may be in /usr/lib/systemd/system instead?

[jwittlincohen]

Nope, I see nothing relevant in /usr/lib/systemd/system. The zfs.target and other ZFS related unit files are packaged by Debian in the zfsutils-linux package. I do not see that package on the VM where I installed the custom kmod packages. I checked the folder where I built the custom binaries and there is no zfsutils-linux binary. I see packages entitled zfs, zfs-dracut, zfs-initramfs, zfs-test, libuutil1, libzfs2, libzfs2-devl, libzpool2, libnvpair, kmod-zfs-devel, and kmod-zfs.

[Conan-Kudo]

@jwittlincohen Huh, I think I know what's happened here. I bet it's installing the sysvinit scripts for Debian/Ubuntu targets.

[jwittlincohen]

@Conan-Kudo Your suspicion appears to be correct. I see init scripts for zfs-import, zfs-mount, zfs-share and zfs-zed in /etc/init.d/ on the VM. My Debian Stretch install with the Debian packages do not contain any ZFS init scripts in /etc/init.d.

[behlendorf]

@jwittlincohen I've proposed a patch in #6548 to try and do the right thing with systemd when building packages. If you have a chance can you try it out and see if it does resolve your issue.

[jwittlincohen]

@behlendorf I built new SPL and ZFS kmod packages on Debian Stretch (9.1) from git using the proposed patch. I am still seeing the same behavior - init scripts are being installed rather than systemd unit files. Happy to test again with a new patch.

[yuqingdeng]

ubuntu zfsutils-linux constains /sbin/zfs /sbin/zpool as well as /lib/systemd/system/zfs*service .
I had to remove it to install zfs-0.7.1 deb, thus losing automount.

[jwittlincohen]

I worked around this issue by simply using the Debian systemd unit and config files. The ZoL systemd files are not packaged correctly, and in any case contain incorrect locations for the zfs and zpool binaries as well as the zpool.cache file (on Debian - presumably they are correct on RHEL/CentOS). The Debian systemd files work without issue. See #6606.

[fmckeogh]

I'm having the same issue on Ubuntu 17.04 with kernel 4.13.2. What would be the next step to fixing this?

[darrenfreeman]

I am having the same issue on Debian Stretch. Checked out the latest git master, did the same as the OP,

"Follow the instructions from the wiki to build custom SPL and ZFS 0.7.0 packages. Install resulting packages"

Now I am seeing "zfs.target: Cannot add dependency job, ignoring: Unit zfs.target is masked." as well.

systemctl enable zfs-import-cache.service
Failed to enable unit: Unit file /etc/systemd/system/zfs-import-cache.service is masked.

I also see symlinks to /dev/null:

ls -lha /etc/systemd/system/zfs*
lrwxrwxrwx 1 root root 9 Oct 7 21:04 /etc/systemd/system/zfs-import-cache.service -> /dev/null
lrwxrwxrwx 1 root root 9 Oct 7 21:04 /etc/systemd/system/zfs-import-scan.service -> /dev/null
lrwxrwxrwx 1 root root 9 Oct 7 21:04 /etc/systemd/system/zfs-mount.service -> /dev/null
lrwxrwxrwx 1 root root 9 Oct 7 21:04 /etc/systemd/system/zfs-share.service -> /dev/null
lrwxrwxrwx 1 root root 9 Oct 7 21:04 /etc/systemd/system/zfs.target -> /dev/null
lrwxrwxrwx 1 root root 9 Oct 7 21:04 /etc/systemd/system/zfs-zed.service -> /dev/null

/etc/systemd/system/zfs-mount.service.wants:
total 8.0K
drwxr-xr-x 2 root root 4.0K Sep 26 16:13 .
drwxr-xr-x 13 root root 4.0K Oct 7 21:04 ..
lrwxrwxrwx 1 root root 44 Sep 26 16:13 zfs-import-cache.service -> /lib/systemd/system/zfs-import-cache.service

/etc/systemd/system/zfs-share.service.wants:
total 8.0K
drwxr-xr-x 2 root root 4.0K Sep 26 16:13 .
drwxr-xr-x 13 root root 4.0K Oct 7 21:04 ..
lrwxrwxrwx 1 root root 37 Sep 26 16:13 zfs-mount.service -> /lib/systemd/system/zfs-mount.service

/etc/systemd/system/zfs.target.wants:
total 8.0K
drwxr-xr-x 2 root root 4.0K Sep 26 16:13 .
drwxr-xr-x 13 root root 4.0K Oct 7 21:04 ..
lrwxrwxrwx 1 root root 44 Sep 26 16:13 zfs-import-cache.service -> /lib/systemd/system/zfs-import-cache.service
lrwxrwxrwx 1 root root 37 Sep 26 16:13 zfs-mount.service -> /lib/systemd/system/zfs-mount.service
lrwxrwxrwx 1 root root 37 Sep 26 16:13 zfs-share.service -> /lib/systemd/system/zfs-share.service
lrwxrwxrwx 1 root root 35 Sep 26 16:13 zfs-zed.service -> /lib/systemd/system/zfs-zed.service

From the timestamps, I'm guessing the *.wants directories are leftovers from the now-uninstalled Debian zfs packages.

[loli10K]

Looks like a duplicate of #6591, closing.