Changed systemd unit ordering with 0.7.4 (zfs-mount/local-fs.target) #6953

Lalufu · 2017-12-13T12:18:56Z

System information

Type	Version/Name
Distribution Name	Fedora
Distribution Version	26
Linux Kernel	4.14.4
Architecture	x86_64
ZFS Version	0.7.4
SPL Version	0.7.4

Describe the problem you're observing

In 0.7.3 zfs-mount.service ran (and completed) before local-fs.target, meaning all file systems on the pool were available before most services on the system started.

With 0.7.4 this does no longer seem to be the case, local-fs.target is reached before zfs file systems are mounted. Also, many file systems remain completely unmounted.

I've seen comments in the release notes that there is now a zfs-import.target, but I'm unsure how this relates to this issue exactly.

Affected file systems which have the 'mountpoint' and 'canmount' property set, but are not legacy in the sense that they're mounted via fstab.

Describe how to reproduce the problem

Update to 0.7.4, reboot

Fabian-Gruenbichler · 2017-12-13T12:29:17Z

what does systemctl status "zfs*" "zpool*" output?

Fabian-Gruenbichler · 2017-12-13T12:37:18Z

not familiar with rpm packaging, but possibly systemd_svcs in https://github.com/zfsonlinux/zfs/blob/master/rpm/generic/zfs.spec.in#L225 is missing the newly introduced zfs-import.target (#6764), which is thus not enabled on upgrades?

Lalufu · 2017-12-13T12:37:34Z

● local-fs.target - Local File Systems
   Loaded: loaded (/usr/lib/systemd/system/local-fs.target; static; vendor preset: disabled)
   Active: active since Wed 2017-12-13 12:20:48 UTC; 11min ago
     Docs: man:systemd.special(7)

Dec 13 12:20:48 ethan.home.dn.lalufu.net systemd[1]: Reached target Local File Systems.

● zfs-import-cache.service - Import ZFS pools by cache file
   Loaded: loaded (/usr/lib/systemd/system/zfs-import-cache.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2017-12-13 12:20:52 UTC; 11min ago
  Process: 778 ExecStart=/sbin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=0/SUCCESS)
  Process: 775 ExecStartPre=/sbin/modprobe zfs (code=exited, status=0/SUCCESS)
 Main PID: 778 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/zfs-import-cache.service

Dec 13 12:20:48 ethan.home.dn.lalufu.net systemd[1]: Starting Import ZFS pools by cache file...
Dec 13 12:20:52 ethan.home.dn.lalufu.net systemd[1]: Started Import ZFS pools by cache file.

● zfs.target - ZFS startup target
   Loaded: loaded (/usr/lib/systemd/system/zfs.target; enabled; vendor preset: enabled)
   Active: active since Wed 2017-12-13 12:21:10 UTC; 11min ago

Dec 13 12:21:10 ethan.home.dn.lalufu.net systemd[1]: Reached target ZFS startup target.

● zfs-zed.service - ZFS Event Daemon (zed)
   Loaded: loaded (/usr/lib/systemd/system/zfs-zed.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2017-12-13 12:20:52 UTC; 11min ago
     Docs: man:zed(8)
 Main PID: 1854 (zed)
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/zfs-zed.service
           └─1854 /sbin/zed -F

Dec 13 12:20:52 ethan.home.dn.lalufu.net systemd[1]: Started ZFS Event Daemon (zed).
Dec 13 12:20:52 ethan.home.dn.lalufu.net zed[1854]: ZFS Event Daemon 0.7.4-1 (PID 1854)
Dec 13 12:20:52 ethan.home.dn.lalufu.net zed[1854]: Processing events since eid=0
Dec 13 12:20:52 ethan.home.dn.lalufu.net zed[1864]: eid=1 class=history_event pool_guid=0xBEF523D1432B1A35
Dec 13 12:20:52 ethan.home.dn.lalufu.net zed[1873]: eid=2 class=config_sync pool_guid=0xBEF523D1432B1A35
Dec 13 12:20:52 ethan.home.dn.lalufu.net zed[1878]: eid=3 class=pool_import pool_guid=0xBEF523D1432B1A35
Dec 13 12:20:52 ethan.home.dn.lalufu.net zed[1891]: eid=4 class=history_event pool_guid=0xBEF523D1432B1A35
Dec 13 12:20:52 ethan.home.dn.lalufu.net zed[1922]: eid=5 class=config_sync pool_guid=0xBEF523D1432B1A35

● zfs-share.service - ZFS file system shares
   Loaded: loaded (/usr/lib/systemd/system/zfs-share.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2017-12-13 12:21:10 UTC; 11min ago
  Process: 2332 ExecStart=/sbin/zfs share -a (code=exited, status=0/SUCCESS)
  Process: 2330 ExecStartPre=/bin/rm -f /etc/dfs/sharetab (code=exited, status=0/SUCCESS)
 Main PID: 2332 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/zfs-share.service

Dec 13 12:20:59 ethan.home.dn.lalufu.net systemd[1]: Starting ZFS file system shares...
Dec 13 12:21:10 ethan.home.dn.lalufu.net systemd[1]: Started ZFS file system shares.

● zfs-mount.service - Mount ZFS filesystems
   Loaded: loaded (/usr/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2017-12-13 12:20:48 UTC; 11min ago
  Process: 768 ExecStart=/sbin/zfs mount -a (code=exited, status=0/SUCCESS)
 Main PID: 768 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/zfs-mount.service

Dec 13 12:20:48 ethan.home.dn.lalufu.net systemd[1]: Starting Mount ZFS filesystems...
Dec 13 12:20:48 ethan.home.dn.lalufu.net systemd[1]: Started Mount ZFS filesystems.

Note that there's some weird ordering here, zfs-mount completes before zfs-import-cache, and that should not be possible.

Looking at a system that still runs 0.7.3 I cannot immediately figure out why zfs-mount completes before local-fs.target, but it does.

From 0.7.4:

Dec 13 12:20:48 ethan.home.dn.lalufu.net systemd[1]: Starting Mount ZFS filesystems...
[...]
Dec 13 12:20:48 ethan.home.dn.lalufu.net systemd[1]: Started Mount ZFS filesystems.
[...]
Dec 13 12:20:48 ethan.home.dn.lalufu.net systemd[1]: Reached target Local Encrypted Volumes.
[...]
Dec 13 12:20:48 ethan.home.dn.lalufu.net systemd[1]: Starting Import ZFS pools by cache file...
Dec 13 12:20:48 ethan.home.dn.lalufu.net systemd[1]: Reached target Local File Systems (Pre).
[...]
Dec 13 12:20:48 ethan.home.dn.lalufu.net systemd[1]: Reached target Local File Systems.

So zfs-mount is still happening before local-fs.target, but... the actual mounting isn't happening. Also, the import....

Lalufu · 2017-12-13T12:39:28Z

zfs-import.target is definitely not enabled. I'll try enabling that and see...

Lalufu · 2017-12-13T12:40:52Z

Enabling zfs-import.target makes things look a lot more reasonable.

Fabian-Gruenbichler · 2017-12-13T13:05:28Z

duplicate: #6951

truatpasteurdotfr · 2017-12-13T13:39:28Z

thanks for the heads-up (still using 6.5.11 here)

behlendorf · 2017-12-13T18:50:49Z

@Fabian-Gruenbichler can we get a PR opened to update the packaging to include zfs-import.target in systemd_svcs. Let's also add a comment to the wiki and release notes with the workaround.

$ systemctl enable zfs-import.target

dcrdev · 2017-12-14T00:21:00Z

So might be worth mentioning in the release notes also...

I did the upgrade, wasn't aware of this change when I did - so naturally rebooted. System booted up into the dracut emergency shell; running a zfs root. It wasn't immediately obvious what was wrong due to me not being able to enter my root password, I think due to keyboard layouts over ipmi.

So came across this issue and had to chroot into the system to enable the zfs-import target, rebooted still went into the emergency shell. chrooted back into the system, examined the previous boot journal and it transpired the file system was failing to mount due to the target directory not being empty.

I believe this is because I (like many probably) am using a mixture of legacy and native zfs mounts for my root filesystem, before zfs-import was enabled I imagine the legacy ones succeeded where as the native mounts did not. So in summary a very simple problem, but not an obvious one.

Add missing zfs-import.target to list of systemd services in zfs RPM spec file. Reviewed-by: Niklas Wagner <[email protected]> Reviewed-by: Giuseppe Di Natale <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ralf Ertzinger <[email protected]> Issue #6953 Closes #6955

behlendorf · 2017-12-14T01:18:50Z

Fix merged to master and a notice was added to the top of the zfs-0.7.4 release notes.

fermulator · 2017-12-14T15:23:26Z

FYI: The manual workaround here is to simply run the zfs mount command on your pool,
zfs mount <pool>
or for all pools:
zfs mount -a

KAMiKAZOW · 2017-12-16T02:50:57Z

Isn't this bug severe enough that a 0.7.4.1 hotfix should be released?

bunder2015 · 2017-12-16T04:39:38Z

I don't believe so, users have had to maintain systemd services in the past, such as 0.6.5.8 release.

KAMiKAZOW · 2017-12-16T10:31:58Z

So "users get thrown into emergency mode" is not a severe bug in your eyes? Wow.

bunder2015 · 2017-12-16T10:48:11Z

For what it's worth, gentoo tells you to add the services to openrc after installation... I don't use systemd personally, but I imagine it does the same there. If your package manager doesn't tell you anything, I think that's more a problem of the package manager than the zfs packages themselves.

		[ -e "${EROOT}etc/runlevels/boot/zfs-import" ] || \
			einfo "You should add zfs-import to the boot runlevel."
		[ -e "${EROOT}etc/runlevels/boot/zfs-mount" ] || \
			einfo "You should add zfs-mount to the boot runlevel."
		[ -e "${EROOT}etc/runlevels/default/zfs-share" ] || \
			einfo "You should add zfs-share to the default runlevel."

		[ -e "${EROOT}etc/runlevels/default/zfs-zed" ] || \
			einfo "You should add zfs-zed to the default runlevel."

grantwwu · 2017-12-16T22:59:47Z

Something similar also appears to be happening with the Ubuntu ZFS PPA.

Lalufu · 2017-12-17T18:32:03Z

systemd does the right thing here, the unit was not marked as to be started automatically after installation, so it didn't. Redhat based systems in general don't ask questions during packet installation A 0.7.4-2 RPM has been spun with this fixed, so this can be closed I believe. December 16, 2017 11:49 AM, "bunder2015" <[email protected] (mailto:%22bunder2015%22%20<[email protected]>)> wrote: For what it's worth, gentoo tells you to add the services to openrc after installation... I don't use systemd personally, but I imagine it does the same there. If your package manager doesn't tell you anything, I think that's more a problem of the package manager than the zfs packages themselves. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub (#6953 (comment)), or mute the thread (https://github.com/notifications/unsubscribe-auth/AFfh5VBg-1_nz6CgjfkcUOYSwhkElcxQks5tA5_0gaJpZM4RAdeR).

grantwwu · 2017-12-18T13:04:18Z

Ubuntu ZFS PPA author has pushed a fix.

behlendorf · 2017-12-18T18:08:35Z

This fix will be included in 0.7.5 which will be released this week. And as previously mentioned the needed fix has been picked up in the packaging for the at least the EPEL, Fedora, Ubuntu PPA and Debian repositories.

Add missing zfs-import.target to list of systemd services in zfs RPM spec file. Reviewed-by: Niklas Wagner <[email protected]> Reviewed-by: Giuseppe Di Natale <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ralf Ertzinger <[email protected]> Issue #6953 Closes #6955

tonyhutter · 2017-12-19T01:44:32Z

https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.7.5 is out and should have all the necessary fixes.

L-B · 2017-12-25T19:16:38Z

FWIW, I upgraded 0.7.3 to 0.7.5 on CentOS 7, and I still had to enable zfs-import.target manually.

Add missing zfs-import.target to list of systemd services in zfs RPM spec file. Reviewed-by: Niklas Wagner <[email protected]> Reviewed-by: Giuseppe Di Natale <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ralf Ertzinger <[email protected]> Issue openzfs#6953 Closes openzfs#6955

ionutz22 · 2018-04-03T16:47:35Z

Hi,

FYI:
I've just upgraded from 0.6.5.10 to 0.7.7 on a CentOS 7.3 machine and had the same issue.
Mounts were missing after reboot ... and zfs-import.target was set to disabled.
Restarting zfs-mount helped ... also all good after I've enabled zfs-import.target manually.

-I.

Lalufu mentioned this issue Dec 13, 2017

Add missing zfs-import.target to list of systemd services #6955

Merged

13 tasks

aerusso mentioned this issue Dec 14, 2017

Use zfs-import.target in contrib/dracut #6964

Merged

13 tasks

behlendorf closed this as completed Dec 18, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changed systemd unit ordering with 0.7.4 (zfs-mount/local-fs.target) #6953

Changed systemd unit ordering with 0.7.4 (zfs-mount/local-fs.target) #6953

Lalufu commented Dec 13, 2017

Fabian-Gruenbichler commented Dec 13, 2017

Fabian-Gruenbichler commented Dec 13, 2017

Lalufu commented Dec 13, 2017

Lalufu commented Dec 13, 2017

Lalufu commented Dec 13, 2017

Fabian-Gruenbichler commented Dec 13, 2017

truatpasteurdotfr commented Dec 13, 2017

behlendorf commented Dec 13, 2017

dcrdev commented Dec 14, 2017

behlendorf commented Dec 14, 2017 •

edited

Loading

fermulator commented Dec 14, 2017 •

edited

Loading

KAMiKAZOW commented Dec 16, 2017

bunder2015 commented Dec 16, 2017

KAMiKAZOW commented Dec 16, 2017

bunder2015 commented Dec 16, 2017 •

edited

Loading

grantwwu commented Dec 16, 2017

Lalufu commented Dec 17, 2017 via email

grantwwu commented Dec 18, 2017

behlendorf commented Dec 18, 2017

tonyhutter commented Dec 19, 2017

L-B commented Dec 25, 2017

ionutz22 commented Apr 3, 2018

Changed systemd unit ordering with 0.7.4 (zfs-mount/local-fs.target) #6953

Changed systemd unit ordering with 0.7.4 (zfs-mount/local-fs.target) #6953

Comments

Lalufu commented Dec 13, 2017

System information

Describe the problem you're observing

Describe how to reproduce the problem

Fabian-Gruenbichler commented Dec 13, 2017

Fabian-Gruenbichler commented Dec 13, 2017

Lalufu commented Dec 13, 2017

Lalufu commented Dec 13, 2017

Lalufu commented Dec 13, 2017

Fabian-Gruenbichler commented Dec 13, 2017

truatpasteurdotfr commented Dec 13, 2017

behlendorf commented Dec 13, 2017

dcrdev commented Dec 14, 2017

behlendorf commented Dec 14, 2017 • edited Loading

fermulator commented Dec 14, 2017 • edited Loading

KAMiKAZOW commented Dec 16, 2017

bunder2015 commented Dec 16, 2017

KAMiKAZOW commented Dec 16, 2017

bunder2015 commented Dec 16, 2017 • edited Loading

grantwwu commented Dec 16, 2017

Lalufu commented Dec 17, 2017 via email

grantwwu commented Dec 18, 2017

behlendorf commented Dec 18, 2017

tonyhutter commented Dec 19, 2017

L-B commented Dec 25, 2017

ionutz22 commented Apr 3, 2018

behlendorf commented Dec 14, 2017 •

edited

Loading

fermulator commented Dec 14, 2017 •

edited

Loading

bunder2015 commented Dec 16, 2017 •

edited

Loading