zfs-mount causes error in mount

[janlam7]

Hi,

on one of my machines I updated to 0.6.4-109_g57ae840 and the zfs-mount initscript seems to work OK, but halfway through the mount help page is shown.
I tracked it down to the method reregister_mounts. It tries to go through all the entries in /proc/mounts and the following line in /proc/mounts causes the problem. On this machine I have the root on zfs.

pool2/ROOT/gentoo / zfs rw,relatime,noxattr,noacl 0

The script ends up trying this:
mount -f -o zfsutil -t zfs --move / /removethismountpointhoweverpossible

Without the '-t zfs' it seems to work.

Jan

[FransUrbo]

That function have been subject to quite some controversy… It's old, and I don't know exactly where it came from. We ended up keeping it, mostly because we weren't exactly sure what it did or under what circumstances it was needed.

"Thanx" for triggering this :).

Now, looking at the function and it's call again, it's supposed to be called ONLY if /etc/mtab != /proc/mounts. Looking at it again and actually LOOKING at the code, I can see the error. The comment talks about it only being called if /etc/mtab isn't a link to /proc/mounts (don't know why that is needed), but the actual code reads /proc/mounts anyway. That might be the first error..

Could you please give me the output of:

# ls -l /etc/mtab /proc/mounts
# cat /etc/mtab
# cat /proc/mounts
# zfs get all pool2/ROOT/gentoo

If you can trigger this at every boot, could you edit the function to read /etc/mtab instead of /proc/mounts and see if that makes any difference?

[FransUrbo]

Since you seem to be ok with using GIT, may I ask you to checkout https://github.com/FransUrbo/zfs/tree/fixes.initscripts3 and install the init scripts from that branch?

Just run autogen.sh and ./configure with the same parameters as you use normally, but instead of doing a make [install|rpm], just run cd etc/init.d && make install instead. That way you ONLY get the init scripts.

Hopefully those should do it. If they do, let me know and I'll create a pull request for @behlendorf to look at.

[FransUrbo]

I managed to find the commit where reregister_mounts() was introduced: f5ef715 by @Rudd-O. @Rudd-O Do you happen to remember what this was supposed to solve?

[FransUrbo]

It seems like this function is usable if/when mounts exists in /proc/mounts (which is authoritative) but not in /etc/mtab. I have not been able to reliably been able to trigger this, but making /etc/mtab a file, copying the contents of /proc/mounts to the file and then remove a Z filesystem from it will now work correctly.

If this is happening in the original issue I'm not sure, but it seems likely.

[FransUrbo]

Ok, I'm think I'm happy with this last commit. @janlam7 if you could test it and make sure it works for you to, I'd appreciate it.

[janlam7]

@FransUrbo commit b7bdebd does NOT work for me. I think the mount message is caused by the combination of -f and -t zfs parameters. If I take -t zfs out, it seems to work.

During the boot process I do get a warning that support for /etc/mtab is deprecated, I'll have a look at that too. It suggests running:

ln -snf /proc/self/mounts /etc/mtab

Too answer the previous questions:

ls -l /etc/mtab /proc/mounts

-rw-r--r-- 1 root root 1338 Jun 20 06:26 /etc/mtab
lrwxrwxrwx 1 root root 11 Jun 20 06:26 /proc/mounts -> self/mounts

cat /etc/mtab

rootfs / rootfs rw 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=10240k,nr_inodes=255386,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
pool2/ROOT/gentoo / zfs rw,relatime,noxattr,noacl 0 0
tmpfs /run tmpfs rw,nodev,relatime,size=204788k,mode=755 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
cgroup_root /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
openrc /sys/fs/cgroup/openrc cgroup rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc 0 0
cpuset /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cpu /sys/fs/cgroup/cpu cgroup rw,nosuid,nodev,noexec,relatime,cpu 0 0
cpuacct /sys/fs/cgroup/cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct 0 0
devices /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
blkio /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nodev,noexec,nosuid 0 0
tmpfs /run tmpfs rw,nodev,mode=0755,size=10% 0 0

cat /proc/mounts

rootfs / rootfs rw 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=10240k,nr_inodes=255386,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
pool2/ROOT/gentoo / zfs rw,relatime,noxattr,noacl 0 0
tmpfs /run tmpfs rw,nodev,relatime,size=204788k,mode=755 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
cgroup_root /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
openrc /sys/fs/cgroup/openrc cgroup rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc 0 0
cpuset /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cpu /sys/fs/cgroup/cpu cgroup rw,nosuid,nodev,noexec,relatime,cpu 0 0
cpuacct /sys/fs/cgroup/cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct 0 0
devices /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
blkio /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0

zfs get all pool2/ROOT/gentoo

NAME PROPERTY VALUE SOURCE
pool2/ROOT/gentoo type filesystem -
pool2/ROOT/gentoo creation Tue Jul 2 8:12 2013 -
pool2/ROOT/gentoo used 691M -
pool2/ROOT/gentoo available 70.4G -
pool2/ROOT/gentoo referenced 507M -
pool2/ROOT/gentoo compressratio 1.63x -
pool2/ROOT/gentoo mounted yes -
pool2/ROOT/gentoo quota none default
pool2/ROOT/gentoo reservation none default
pool2/ROOT/gentoo recordsize 128K default
pool2/ROOT/gentoo mountpoint / local
pool2/ROOT/gentoo sharenfs off default
pool2/ROOT/gentoo checksum on default
pool2/ROOT/gentoo compression lz4 inherited from pool2
pool2/ROOT/gentoo atime off inherited from pool2
pool2/ROOT/gentoo devices on default
pool2/ROOT/gentoo exec on default
pool2/ROOT/gentoo setuid on default
pool2/ROOT/gentoo readonly off default
pool2/ROOT/gentoo zoned off default
pool2/ROOT/gentoo snapdir hidden default
pool2/ROOT/gentoo aclinherit restricted default
pool2/ROOT/gentoo canmount noauto local
pool2/ROOT/gentoo xattr off inherited from pool2
pool2/ROOT/gentoo copies 2 inherited from pool2
pool2/ROOT/gentoo version 5 -
pool2/ROOT/gentoo utf8only off -
pool2/ROOT/gentoo normalization none -
pool2/ROOT/gentoo casesensitivity sensitive -
pool2/ROOT/gentoo vscan off default
pool2/ROOT/gentoo nbmand off default
pool2/ROOT/gentoo sharesmb off default
pool2/ROOT/gentoo refquota none default
pool2/ROOT/gentoo refreservation none default
pool2/ROOT/gentoo primarycache all default
pool2/ROOT/gentoo secondarycache none inherited from pool2
pool2/ROOT/gentoo usedbysnapshots 183M -
pool2/ROOT/gentoo usedbydataset 507M -
pool2/ROOT/gentoo usedbychildren 0 -
pool2/ROOT/gentoo usedbyrefreservation 0 -
pool2/ROOT/gentoo logbias latency default
pool2/ROOT/gentoo dedup off default
pool2/ROOT/gentoo mlslabel none default
pool2/ROOT/gentoo sync standard default
pool2/ROOT/gentoo refcompressratio 1.61x -
pool2/ROOT/gentoo written 129M -
pool2/ROOT/gentoo logicalused 604M -
pool2/ROOT/gentoo logicalreferenced 411M -
pool2/ROOT/gentoo filesystem_limit none default
pool2/ROOT/gentoo snapshot_limit none default
pool2/ROOT/gentoo filesystem_count none default
pool2/ROOT/gentoo snapshot_count none default
pool2/ROOT/gentoo snapdev hidden default
pool2/ROOT/gentoo acltype off default
pool2/ROOT/gentoo context none default
pool2/ROOT/gentoo fscontext none default
pool2/ROOT/gentoo defcontext none default
pool2/ROOT/gentoo rootcontext none default
pool2/ROOT/gentoo relatime off default
pool2/ROOT/gentoo redundant_metadata all default
pool2/ROOT/gentoo overlay off default

[janlam7]

The code does not get triggered anymore after executing

ln -snf /proc/self/mounts /etc/mtab

[FransUrbo]

@FransUrbo commit b7bdebd does NOT work for me. I think the mount message is caused by the combination of -f and -t zfs parameters. If I take -t zfs out, it seems to work.

That's very weird, that commit was supposed to fix that.

The thing is, the options ARE (supposed to be) correct. They depend on if the filesystem/dataset is a native Z filesystem or a legacy one.

ln -snf /proc/self/mounts /etc/mtab

Yes, that's always a better alternative, but your dist might not have correct support for that, so that might break something else if you're unlucky.

I notice that Gentoo (latest) also does this. Have no idea why. /proc/mounts (which is a link to /proc/self/mounts) is authoritative. Meaning, that's the kernel updating that and it is ALWAYS right. /etc/mtab on the other hand is updated by any command you run (such as mount/umount etc) and can (and apparently do) get out of sync with the kernels opinion on what's mounted and not.

Which is why the link came to be years ago in the first place. No idea why some dists still insist on having it as a plain file...

-rw-r--r-- 1 root root 1338 Jun 20 06:26 /etc/mtab

Ok, so not a link. We kind'a knew that, but it's nice to get it confirmed.
cat /etc/mtab

cat /proc/mounts

So they're not the same. There's one entry in mtab that doesn't exist in mounts, and two that exists in mounts but not in mtab. They're out of sync.

That's what reregister_mounts() is supposed to fix.

pool2/ROOT/gentoo mountpoint / local

Ok, so it's a native fs.

The correct (!!) command to mount a native Z fs is one of:

mount -f -t zfs -o zfsutil rpool/tests/test09 /rpool/tests/test09
zfs mount rpool/tests/test09

and to mount a legacy fs:

mount -f -t zfs rpool/tests/test09 /rpool/tests/test09

The -t option needs to be there, it is correct. Why it doesn't work for you, I don't know. Work just fine on every machine here…

How did you ./configure and install the new version? Did you have a previous one, from a package or did you install the new one in another place than the first?

Could it be that the mount.zfs helper command got "lost" somewhere? You should have only ONE and the correct and only place for it is in /sbin. If that's an older one, that MIGHT (although I kind'a doubt it, it really haven't changed in years) explain it… Or if it (the actual "mount" command) can't find the helper as "/sbin/mount.zfs" (and only that, anything else is a failure), that's an ever bigger clue.

[janlam7]

A command similiar to mount -f -t zfs -o zfsutil rpool/tests/test09 /rpool/tests/test09 works fine on my machine. In the initscript it is also called with the --move option. That does not work on my machine.
When I tried to get the script to work I figured the --move would be more important than -t zfs.

[FransUrbo]

Works for me...

[janlam7]

Ok, I'll get around it by turning /etc/mtab into a symlink on my machines.

[FransUrbo]

If you have the time, I'd appreciate if you could try to figure out why the command doesn't work for you. Since I can't reproduce it here on any of my machines, there must (?) be something special about your setup...

[jameslikeslinux]

For what it's worth, I am seeing the same problems as @janlam7 using git HEAD on my Funtoo system. I haven't posted because I haven't had any other details to share, but I can at least say it's not just his setup, though there may be something about Gentoo/Funtoo at play. I've been using ZFS on Funtoo for two years without issue using the old zfs OpenRC service, so it is frustrating to see what I perceive as needless change breaking what has worked for so long.

Applying b7bdebd does not stop the error:

 * Checking if ZFS userspace tools present ...                                                                                                                                                                                                                                
Warning: You're not running 64bit. Currently native zfs in
         Linux is only supported and tested on 64bit.
 [ ok ]
 * Loading kernel ZFS infrastructure ...
 [ ok ]
 * Importing ZFS pool rpool ...
 [ !! ]
 * Registering already-mounted ZFS filesystems and volumes  ...

Usage:
 mount [-lhV]
 mount -a [options]
 mount [options] [--source] <source> | [--target] <directory>
 mount [options] <source> <directory>
 mount <operation> <mountpoint> [<target>]

Options:
 -a, --all               mount all filesystems mentioned in fstab
 -c, --no-canonicalize   don't canonicalize paths
 -f, --fake              dry run; skip the mount(2) syscall
 -F, --fork              fork off for each device (use with -a)
 -T, --fstab <path>      alternative file to /etc/fstab
 -h, --help              display this help text and exit
 -i, --internal-only     don't call the mount.<type> helpers
 -l, --show-labels       lists all mounts with LABELs
 -n, --no-mtab           don't write to /etc/mtab
 -o, --options <list>    comma-separated list of mount options
 -O, --test-opts <list>  limit the set of filesystems (use with -a)
 -r, --read-only         mount the filesystem read-only (same as -o ro)
 -t, --types <list>      limit the set of filesystem types
     --source <src>      explicitly specifies source (path, label, uuid)
     --target <target>   explicitly specifies mountpoint
 -v, --verbose           say what is being done
 -V, --version           display version information and exit
 -w, --rw, --read-write  mount the filesystem read-write (default)

 -h, --help     display this help and exit
 -V, --version  output version information and exit

Source:
 -L, --label <label>     synonym for LABEL=<label>
 -U, --uuid <uuid>       synonym for UUID=<uuid>
 LABEL=<label>           specifies device by filesystem label
 UUID=<uuid>             specifies device by filesystem UUID
 PARTLABEL=<label>       specifies device by partition label
 PARTUUID=<uuid>         specifies device by partition UUID
 <device>                specifies device by path
 <directory>             mountpoint for bind mounts (see --bind/rbind)
 <file>                  regular file for loopdev setup

Operations:
 -B, --bind              mount a subtree somewhere else (same as -o bind)
 -M, --move              move a subtree to some other place
 -R, --rbind             mount a subtree and all submounts somewhere else
 --make-shared           mark a subtree as shared
 --make-slave            mark a subtree as slave
 --make-private          mark a subtree as private
 --make-unbindable       mark a subtree as unbindable
 --make-rshared          recursively mark a whole subtree as shared
 --make-rslave           recursively mark a whole subtree as slave
 --make-rprivate         recursively mark a whole subtree as private
 --make-runbindable      recursively mark a whole subtree as unbindable

For more details see mount(8).
 [ ok ]
 * Mounting ZFS filesystem(s)  ...
 [ ok ]
 * Mounting volumes and filesystems registered in fstab ...
 [ ok ]

(I think it's weird that zfs-import tries to import a pool called rpool when there is no rpool attached to the system and it's not even referenced in /etc/conf.d/zfs, but that's another bug report.)

However, the system does come up with all of the datasets mounted. I don't really understand the point of reregister_mounts subroutine.

[FransUrbo]

I am seeing the same problems as @janlam7 using git HEAD on my Funtoo system.

That is actually 'good' news! That means there's something with Gentoo…

@ryao, do you have any idea?

I've been using ZFS on Funtoo for two years without issue using the old zfs OpenRC service, so it is frustrating to see what I perceive as needless change breaking what has worked for so long.

Unfortunately, the change was inevitable. The old script just wasn't maintainable any more, we had five (!) scripts that did the exact same thing, but slightly different. And a lot of functionality that was missing between them.
(I think it's weird that zfs-import tries to import a pool called rpool when there is no rpool attached to the system and it's not even referenced in /etc/conf.d/zfs, but that's another bug report.)

That is "most likely" a "shadow pool". You've had a pool created (or on a zvol) but it wasn't destroyed properly, so there's still information about in the labels…

Verify this by running "zpool import" to see what it finds. It should print out all pools that isn't already imported that it can find labels for on the disks/devices (including zvols etc).

I don't really understand the point of reregister_mounts subroutine.

As far as i can tell, it's supposed to fix a mismatch between /proc/mounts (which is authoritative) and /etc/mtab. That code came from the old init scripts (at least the RedHat one), but why it (the problem) wasn't triggered with them, I don't know.

Trying to compare "mount" between Wheezy, Jessie and Gentoo I see:

Wheezy:     mount from util-linux 2.20.1 (with libblkid and selinux support)
Jessie:     mount from util-linux 2.25.2 (libmount 2.25.0: selinux, assert, debug)
Gentoo:     mount from util-linux 2.25.2 (libmount 2.25.0: assert, debug)

This is how I test:

zpool create -f rpool /dev/sdc
zfs create -o mountpoint=legacy rpool/test1
mkdir /mnt/test1
mount -f -t zfs --move rpool/test1 /mnt/test1

Now that we have a clue about Gentoo, I notice that this only works on Wheezy, which is my primary test platform… !

It seems, like @janlam7 have said several times, that the only way to resolve this is to remove the "-t zfs" option. It seems to be working on all three systems. I'm just worried that that might introduce something else…

I've removed this option and pushed another version (commit 4215fda) which according to information in this issue should solve the problem. If you could just triple check that, I'd appreciate it.

[jameslikeslinux]

Verify this by running "zpool import" to see what it finds. It should print out all pools that isn't already imported that it can find labels for on the disks/devices (including zvols etc).

Nope:

> zpool import   
no pools available to import

I'll look into it some more and open a new issue if I find anything.

For this issue, 4215fda appears to solve the problem.

Your test case doesn't make sense, because it includes the --move argument to mount but it is not testing moving one mountpoint to another. Using your test case, this is what I see:

> zfs create -o mountpoint=legacy rpool/test1
> mkdir /mnt/test1
> mount -f -t zfs --move rpool/test1 /mnt/test1

Usage:
 mount [-lhV]
 mount -a [options]
 mount [options] [--source] <source> | [--target] <directory>
 mount [options] <source> <directory>
 mount <operation> <mountpoint> [<target>]
...

According to mount(8), the --move argument stands alone and takes two directories:

mount --move olddir newdir

[janlam7]

@FransUrbo the version from 4215fda seems to work for me and does not print the mount manual anymore.

[FransUrbo]

zpool import
no pools available to import
Try different "-d

" options. I've had a couple of instances where a pool doesn't shown up without them...
Your test case doesn't make sense, because it includes the --move argument to mount

Exactly! Apparently, the command works if this is omitted. With that, one needs to remove the '-t' option. At least on newer "mount".

[jameslikeslinux]

Thanks, @FransUrbo. The phantom pool showed up when I ran zpool import -d /dev/disk/by-id. I zeroed out the disk in question and the failed import went away.

Also, thanks for 4215fda. It seems to be working reliably. I even tested a legacy mountpoint dataset in /etc/fstab and it booted/mounted fine.

[FransUrbo]

Also, thanks for 4215fda. It seems to be working reliably.

Awesome, thanx for double-checking! I'll be updating the PR in a while to completely remove that code because it isn't needed any more.

You don't have to update to that if you don't want to, since your version now works and your system boots. But just remember that the init script and the function script will be updated as soon as Brian accepts this PR.

[tobiaspal]

I know I'm a bit late to this party, but gentoo initscripts have a --debug parameter, which is like running the initstcript under bash -x, very useful for figuring out what's going on.