Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snapshot automount (.zfs) still fails under chroot (OpenSSH ChrootDirectory) #8903

Closed
mattgarber opened this issue Jun 14, 2019 · 9 comments
Closed
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@mattgarber
Copy link

mattgarber commented Jun 14, 2019

System information

Type Version/Name
Distribution Name Debian
Distribution Version 9.9 (Stretch)
Linux Kernel 4.9.0-9-amd64
Architecture amd64
ZFS Version 0.8.0-1
SPL Version 0.8.0-1

Describe the problem you're observing

When attempting to browse auto-mounting ZFS dataset snapshots (.zfs/snapshot directory) over a chrooted SFTP (OpenSSH’s ChrootDirectory directive), snapshot access fails with the following error:

WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512

Browsing snapshots in this manner works as intended if ChrootDirectory is not set (not a workable solution, due to needing to restrict SFTP client access to the filesystem), and also works correctly on FreeBSD (11.2-RELEASE, 12.0-RELEASE), presumably because of differing methods of auto-mounting snapshot access.

This appears to be extremely similar to #3778, and @behlendorf indicated various fixes landing in 0.6.x and 0.7.x, although perhaps it wasn’t fully resolved for all kinds of chroot? The error is clear, in that the mount operation is being called on the SFTP client’s view of the filesystem (where /tank/backups/test01 appears as /), even though it should be resolved outside of the chroot.

I first ran into this issue on Ubuntu 18.04, running distro-provided zfs/spl 0.7.5, although attempted this reproduction on Debian with the latest 0.8.0 release to see whether the #3778 discussion/fixes were landed after Ubuntu’s version.

Describe how to reproduce the problem

  1. Create a new dataset for hosting restricted client content, e.g. tank/backups/test01, mounted at /tank/backups/test01.

  2. Create some dummy files, and create a symlink at the top of the dataset, pointing to the hidden snapshots directory: ln -s .zfs/snapshot snapshots. (The symlink is only so that SFTP clients don’t have to explicitly view hidden files, it’s not a prerequisite for reproducing the issue!)

  3. Create a few snapshots of the dataset: zfs snapshot tank/backups/test01@one, zfs snapshot tank/backups/test01@two.

  4. Create a test user (e.g. test01) and configure OpenSSH to restrict access for that particular user, including ChrootDirectory, and restart sshd. E.g.:

Match User test01
	ChrootDirectory /tank/backups/test01
	ForceCommand internal-sftp -l INFO -u 022
	PasswordAuthentication yes
  1. Connect via SFTP as specified test user. Verify that dummy files and directories are successfully browsable.

  2. Change into the snapshots symlink (.zfs/snapshot), and verify that snapshots list is accurate. (The symlink is only so that SFTP clients don’t have to explicitly view hidden files, it’s not a prerequisite for reproducing the issue!)

  3. Attempt to enter one or more snapshot directories. No content will appear, and kernel log on host system will include one or more errors similar to above: WARNING: Unable to automount /.zfs/snapshot/one/tank/backups/test01@one: 512.

Include any warning/errors/backtraces from the system logs

[ 5061.371785] WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512
[ 5061.457781] WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512
[ 5061.464070] WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512
[ 5063.660823] WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512
[ 5063.747940] WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512
[ 5063.757878] WARNING: Unable to automount /.zfs/snapshot/2019-06-13T221233/tank/backups/test01@2019-06-13T221233: 512
@johnnyjacq16
Copy link

Run zfs get snapdir,snapdev <your dataset> the value for snapdir should be visible, the default is hidden.

@mattgarber
Copy link
Author

# zfs get snapdir,snapdev tank/backups/test01

NAME                 PROPERTY  VALUE    SOURCE
tank/backups/test01  snapdir   visible  local
tank/backups/test01  snapdev   hidden   default

In /var/log/messages:

[70421.106185] WARNING: Unable to automount /.zfs/snapshot/one/tank/backups/test01@one: 512
[70421.201172] WARNING: Unable to automount /.zfs/snapshot/one/tank/backups/test01@one: 512
[70421.208336] WARNING: Unable to automount /.zfs/snapshot/one/tank/backups/test01@one: 512

@johnnyjacq16
Copy link

for logging run journalctl -k -b 0 -f
then in another terminal try to mount the snapshot manually with mount -t zfs <the dataset snapshot> <a directory>

@mattgarber
Copy link
Author

mattgarber commented Jun 14, 2019

Mounting all snapshots manually (mount -t zfs <snapshot> <mountpoint>), or auto-mounted via the .zfs/snapshot directory, works completely fine on the host itself (not chrooted), or over SFTP when ChrootDirectory isn’t being used.

This issue is only occurring when trying to auto-mount the snapshots through a chrooted SFTP, because the auto-mounter is using the client’s view of the filesystem (chrooted to /) instead of the true filesystem location. And obviously, the .zfs/snapshot directory isn’t really at /.zfs, it’s at /tank/backups/test01/.zfs.

@mattgarber
Copy link
Author

@behlendorf In #3778, you mentioned the ability to get full mount path in a chroot was resolved “in the 0.7 series,” but I didn’t see any specific linked commits, or other indication of when exactly it landed in 0.7 (comment date puts it at 0.7.6 or prior). Was it reverted out at some point, or does this potentially appear to be a different enough circumstance based on how OpenSSH might be performing the user chroot?

I also noticed somewhat similar issue #3512 regarding Linux namespace usage preventing full path resolution of mounts or snapshots in containers, although in this case OpenSSH should only be chrooting the user, not in a different namespace, so I’m a little stumped – not running in any containers here.

@behlendorf
Copy link
Contributor

@mattgarber multiple issues were resolved regarding the automounting of snapshots. Unfortunately, automounting in your circumstance appears not to be handled. As you correctly pointed it, it appears to be because the zfsctl_snapshot_path() function is generating the wrong full path for the mount when triggered in the chroot environment. One possible way to handle this could be to fallback to the full path returned by zfsctl_snapshot_path_objset() when the zfsctl_snapshot_path() clearly isn't valid.

@behlendorf behlendorf added the Type: Defect Incorrect behavior (e.g. crash, hang) label Jun 20, 2019
@mattgarber
Copy link
Author

@behlendorf Thanks for taking a look and confirming this as a different case. Do you remember if there was a reason the zfsctl_snapshot_path_objset() fallback wasn’t added during the #3778 discussion, such as a performance hit? Or just dropped due to the other fixes at the time for automounting in separate cases (e.g. NFS)?

Is there an easy/good way to “clearly” determine the path isn’t valid, or is the best solution to do a quick compare between the values of zfsctl_snapshot_path() and zfsctl_snapshot_path_objset(), and using the value of zfsctl_snapshot_path_objset() as the mountpoint if there’s a mismatch? (I’m not sure if that would then cause any other kind of regression…)

@behlendorf
Copy link
Contributor

@mattgarber if possible, would you mind verifying the proposed fix in #8966.

@mattgarber
Copy link
Author

mattgarber commented Jun 29, 2019

@behlendorf @kusumi Thank you! Was just able to confirm on a fresh installation that the proposed fix #8966 eliminates the issue preventing automounting snapshots in this chroot.

sftp> ls -lh /
lrwxrwxrwx    ? 0        0             13B Jun 29 22:31 /snapshots
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /test1
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /test2
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /test3
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /test4
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /test5
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /test6
sftp> ls -lh /snapshots/
drwxrwxrwx    ? 0        0              0B Jun 29 22:31 /snapshots/one
drwxrwxrwx    ? 0        0              0B Jun 29 22:39 /snapshots/two
sftp> ls -lh /snapshots/one/
lrwxrwxrwx    ? 0        0             13B Jun 29 22:31 /snapshots/one/snapshots
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /snapshots/one/test1
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /snapshots/one/test2
-rw-r--r--    ? 0        0              0B Jun 29 22:39 /snapshots/one/test3
sftp> 
tank/backups/test01@one   97G  128K   97G   1% /tank/backups/test01/.zfs/snapshot/one

tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Aug 13, 2019
Chroot'd process fails to automount snapshots due to realpath(3)
failure in mount.zfs(8).

Construct a mount point path from sb of the ctldir inode and dirent
name, instead of from d_path(), so that chroot'd process doesn't get
affected by its view of fs.

Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Tomohiro Kusumi <[email protected]>
Closes openzfs#8903
Closes openzfs#8966
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Aug 22, 2019
Chroot'd process fails to automount snapshots due to realpath(3)
failure in mount.zfs(8).

Construct a mount point path from sb of the ctldir inode and dirent
name, instead of from d_path(), so that chroot'd process doesn't get
affected by its view of fs.

Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Tomohiro Kusumi <[email protected]>
Closes openzfs#8903
Closes openzfs#8966
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Aug 23, 2019
Chroot'd process fails to automount snapshots due to realpath(3)
failure in mount.zfs(8).

Construct a mount point path from sb of the ctldir inode and dirent
name, instead of from d_path(), so that chroot'd process doesn't get
affected by its view of fs.

Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Tomohiro Kusumi <[email protected]>
Closes openzfs#8903
Closes openzfs#8966
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Sep 17, 2019
Chroot'd process fails to automount snapshots due to realpath(3)
failure in mount.zfs(8).

Construct a mount point path from sb of the ctldir inode and dirent
name, instead of from d_path(), so that chroot'd process doesn't get
affected by its view of fs.

Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Tomohiro Kusumi <[email protected]>
Closes openzfs#8903
Closes openzfs#8966
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Sep 18, 2019
Chroot'd process fails to automount snapshots due to realpath(3)
failure in mount.zfs(8).

Construct a mount point path from sb of the ctldir inode and dirent
name, instead of from d_path(), so that chroot'd process doesn't get
affected by its view of fs.

Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Tomohiro Kusumi <[email protected]>
Closes openzfs#8903
Closes openzfs#8966
tonyhutter pushed a commit that referenced this issue Sep 26, 2019
Chroot'd process fails to automount snapshots due to realpath(3)
failure in mount.zfs(8).

Construct a mount point path from sb of the ctldir inode and dirent
name, instead of from d_path(), so that chroot'd process doesn't get
affected by its view of fs.

Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Tomohiro Kusumi <[email protected]>
Closes #8903
Closes #8966
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Development

No branches or pull requests

3 participants