Too many levels of symbolic links when accessing data via snapshot (0.7.0-rc2)

[JKDingwall]

We are using the 0.7.0-rc2 packages from https://launchpad.net/~zfs-native/+archive/ubuntu/daily for testing. Our environment loads a squashfs image from a zfs snapshot in the initrd. We needed some patches for 0.6.4/0.6.5 for accessing snapshots but we are seeing similar problems on 0.7.0-rc2, from the busybox initrd:

    # cd /ztank/boot/.zfs/snapshot/thesnap
    /bin/sh: cd: can't cd to /ztank/boot/.zfs/snapshot/thesnap
    # ls -l /ztank/boot/.zfs/snashot/thesnap/root.squashfs
    ls: /ztank/boot/.zfs/snashot/thesnap/root.squashfs: Too many levels of symbolic links
    # cd /ztank/boot/.zfs/snapshot
    # pwd
    /ztank/boot/.zfs/snapshot
    # ls
    drwxrwxrwx    1    0 thesnap
    drwxrwxrwx    1    0 ..
    drwxrwxrwx    2    2 .

The initrd basically does:

    load_squashfs() {
        if ! [ -f /ztank/boot/.zfs/snashot/thesnap/root.squashfs ] ; then
           return 1
        fi
        # create a tmpfs
        # copy squashfs to tmpfs
        # mount squashfs as root filesystem
    }
    
    while  ! load_squashfs ; do
        /bin/sh
    done

On the first load_squashfs we get a return 1 and a shell is is launched which is where I executed the commands in the block above. When I exit that shell the second load_squashfs succeeds.

[chrisjohnson]

I'm seeing this too, in folders I know for sure don't have symlinks. I don't believe this to be a new (last 4 months at least) bug.

Reproduceably, shortly after triggering this bug, my machine seems to hard crash and stop responding to ssh, and I have to reboot it. I'm not really sure where is the best place to look for relevant crash logs.

Context:

This is an arch machine that I set up a pool on a year+ ago. I installed the zfs auto snapshot script and thought nothing of it until now. I used the zfs-git package (and its siblings).

I hadn't updated my machine in at least 4 months, so this issue (for me, anyway) is at least that old. This morning I got curious about how I could browse my snapshots, and issued ls /data/.zfs/snapshot/zfs-auto-snap_daily-2016-10-05-0720 and got: ls: cannot access '/data/.zfs/snapshot/zfs-auto-snap_daily-2016-10-05-0720/shows': Too many levels of symbolic links

I read up and saw some issues from 2013 and 2015 about this, but they had been closed and marked resolved. Still, I figured I would upgrade my system to double check.

It appears that since I last updated, the zfs-git package has gone away in favor of a zfs-linux package. I removed the old zfs-git and installed the latest zfs-linux package (and updated my system in the process). I tried the ls command again, and the exact same issue.

The current zfs-linux version is listed as: 0.6.5.8_4.8.6_1-1

[chrisjohnson]

Ok, I'm dumb. zfs-git was just renamed to zfs-linux-git. I updated to the latest -git package set (zfs-linux-git 0.7.0_rc2_r5_g2bac681_4.8.6_1-1). Issue no longer happens for me. I can browse my snapshots just fine. Sorry for the confusion!

[JKDingwall]

A year later and I've had time to come back to this... The cause of the issue is 5dc1ff2 which changes the call of mount/umount to be done via /usr/bin/env instead of /bin/sh. In the default Ubuntu ramdisk /usr/bin/env is not present but adding an initramfs-tools hook script to install it solves the problem. (That or manually mounting the snapshot.)

In the original report where we said that a second attempt worked this was due to a bug in our own initramfs scripts which erroneously entered fall back code after the failed copy squashfs to ram step.

I'm leaving this issue open for discussion whether it would be correct to provide /usr/bin/env in the ramdisk.

[behlendorf]

@JKDingwall thanks for following up. Yes, I think your fix of adding /usr/bin/env to the zfs initramfs hook would be a reasonable way to address this. If you could open a PR that would be great.