Cannot mount encrypted volume on thumb drive

[gratay]

Type	Version/Name
Distribution Name	Gentoo & Knoppix (Debian)
Distribution Version	N/A (Gentoo) & 8.1 (Knoppix)
Linux Kernel	4.14.15-aufs (Gentoo) & 4.12.7-64 (Knoppix)
Architecture	amd64
ZFS Version	git repository HEAD 2018/2/14
SPL Version	git repository HEAD 2018/2/14

Describe the problem you're observing

Using a thumb drive with zpool on it: one volume (video) created without encryption, one volume (data) created with encryption. Have been exporting and importing it between different machines with different kernel versions as noted above, but always same version of git repository on them (kept in sync).
Now it has started failing to mount the encrypted volume (sometimes) and I can't explain why. Doesn't happen 100% of the time, but once it fails to mount it will continue failing to mount, no matter how many times kernel modules are unloaded and reloaded. Reboot doesn't always help either.

eg:
root@atlantis:~ $ zpool import -l thumb
Enter passphrase for 'thumb/data':
1 / 1 keys successfully loaded
filesystem 'thumb/data' can not be mounted: Read-only file system
cannot mount 'thumb/data': Invalid argument

If we look at the volumes where they should be mounted, the files on the unencrypted volume are fine, but nothing shows up for the encrypted volume (as expected from error above):

root@atlantis:~ $ ls -la /thumb/video
total 3627904
drwxr-xr-x 2 gtaylor root 11 Feb 14 21:16 .
drwxr-xr-x 4 root root 4 Dec 18 11:15 ..
-rw-rw-r-- 1 gtaylor gtaylor 371 Feb 12 10:11 m4v.md5

root@atlantis:~ $ ls -la /thumb/data
total 1
drwxr-xr-x 2 root root 2 Dec 18 11:15 .
drwxr-xr-x 4 root root 4 Dec 18 11:15 ..

So I did some basic property checks to see if I was missing a feature somehow, and I can't see a problem here:

root@atlantis:~ $ zpool get all thumb
NAME PROPERTY VALUE SOURCE
thumb size 58.5G -
thumb capacity 12% -
thumb altroot - default
thumb health ONLINE -
thumb guid 18171553514569541670 -
thumb version - default
thumb bootfs - default
thumb delegation on default
thumb autoreplace off default
thumb cachefile - default
thumb failmode wait default
thumb listsnapshots off default
thumb autoexpand off default
thumb dedupditto 0 default
thumb dedupratio 1.00x -
thumb free 50.9G -
thumb allocated 7.60G -
thumb readonly off -
thumb ashift 0 default
thumb comment - default
thumb expandsize - -
thumb freeing 0 -
thumb fragmentation 0% -
thumb leaked 0 -
thumb multihost off default
thumb feature@async_destroy enabled local
thumb feature@empty_bpobj active local
thumb feature@lz4_compress active local
thumb feature@multi_vdev_crash_dump enabled local
thumb feature@spacemap_histogram active local
thumb feature@enabled_txg active local
thumb feature@hole_birth active local
thumb feature@extensible_dataset active local
thumb feature@embedded_data active local
thumb feature@bookmarks enabled local
thumb feature@filesystem_limits enabled local
thumb feature@large_blocks enabled local
thumb feature@large_dnode enabled local
thumb feature@sha512 enabled local
thumb feature@skein enabled local
thumb feature@edonr enabled local
thumb feature@userobj_accounting active local
thumb feature@encryption active local

root@atlantis:~ $ zfs get all thumb/data
NAME PROPERTY VALUE SOURCE
thumb/data type filesystem -
thumb/data creation Mon Dec 18 11:15 2017 -
thumb/data used 4.14G -
thumb/data available 648G -
thumb/data referenced 4.14G -
thumb/data compressratio 1.04x -
thumb/data mounted no -
thumb/data quota none default
thumb/data reservation none default
thumb/data recordsize 128K default
thumb/data mountpoint /thumb/data default
thumb/data sharenfs off default
thumb/data checksum on default
thumb/data compression on local
thumb/data atime on default
thumb/data devices on default
thumb/data exec on default
thumb/data setuid on default
thumb/data readonly off default
thumb/data zoned off default
thumb/data snapdir hidden default
thumb/data aclinherit restricted default
thumb/data createtxg 255 -
thumb/data canmount on default
thumb/data xattr on default
thumb/data copies 1 default
thumb/data version 5 -
thumb/data utf8only off -
thumb/data normalization none -
thumb/data casesensitivity sensitive -
thumb/data vscan off default
thumb/data nbmand off default
thumb/data sharesmb off default
thumb/data refquota none default
thumb/data refreservation none default
thumb/data guid 8854241042863058303 -
thumb/data primarycache all default
thumb/data secondarycache all default
thumb/data usedbysnapshots 0B -
thumb/data usedbydataset 4.14G -
thumb/data usedbychildren 0B -
thumb/data usedbyrefreservation 0B -
thumb/data logbias latency default
thumb/data dedup on local
thumb/data mlslabel none default
thumb/data sync standard default
thumb/data dnodesize legacy default
thumb/data refcompressratio 1.04x -
thumb/data written 4.14G -
thumb/data logicalused 4.31G -
thumb/data logicalreferenced 4.31G -
thumb/data volmode default default
thumb/data filesystem_limit none default
thumb/data snapshot_limit none default
thumb/data filesystem_count none default
thumb/data snapshot_count none default
thumb/data snapdev hidden default
thumb/data acltype off default
thumb/data context none default
thumb/data fscontext none default
thumb/data defcontext none default
thumb/data rootcontext none default
thumb/data relatime off default
thumb/data redundant_metadata all default
thumb/data overlay off default
thumb/data encryption aes-256-ccm -
thumb/data keylocation prompt local
thumb/data keyformat passphrase -
thumb/data pbkdf2iters 342K -
thumb/data encryptionroot thumb/data -
thumb/data keystatus available -

So I reloaded the module with some debug, and I've attached that here. I've looked at it myself, though, and I can't see any debug that looks helpful in working out why it is failing. The volume is definitely not read-only, just encrypted. If I take it to another machine after it fails this, it can sometimes mount there, despite the fact both are running same zfs version.

root@atlantis:~ $ insmod /lib/modules/4.14.5-aufs/extra/zfs/zfs.ko zfs_dbgmsg_enable=1 zfs_flags=545 zfs_dbgmsg_maxsize=67108864

(reran all the commands above)

root@atlantis:~ $ zpool export thumb

It doesn't make any difference if I try to manually load the key and mount the volume. That fails the same as the import -l.

proc_spl_kstat_zfs_dbgmsg.gz

[tcaputi]

@gratay We have fixed a number of bugs with encryption since you created the dataset, some of which technically required rebuilding the encrypted datasets due to on-disk format problems that could arise. Can you try doing that with the latest code from master? Would it be ok if we closed this ticket for now and you can reopen it and @ me if it happens again?

[gratay]

Ok, that sounds all right. I was using the key for a couple of weeks, so if there's still a problem it will probably take that long again for it to show. I'll nuke the key and restart.

[rageagainstthebugs]

@gratay can this be close?

[behlendorf]

Closing, this was very likely resolved by one of the follow up encryption fixed. Let's open a new issue if you run in to problems with the latest version from master.