Encrypted filesystem is unmountable on kernel 3.12

[FransUrbo]

This is a fork of #32.

After finally mustering enough courage to run 'zfs mount -a', I get:

celia:~# zfs mount -a
filesystem 'share/home/turbo/Crypted' can not be mounted due to error 52
cannot mount 'share/home/turbo/Crypted': Invalid argument

and the syslog say:

Nov 13 03:24:53 Celia modprobe: WARNING: Error inserting ghash_clmulni_intel (/lib/modules/3.12.0+scst+tf.1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko): No such device
Nov 13 03:24:54 Celia kernel: [ 2053.440315] spl-crypto: Cipher test 'CKM_AES_GCM' -> 'sun-gcm(aes)' successful.
Nov 13 03:25:01 Celia kernel: [ 2061.223198] sha256_ssse3: Neither AVX nor SSSE3 is available/usable.
Nov 13 03:25:01 Celia modprobe: WARNING: Error inserting sha256_ssse3 (/lib/modules/3.12.0+scst+tf.1/kernel/arch/x86/crypto/sha256-ssse3.ko): No such device
Nov 13 03:25:01 Celia modprobe: FATAL: Error inserting padlock_sha (/lib/modules/3.12.0+scst+tf.1/kernel/drivers/crypto/padlock-sha.ko): No such device

The SSSE3 crypto is selected as a module:

celia:~# grep -i SSSE3 /boot/config-`uname -r`
CONFIG_CRYPTO_SHA1_SSSE3=m
CONFIG_CRYPTO_SHA256_SSSE3=m
CONFIG_CRYPTO_SHA512_SSSE3=m

and is availible:

celia:/lib/modules/3.12.0+scst+tf.1/kernel# find -name '*ssse3*'
./arch/x86/crypto/sha512-ssse3.ko
./arch/x86/crypto/sha1-ssse3.ko
./arch/x86/crypto/sha256-ssse3.ko

but not loaded, because:

celia:/lib/modules/3.12.0+scst+tf.1/kernel# modprobe sha256_ssse3
FATAL: Error inserting sha256_ssse3 (/lib/modules/3.12.0+scst+tf.1/kernel/arch/x86/crypto/sha256-ssse3.ko): No such device

and

Nov 13 03:30:15 Celia kernel: [ 2375.004027] sha256_ssse3: Neither AVX nor SSSE3 is available/usable.

[FransUrbo]

Ok, so digging through the kernel source, I noticed that SSE? is apparently CPU features. This is an AMD Phenom II X6 1100T which seems to only support SSE and SSE2 (according to /proc/cpuinfo) which might explain the No such
device.

Now my question instead is, why did it try to load sha256_ssse3 in the first place?

I'm using 'checksum=sha256-mac,compression=lz4,encryption=aes-256-gcm' on this filesystem...

[FransUrbo]

Loading the following modules before zfs, the import and the mount:

gf128mul
ghash_generic
sun_ctr
sun_gcm
sha256_generic

then I don't get any problems in the syslog. It still refuses to mount the filesystem though:

celia:~# zfs mount share/home/turbo/Crypted
filesystem 'share/home/turbo/Crypted' can not be mounted due to error 52
cannot mount 'share/home/turbo/Crypted': Invalid argument

[FransUrbo]

Trying to create a new filesystem with basically the same values as the problematic one

celia:~# zfs get all share/home/turbo/Crypted | egrep -v 'inherited|default|-$'
NAME                      PROPERTY              VALUE                                  SOURCE
share/home/turbo/Crypted  mountpoint            /home/turbo/Documents/Personal/secret  local
share/home/turbo/Crypted  checksum              sha256-mac                             local
share/home/turbo/Crypted  compression           lz4                                    local
share/home/turbo/Crypted  atime                 on                                     local
share/home/turbo/Crypted  exec                  off                                    local
share/home/turbo/Crypted  copies                2                                      local
share/home/turbo/Crypted  dedup                 on                                     local
share/home/turbo/Crypted  encryption            aes-256-gcm                            local
share/home/turbo/Crypted  keysource             raw,file:///home/turbo/.zfskey         local
share/home/turbo/Crypted  rekeydate             Thu May  2 13:36 2013                  local

succeeds

celia:~# zfs create -o compression=lz4 -o atime=on -o exec=off -o copies=2 -o dedup=on -o encryption=aes-256-gcm -o keysource="raw,file:///home/turbo/.zfskey" share/tmp2

[FransUrbo]

Running 'zfs mount' under strace, I see:

11615 mount("share/home/turbo/Crypted", "/home/turbo/Documents/Personal/secret", "zfs", MS_NOSUID|MS_NODEV|MS_NOEXEC, "rw,noexec,nosuid,nodev,noxattr,zfsutil") = -1 EBADE (Invalid exchange)
11615 write(2, "filesystem 'share/home/turbo/Crypted' can not be mounted due to error 52\n"..., 73) = 73

mounting the newly created encrypted test filesystem share/tmp2, I instead see:

11626 mount("share/tmp2", "/share/tmp2", "zfs", MS_NOSUID|MS_NOEXEC, "rw,noexec,nosuid,noxattr,zfsutil") = 0

The direct difference I could see is the 'nodev', so disabling that in share/tmp2 and tried again. No change, it still tries to use 'nodev' (and succeeds) so that can't be it...

However, MS_NODEV...

Setting 'device=off' again, just for completness on share/home/turbo/Crypted both are now using MS_NODEV and share/tmp2 is still mountable. Still can't mount 'share/home/turbo/Crypted' though.

[lundman]

can the newly created test filesystem be mounted (after export/import) ?

[FransUrbo]

Yes. I've been rebooting a couple of times to see if that did any change and share/tmp2 still mounts without any problems.

[lundman]

Ok so its not a patch code merge breaking mount, there is something with the dataset. You tried disabling the checksum thing? although I would expect to see a degraded pool if that was the issue, and not an error from mount

[FransUrbo]

cannot set property for 'share/home/turbo/Crypted': 'checksum' cannot be changed on encrypted datasets.

[FransUrbo]

The reason I think/thought it had something to do with the crypto code is that all the other filesystems (591) mounts just fine...

[FransUrbo]

Even calling mount.zfs manually fails:

celia:~# /sbin/mount.zfs -vn share/tmp2 /share/tmp2 -o rw,noexec,nosuid,nodev,noxattr,noatime,zfsutil
mount.zfs:
  dataset:    "share/tmp2"
  mountpoint: "/share/tmp2"
  mountflags: 0x40e
  zfsflags:   0x1
  mountopts:  "rw,noexec,nosuid,nodev,noxattr,noatime,zfsutil"
  mtabopts:   "rw,noexec,nosuid,nodev,noxattr,noatime"
celia:~# /sbin/mount.zfs -vn share/home/turbo/Crypted /home/turbo/Documents/Personal/secret -o rw,noexec,nosuid,nodev,noxattr,noatime,zfsutil                 mount.zfs:
  dataset:    "share/home/turbo/Crypted"
  mountpoint: "/home/turbo/Documents/Personal/secret"
  mountflags: 0x40e
  zfsflags:   0x1
  mountopts:  "rw,noexec,nosuid,nodev,noxattr,noatime,zfsutil"
  mtabopts:   "rw,noexec,nosuid,nodev,noxattr,noatime"
filesystem 'share/home/turbo/Crypted' can not be mounted due to error 52

[lundman]

Well, the error comes from the kernel on the mount request, so it should not matter who issues it.

But 52 is EBADE, or in ZFS it is used as:

include//sys/zio.h:#define ECKSUM EBADE

So, it is failing due to checksum. Take out the returns for bad checksums and you should be able to get further maybe, copy the data off etc.

[FransUrbo]

Ok, so trying to disable the checksum again works to mount the filesystem. I can now copy the data out and recreate the filesystem.

Thanx for the heads up on that. It didn't work for openzfs/zfs#1862 though. But this bug is now resolved.

Maybe we/someone should document the issue about the modules loading... ?

[lundman]

Yes it is a bit unfortunate. The new version is clearly better as it actually HAS checksums, but it will confuse users that they can no longer import old crypto volumes. Perhaps documentation is in order, and then maybe make the "ignore checksum" be a module loading optional argument, so it can be more easily disabled.

[FransUrbo]

Sounds like a plan... Do you want me to do an attempt at this, or do you have time (it will probably take you a lot quicker than me :)?

Both documentation and module param is probably required... Can't expect people to read documentation, BEFORE they do something, right? :)

[lundman]

Please feel free to do documentation, I am terrible at that :) I can
probably look at the module loading thing eventually, unless you want to
have a crack at it. If so, feel free to do that.

:)

Turbo Fredriksson wrote:

Sounds like a plan... Do you want me to do an attempt at this, or do you
have time (it will probably take you a lot quicker than me :)?

Both documentation and module param is probably required... Can't expect
people to read documentation, BEFORE they do something, right? :)

—
Reply to this email directly or view it on GitHub
#34 (comment).

Jorgen Lundman | [email protected]
Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell)
Japan | +81 (0)3 -3375-1767 (home)