With "casesensitivity=mixed" hitting an assert in ZAP code

[sanjeevbagewadi]

System information

Type	CentOS release 6.8 (Final)
Distribution Name	CentOS
Distribution Version	6.8
Linux Kernel	4.4.14-1.el6
Architecture	x86
ZFS Version	0.7.1-1
SPL Version	0.7.1-1

Describe the problem you're observing

With casesensitivity=mixed was running the following test :
root@NTNX-10-5-137-31-A-FSVM:/home/nutanix# cat names.py
#!/usr/bin/python
import itertools
s="abcdefghijklmnopqrstuvwxyz"
length = len(s)
names = map(''.join, itertools.product(*zip(s.upper(), s.lower())))
for name in names:
print name

root@NTNX-10-5-137-31-A-FSVM:/home/nutanix# ./names.py | while read file

do
touch /test/fs2/dir1/$file
done

And hit the following panic
-- snip --
[ 1.068019] VERIFY(!RW_LOCK_HELD(&l->l_rwlock)) failed
[ 1.068077] PANIC at zap.c:407:zap_leaf_evict_sync()
[ 1.068113] Showing stack for process 67625
[ 1.068116] CPU: 0 PID: 67625 Comm: touch Tainted: P OE 4.4.14-1.el6.nutanix.10272016.x86_64 #1
[ 1.068117] Hardware name: Nutanix AHV, BIOS seabios-1.7.5-11.el6 04/01/2014
[ 1.068122] 0000000000000000 ffff88015312b2a8 ffffffff81319ae3 0000000000000001
[ 1.068125] 0000000100480b8b ffff88015312b2f8 ffffffffa0b5a9c0 ffff88015312b2b8
[ 1.068127] ffffffffa09918c4 ffff88015312b458 ffffffffa0991aeb 0000000000000040
[ 1.068129] Call Trace:
[ 1.068136] [] dump_stack+0x67/0x94
[ 1.068146] [] spl_dumpstack+0x44/0x50 [spl]
[ 1.068150] [] spl_panic+0xcb/0xe0 [spl]
[ 1.068153] [] ? __sg_free_table+0x63/0x90
[ 1.068157] [] ? kmem_cache_free+0x1ee/0x210
[ 1.068160] [] ? spl_kmem_cache_free+0x117/0x140 [spl]
[ 1.068200] [] ? arc_hdr_destroy+0x17c/0x1d0 [zfs]
[ 1.068231] [] zap_leaf_evict_sync+0x57/0x60 [zfs]
[ 1.068248] [] dbuf_evict_user+0x45/0x70 [zfs]
[ 1.068265] [] dbuf_destroy+0x4f/0x330 [zfs]
[ 1.068282] [] dbuf_rele_and_unlock+0x221/0x3e0 [zfs]
[ 1.068313] [] ? zap_lockdir+0x7f/0xa0 [zfs]
[ 1.068344] [] ? zap_grow_ptrtbl+0x186/0x1a0 [zfs]
[ 1.068361] [] dbuf_rele+0x40/0x50 [zfs]
[ 1.068394] [] dmu_buf_rele+0xe/0x10 [zfs]
[ 1.068427] [] zap_put_leaf+0x3d/0x60 [zfs]
[ 1.068460] [] zap_put_leaf_maybe_grow_ptrtbl+0xc7/0x130 [zfs]
[ 1.068492] [] fzap_add_cd+0xd8/0x130 [zfs]
[ 1.068541] [] mzap_upgrade+0x194/0x210 [zfs]
[ 1.068593] [] zap_lockdir_impl+0x25a/0x370 [zfs]
[ 1.068628] [] zap_lockdir+0x7f/0xa0 [zfs]
[ 1.068664] [] zap_add+0x5b/0xa0 [zfs]
[ 1.068668] [] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
[ 1.068703] [] zfs_link_create+0x37f/0x520 [zfs]
[ 1.068761] [] zfs_create+0x62a/0x810 [zfs]
[ 1.068764] [] ? __kmalloc_node+0x1f6/0x2b0
[ 1.068798] [] zpl_create+0xb2/0x160 [zfs]
[ 1.068802] [] vfs_create+0xd4/0x100
[ 1.068804] [] ? lookup_real+0x1d/0x60
[ 1.068806] [] lookup_open+0x173/0x1a0
[ 1.068808] [] do_last+0x299/0x760
[ 1.068811] [] ? get_empty_filp+0xd7/0x1c0
[ 1.068813] [] path_openat+0x7c/0x140
[ 1.068832] [] ? __pte_alloc+0xe2/0x190
[ 1.068834] [] do_filp_open+0x85/0xe0
[ 1.068836] [] ? getname_flags+0xce/0x1f0
[ 1.068838] [] do_sys_open+0x11a/0x220
[ 1.068842] [] ? syscall_trace_enter_phase1+0x133/0x150
[ 1.068844] [] SyS_open+0x1e/0x20
[ 1.068850] [] entry_SYSCALL_64_fastpath+0x12/0x71
-- snip --

Describe how to reproduce the problem

The following are the steps:

• Create zfs dataset with casesensitivity=mixed
• Run the above listed code.

Include any warning/errors/backtraces from the system logs

[sanjeevbagewadi]

The basic problem is as below:
We have a fileset with casesensitivity=mixed. The test case is creating files with same name but, just varying in case:
e.g. ABCDEFGHIJkLmnOpQRSTUVwXyZ ABCDEFGHIJkLMNOpqrSTUVWxyz and so on.

With mixed mode the hash of the filename is computed after the name has been normalised (aka. converted to upper). Thus all the names would endup with the same hash value. And they would be placed in the same zap-leaf-block. Thus when the number of entries exceed the capacity of the leaf block, zap_leaf_split() is called. However, this does not help because, zap_leaf_split() tries to split the entries between the old and new leaf using the bits of the hash. If the bit is set, the entry is moved to new leaf else it is retained in the old leaf. However, since the hash value is exactly the same for all the entries, all the entries either move to new leaf or stay in the old leaf. Thus the split does not result in free space in the leaf. And this eventually results in the failure.

Another variant of this bug is:
-- snip --
crash> bt
PID: 22849 TASK: ffff88032d0fd500 CPU: 1 COMMAND: "touch"
#0 [ffff8802aa92f150] machine_kexec at ffffffff8105d4f0
#1 [ffff8802aa92f1c0] crash_kexec at ffffffff8110c288
#2 [ffff8802aa92f290] oops_end at ffffffff8101a976
#3 [ffff8802aa92f2c0] die at ffffffff8101ae2b
#4 [ffff8802aa92f2f0] do_trap at ffffffff81017e5f
#5 [ffff8802aa92f350] do_error_trap at ffffffff8101810d
#6 [ffff8802aa92f410] do_invalid_op at ffffffff81018260
#7 [ffff8802aa92f420] invalid_op at ffffffff816d12de
[exception RIP: spl_panic+203]
RIP: ffffffffa0991aeb RSP: ffff8802aa92f4d8 RFLAGS: 00010282
RAX: 00000000ffffffff RBX: ffff8802aa92f508 RCX: ffff8802aa92f288
RDX: 0000000100028bfc RSI: 0000000000000001 RDI: 0000000000000282
RBP: ffff8802aa92f668 R8: 000000000000fffe R9: 00000000000007da
R10: 0000000000000000 R11: 00000000000007da R12: ffffffffa0bc9138
R13: 00000000000001c8 R14: ffffffffa0c1d24e R15: 0000000000000001
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff8802aa92f670] zap_put_leaf at ffffffffa0b1ec58 [zfs]
#9 [ffff8802aa92f690] zap_put_leaf_maybe_grow_ptrtbl at ffffffffa0b216a7 [zfs]
#10 [ffff8802aa92f6f0] fzap_add_cd at ffffffffa0b223c2 [zfs]
#11 [ffff8802aa92f790] fzap_add at ffffffffa0b2254f [zfs]
#12 [ffff8802aa92f7f0] zap_add_impl at ffffffffa0b284ee [zfs]
#13 [ffff8802aa92f840] zap_add at ffffffffa0b28711 [zfs]
#14 [ffff8802aa92f8b0] zfs_link_create at ffffffffa0b33961 [zfs]
#15 [ffff8802aa92fa90] zfs_create at ffffffffa0b58dbb [zfs]
#16 [ffff8802aa92fb80] zpl_create at ffffffffa0b7dc4f [zfs]
#17 [ffff8802aa92fc00] vfs_create at ffffffff81210424
#18 [ffff8802aa92fc40] lookup_open at ffffffff812111e3
#19 [ffff8802aa92fcd0] do_last at ffffffff812138d9
#20 [ffff8802aa92fd70] path_openat at ffffffff81213e1c
#21 [ffff8802aa92fdd0] do_filp_open at ffffffff81213f65
#22 [ffff8802aa92fed0] do_sys_open at ffffffff8120311a
#23 [ffff8802aa92ff40] sys_open at ffffffff8120325e
#24 [ffff8802aa92ff50] entry_SYSCALL_64_fastpath at ffffffff816cf76e
-- snip --

Walking through the code :

 822 int
 823 fzap_add_cd(zap_name_t *zn,
 824     uint64_t integer_size, uint64_t num_integers,
 825     const void *val, uint32_t cd, void *tag, dmu_tx_t *tx)
 826 {
.....
 855         if (err == 0) {
 856                 zap_increment_num_entries(zap, 1, tx);
 857                 /* Return the cd for the newly added entry */
 858                 zn->zn_cd = zeh.zeh_cd;
 859         } else if (err == EAGAIN) {
 860                 err = zap_expand_leaf(zn, l, tag, tx, &l);
 861                 zap = zn->zn_zap;       /* zap_expand_leaf() may change zap */
 862                 if (err == 0) {
 863                         goto retry;
 864                 } else {
 865                         cmn_err(CE_WARN, "%d err = %d zn %p l %p\n", __LINE__, err, zn, l);
 866                 }
 867         }
 868
 869 out:
 870         if (zap != NULL) {
 871                 cmn_err(CE_WARN, "%d err = %d zn %p l %p\n", __LINE__, err, zn, l);
 872                 zap_put_leaf_maybe_grow_ptrtbl(zn, l, tag, tx);
 873         }

In the above code, zap_expand_leaf() failed with ENOSPC. It eventually fell
through and zap_put_leaf_maybe_grow_ptrtbl() eventually fails in zap_put_leaf()
as the zap_leaf_t's lock is not held.

Also, zfs_link_create() today does not expect zap_add() to fail.

[sanjeevbagewadi]

We need to mitigate the panic by handling the failure and ensure that zfs_link_create() fails gracefully and bails out. This would mean that the ZPL interfaces zfs_create(), zfs_mkdir(), zfs_rename(), zfs_symlink() and zfs_link() would need to handle the zfs_link_create() failure and undo the changes done.

I am working on the fix.

[sanjeevbagewadi]

Thank you for the alert ! Is there is someplace that I can refer to where the problems are mentioned or listed ? This would be of great help ! Regards, Sanjeev

…

On Tue, Jan 9, 2018 at 10:38 PM, ptx0 ***@***.***> wrote: jfyi there are many other unresolved issues with case sensitivity including deadlocks and extreme CPU overhead (likely related to the deadlocks) especially when sharing over Samba due to its *own* case sensitivity handling and how they interact... it is not recommended for use. the implementation is far too simplistic and "stupid". — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7011 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AK9qqDHX5BP-jC8MVKiKGPY0536MpfhPks5tI50hgaJpZM4RSmpD> .

[behlendorf]

@sanjeevbagewadi a good place to start would be to get the casenorm test cases working in the ZFS Test Suite. This is the last remaining upstream test group which hasn't been updated for Linux. You'll need to uncomment then the tests/runfile/linux.run.

# 'sensitive_none_lookup', 'sensitive_none_delete',
# 'sensitive_formd_lookup', 'sensitive_formd_delete',
# 'insensitive_none_lookup', 'insensitive_none_delete',
# 'insensitive_formd_lookup', 'insensitive_formd_delete',
# 'mixed_none_lookup', 'mixed_none_lookup_ci', 'mixed_none_delete',
# 'mixed_formd_lookup', 'mixed_formd_lookup_ci', 'mixed_formd_delete']
[tests/functional/casenorm]
tests = ['case_all_values', 'norm_all_values']
tags = ['functional', 'casenorm']

[sanjeevbagewadi]

Thank you Brian ! I will check this. The lookup mode (sensitive/insensitive) is not configurable if the protocol implementation is in user land. So, I will need to get some additional work done before I can enable the above tests. Thank you for the alert though ! Because, I have been trying to figure out if anyone is using the mixed mode and what/if any issues were seen. Regards, Sanjeev

…

On Tue, Jan 9, 2018 at 11:55 PM, Brian Behlendorf ***@***.***> wrote: @sanjeevbagewadi <https://github.com/sanjeevbagewadi> a good place to start would be to get the casenorm test cases working in the ZFS Test Suite. This is the last remaining upstream test group which hasn't been updated for Linux. You'll need to uncomment then the tests/runfile/linux.run. # 'sensitive_none_lookup', 'sensitive_none_delete', # 'sensitive_formd_lookup', 'sensitive_formd_delete', # 'insensitive_none_lookup', 'insensitive_none_delete', # 'insensitive_formd_lookup', 'insensitive_formd_delete', # 'mixed_none_lookup', 'mixed_none_lookup_ci', 'mixed_none_delete', # 'mixed_formd_lookup', 'mixed_formd_lookup_ci', 'mixed_formd_delete'] [tests/functional/casenorm] tests = ['case_all_values', 'norm_all_values'] tags = ['functional', 'casenorm'] — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7011 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AK9qqIDxiaSXOkLrmBeDjH_-hGBboYlWks5tI68ygaJpZM4RSmpD> .

[sanjeevbagewadi]

@ptx0, Thank you very much for the pointers !

[andew-lin]

Is this panic exploitable via the remote file system interface, if the mixed mode is enabled on the ZFS volume backing the remote file system share?