arc_prune list_del corruption with 0.8.4

[stuartthebruce]

System information

Type	Version/Name
Distribution Name	Scientific Linux
Distribution Version	7.8
Linux Kernel	3.10.0-1127.13.1.el7
Architecture	x86_64
ZFS Version	0.8.4
SPL Version	0.8.4

Describe the problem you're observing

kernel panic

Describe how to reproduce the problem

Was running some tests on a large directory with 75M files, but system has not crashed again after a few hours of additional tests.

Include any warning/errors/backtraces from the system logs

[3863467.892069] ------------[ cut here ]------------
[3863467.892082] WARNING: CPU: 2 PID: 267281 at lib/list_debug.c:53 __list_del_entry+0x63/0xd0
[3863467.892084] list_del corruption, ffff95f326db1b90->next is LIST_POISON1 (dead000000000100)
[3863467.892086] Modules linked in: nf_conntrack_netlink nf_conntrack nfnetlink fuse btrfs raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nfsv4 dns_resolver netlink_diag rpcsec_gss_krb5 nfsv3 nfs fscache 8021q garp mrp stp llc nvmet_rdma bonding ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt i40iw rpcrdma rdma_ucm ib_iser ib_umad libiscsi ib_ipoib scsi_transport_iscsi iTCO_wdt iTCO_vendor_support dm_service_time skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mlx5_ib ib_uverbs pcspkr ses enclosure lpc_ich i2c_i801 joydev sg mei_me mei wmi ipmi_si ipmi_devintf ipmi_msghandler zfs(POE) zunicode(POE) zlua(POE) zcommon(POE) znvpair(POE) zavl(POE) icp(POE)
[3863467.892146]  spl(OE) dax_pmem device_dax dm_multipath acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sch_fq sunrpc tcp_htcp binfmt_misc ip_tables xfs libcrc32c nd_pmem nd_btt raid1 sd_mod crc_t10dif crct10dif_generic mlx5_core ast i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect mpt3sas i40e sysimgblt fb_sys_fops ttm ixgbe mlxfw ahci devlink libahci drm raid_class scsi_transport_sas libata nvme mdio ptp pps_core dca nfit drm_panel_orientation_quirks libnvdimm dm_mirror dm_region_hash dm_log dm_mod nvmet nvme_rdma nvme_fabrics rdma_cm iw_cm ib_cm ib_core nvme_core
[3863467.892188] CPU: 2 PID: 267281 Comm: arc_prune Kdump: loaded Tainted: P           OE  ------------   3.10.0-1127.13.1.el7.x86_64 #1
[3863467.892189] Hardware name: Supermicro SYS-2029U-TN24R4T/X11DPU, BIOS 3.2 10/16/2019
[3863467.892191] Call Trace:
[3863467.892200]  [<ffffffffbbf7ffa5>] dump_stack+0x19/0x1b
[3863467.892205]  [<ffffffffbb89bd18>] __warn+0xd8/0x100
[3863467.892208]  [<ffffffffbb89bd9f>] warn_slowpath_fmt+0x5f/0x80
[3863467.892218]  [<ffffffffc0f64820>] ? spl_kmem_zalloc+0xe0/0x140 [spl]
[3863467.892220]  [<ffffffffbbba4d23>] __list_del_entry+0x63/0xd0
[3863467.892224]  [<ffffffffbba663bf>] __dentry_kill+0x7f/0x1d0
[3863467.892226]  [<ffffffffbba66b35>] dput+0xb5/0x1a0
[3863467.892228]  [<ffffffffbba66f46>] d_prune_aliases+0xb6/0xf0
[3863467.892282]  [<ffffffffc15b26c3>] zfs_prune+0x253/0x2a0 [zfs]
[3863467.892312]  [<ffffffffc15e0105>] zpl_prune_sb+0x35/0x50 [zfs]
[3863467.892324]  [<ffffffffc14c97b2>] arc_prune_task+0x22/0x40 [zfs]
[3863467.892330]  [<ffffffffc0f69a9c>] taskq_thread+0x2ac/0x4f0 [spl]
[3863467.892344]  [<ffffffffbb8db990>] ? wake_up_state+0x20/0x20
[3863467.892350]  [<ffffffffc0f697f0>] ? taskq_thread_spawn+0x60/0x60 [spl]
[3863467.892355]  [<ffffffffbb8c6691>] kthread+0xd1/0xe0
[3863467.892358]  [<ffffffffbb8c65c0>] ? insert_kthread_work+0x40/0x40
[3863467.892364]  [<ffffffffbbf92d37>] ret_from_fork_nospec_begin+0x21/0x21
[3863467.892366]  [<ffffffffbb8c65c0>] ? insert_kthread_work+0x40/0x40
[3863467.892368] ---[ end trace 6f89f935d2b303b3 ]---
[3863467.892376] general protection fault: 0000 [#1] SMP
[3863467.892878] Modules linked in: nf_conntrack_netlink nf_conntrack nfnetlink fuse btrfs raid6_pq xor vfat msdos fat ext4 mbcache jbd2 nfsv4 dns_resolver netlink_diag rpcsec_gss_krb5 nfsv3 nfs fscache 8021q garp mrp stp llc nvmet_rdma bonding ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt i40iw rpcrdma rdma_ucm ib_iser ib_umad libiscsi ib_ipoib scsi_transport_iscsi iTCO_wdt iTCO_vendor_support dm_service_time skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mlx5_ib ib_uverbs pcspkr ses enclosure lpc_ich i2c_i801 joydev sg mei_me mei wmi ipmi_si ipmi_devintf ipmi_msghandler zfs(POE) zunicode(POE) zlua(POE) zcommon(POE) znvpair(POE) zavl(POE) icp(POE)
[3863467.896020]  spl(OE) dax_pmem device_dax dm_multipath acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sch_fq sunrpc tcp_htcp binfmt_misc ip_tables xfs libcrc32c nd_pmem nd_btt raid1 sd_mod crc_t10dif crct10dif_generic mlx5_core ast i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect mpt3sas i40e sysimgblt fb_sys_fops ttm ixgbe mlxfw ahci devlink libahci drm raid_class scsi_transport_sas libata nvme mdio ptp pps_core dca nfit drm_panel_orientation_quirks libnvdimm dm_mirror dm_region_hash dm_log dm_mod nvmet nvme_rdma nvme_fabrics rdma_cm iw_cm ib_cm ib_core nvme_core
[3863467.899800] CPU: 2 PID: 267281 Comm: arc_prune Kdump: loaded Tainted: P        W  OE  ------------   3.10.0-1127.13.1.el7.x86_64 #1
[3863467.901113] Hardware name: Supermicro SYS-2029U-TN24R4T/X11DPU, BIOS 3.2 10/16/2019
[3863467.901799] task: ffff95bd92951070 ti: ffff959bb06a8000 task.ti: ffff959bb06a8000
[3863467.902497] RIP: 0010:[<ffffffffbba663d3>]  [<ffffffffbba663d3>] __dentry_kill+0x93/0x1d0
[3863467.903205] RSP: 0018:ffff959bb06abc28  EFLAGS: 00010286
[3863467.904054] RAX: dead000000000100 RBX: ffff95f326db1b00 RCX: 0000000000000006
[3863467.904772] RDX: 00000000000000a0 RSI: 0000000000000000 RDI: 0000000000000009
[3863467.905547] RBP: ffff959bb06abc40 R08: 000000000000000a R09: 0000000000000000
[3863467.906293] R10: 00000000000303fd R11: ffff959bb06ab85e R12: 0000000000000000
[3863467.906982] R13: ffff95f326db1b58 R14: 0000000000000000 R15: ffff95ac96238000
[3863467.907669] FS:  0000000000000000(0000) GS:ffff95c97ec80000(0000) knlGS:0000000000000000
[3863467.908355] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3863467.909043] CR2: 00007fb29bf76c70 CR3: 0000000940f3a000 CR4: 00000000007607e0
[3863467.909743] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3863467.910440] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3863467.911123] PKRU: 00000000
[3863467.911799] Call Trace:
[3863467.912474]  [<ffffffffbba66b35>] dput+0xb5/0x1a0
[3863467.913269]  [<ffffffffbba66f46>] d_prune_aliases+0xb6/0xf0
[3863467.913997]  [<ffffffffc15b26c3>] zfs_prune+0x253/0x2a0 [zfs]
[3863467.914681]  [<ffffffffc15e0105>] zpl_prune_sb+0x35/0x50 [zfs]
[3863467.915339]  [<ffffffffc14c97b2>] arc_prune_task+0x22/0x40 [zfs]
[3863467.915988]  [<ffffffffc0f69a9c>] taskq_thread+0x2ac/0x4f0 [spl]
[3863467.916626]  [<ffffffffbb8db990>] ? wake_up_state+0x20/0x20
[3863467.917257]  [<ffffffffc0f697f0>] ? taskq_thread_spawn+0x60/0x60 [spl]
[3863467.917885]  [<ffffffffbb8c6691>] kthread+0xd1/0xe0
[3863467.918505]  [<ffffffffbb8c65c0>] ? insert_kthread_work+0x40/0x40
[3863467.919254]  [<ffffffffbbf92d37>] ret_from_fork_nospec_begin+0x21/0x21
[3863467.919851]  [<ffffffffbb8c65c0>] ? insert_kthread_work+0x40/0x40
[3863467.920440] Code: 10 00 48 8d bb 90 00 00 00 48 3b bb 90 00 00 00 74 26 e8 01 e9 13 00 48 8b 83 90 00 00 00 49 8d 94 24 a0 00 00 00 48 39 d0 74 0d <f6> 80 73 ff ff ff 20 0f 85 11 01 00 00 4d 85 e4 74 0c 49 8d 7c
[3863467.921826] RIP  [<ffffffffbba663d3>] __dentry_kill+0x93/0x1d0
[3863467.922429]  RSP <ffff959bb06abc28>

[stuartthebruce]

The stack trace looks different enough from #9068 that I opened a new issue.

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.