kernel panic when LZ4_uncompress_unknownOutputSize

[loyou]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	trusty(14.04)
Linux Kernel	4.4.0-45-generic #66~14.04.1
Architecture	x64
ZFS Version	0.6.5.8
SPL Version	0.6.5.8

Describe the problem you're observing

kernel panic and system crashed.

Describe how to reproduce the problem

there is no reproduce way. On our server, we take snapshots and send.

Include any warning/errors/backtraces from the system logs

[10844986.528494] BUG: unable to handle kernel paging request at ffffc90b5b6bf000
[10844986.537566] IP: [<ffffffffc06925e3>] LZ4_uncompress_unknownOutputSize+0x203/0x320 [zfs]
[10844986.546978] PGD 27dc0a2067 PUD 1edde6e067 PMD 1f272db067 PTE 8000820504026163
[10844986.556426] Oops: 000b [#1] SMP 
[10844986.561225] Modules linked in: binfmt_misc ip_vs_rr dummy xt_mac xt_connmark xt_multiport xt_mark xt_socket nf_defrag_ipv6 aufs sch_sfq sch_htb xt_nat iptable_nat nf_nat_ipv4 act_police cls_u32 sch_ingress xt_comment xt_iprange xt_physdev ebt_dnat ebt_ip ebtable_nat ebt_arp veth xt_conntrack nbd xt_TCPMSS xt_CT xt_set ip_set_hash_net ip_set iptable_raw nf_conntrack_netlink nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter netconsole configfs xt_CHECKSUM xt_tcpudp iptable_mangle ip_tables x_tables iscsi_trgt(OE) ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_vs nf_nat nf_conntrack_pptp nf_conntrack_proto_gre br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack xfs ipmi_ssif ipmi_devintf
[10844986.613683]  zfs(POE) bonding intel_rapl mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul zunicode(POE) aesni_intel dcdbas zcommon(POE) znvpair(POE) aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd spl(OE) zavl(POE) vhost_net vhost macvtap macvlan shannon(OE) kvm_intel mei_me mei kvm sb_edac shpchp edac_core lpc_ich ipmi_si ipmi_msghandler 8250_fintek acpi_power_meter irqbypass mac_hid libcrc32c wmi lp parport aacraid ixgbe igb vxlan i2c_algo_bit ip6_udp_tunnel ahci dca udp_tunnel libahci megaraid_sas ptp fjes pps_core mdio
[10844986.660293] CPU: 12 PID: 7924 Comm: z_rd_int_1 Tainted: P           OE   4.4.0-45-generic #66~14.04.1
[10844986.672588] Hardware name: Dell Inc. PowerEdge R730/0WCJNT, BIOS 2.2.5 09/06/2016
[10844986.684859] task: ffff8827b5d13700 ti: ffff8827bae14000 task.ti: ffff8827bae14000
[10844986.697161] RIP: 0010:[<ffffffffc06925e3>]  [<ffffffffc06925e3>] LZ4_uncompress_unknownOutputSize+0x203/0x320 [zfs]
[10844986.709606] RSP: 0018:ffff8827bae17b78  EFLAGS: 00010286
[10844986.715765] RAX: ffffc90b5b6dc000 RBX: 8100b50010301c02 RCX: ffffc90b5b6bf000
[10844986.728014] RDX: ffffc90b5b6bf005 RSI: ffffc902adf64b62 RDI: ffffc902adf5d004
[10844986.740433] RBP: ffff8827bae17c20 R08: ffffc90b5b6bc9eb R09: ffffc902adf5de6e
[10844986.752851] R10: ffffc90b5b6dbff8 R11: ffffc902adf64b5a R12: ffffc90b5b6bc000
[10844986.765285] R13: ffffc90b5b6dbff7 R14: 000000000000000f R15: ffffc90b5b6bc97b
[10844986.777681] FS:  0000000000000000(0000) GS:ffff8827dc900000(0000) knlGS:0000000000000000
[10844986.790096] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[10844986.796305] CR2: ffffc90b5b6bf000 CR3: 000000106d963000 CR4: 00000000003426e0
[10844986.808462] Stack:
[10844986.814337]  0000000000000000 0000000000000003 0000000000000002 0000000000000003
[10844986.826144]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[10844986.837960]  0000000000000000 0000000000000000 0000000000000000 ffffffffffffffff
[10844986.849909] Call Trace:
[10844986.855777]  [<ffffffffc0692f3c>] lz4_decompress_zfs+0x4c/0x60 [zfs]
[10844986.861725]  [<ffffffffc06fe810>] zio_decompress_data+0x40/0x50 [zfs]
[10844986.867556]  [<ffffffffc06f87dd>] zio_decompress+0x3d/0x50 [zfs]
[10844986.873241]  [<ffffffffc06f7c7f>] zio_pop_transforms+0x6f/0x90 [zfs]
[10844986.878832]  [<ffffffffc06fd3c7>] zio_done+0x197/0xe10 [zfs]
[10844986.884264]  [<ffffffff811ddba4>] ? kfree+0x114/0x150
[10844986.889624]  [<ffffffffc03c171a>] ? spl_kmem_free+0x2a/0x40 [spl]
[10844986.894989]  [<ffffffffc06bd570>] ? vdev_mirror_map_free+0x20/0x30 [zfs]
[10844986.900291]  [<ffffffffc06fda8e>] zio_done+0x85e/0xe10 [zfs]
[10844986.905482]  [<ffffffffc06f7e94>] ? zio_checksum_verify+0x24/0x90 [zfs]
[10844986.910623]  [<ffffffffc06fda8e>] zio_done+0x85e/0xe10 [zfs]
[10844986.915615]  [<ffffffff817f8852>] ? mutex_lock+0x12/0x2f
[10844986.920525]  [<ffffffffc06f8933>] zio_execute+0xc3/0x170 [zfs]
[10844986.925316]  [<ffffffffc03c51b1>] taskq_thread+0x211/0x420 [spl]
[10844986.930012]  [<ffffffff810a7640>] ? wake_up_q+0x80/0x80
[10844986.934600]  [<ffffffffc03c4fa0>] ? taskq_thread_spawn+0x50/0x50 [spl]
[10844986.939129]  [<ffffffff8109b959>] kthread+0xc9/0xe0
[10844986.943532]  [<ffffffff8109b890>] ? kthread_park+0x60/0x60
[10844986.947839]  [<ffffffff817fa8cf>] ret_from_fork+0x3f/0x70
[10844986.952071]  [<ffffffff8109b890>] ? kthread_park+0x60/0x60
[10844986.956226] Code: f1 0f 82 f1 fe ff ff 48 83 ec 80 89 c8 5b 44 29 e0 41 5c 41 5d 41 5e 41 5f 5d c3 48 39 ca 73 de 49 8b 18 48 83 c2 08 49 83 c0 08 <48> 89 5a f8 48 39 d1 77 ec eb c8 4c 39 c6 77 13 e9 d4 fe ff ff 
[10844986.969200] RIP  [<ffffffffc06925e3>] LZ4_uncompress_unknownOutputSize+0x203/0x320 [zfs]
[10844986.977660]  RSP <ffff8827bae17b78>
[10844986.981757] CR2: ffffc90b5b6bf000

[kernelOfTruth]

related:

https://www.illumos.org/issues/5522

https://lkml.org/lkml/2014/7/3/833 [PATCH] lz4: add overrun checks to lz4_uncompress_unknownoutputsize()

https://github.com/zfsonlinux/zfs/blob/4e33ba4c389f59b74138bf7130e924a4230d64e9/module/zfs/lz4.c#L903

[loyou]

seems kernel has applied this patch, is there any plan to apply?

[loyou]

I also checked the kernel git tree, another data abort patch also applied,
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/lib/lz4/lz4_decompress.c?id=d5e7cafd69da24e6d6cc988fab6ea313a2577efc