page allocation failure #465

jeremysanders · 2011-11-26T09:16:51Z

I got this on Fedora 16 (kernel 3.1.1 on x86_64). It didn't do it on Fedora 14. Is it just using too much memory and needs an ARC limit? The system has 4GB of RAM.

[128413.783954] z_rd_int/2: page allocation failure: order:0, mode:0x20
[128413.784001] Pid: 1132, comm: z_rd_int/2 Tainted: P 3.1.1-2.fc16.x86_64 #1
[128413.784001] Call Trace:
[128413.784001] [] warn_alloc_failed+0x111/0x13d
[128413.784001] [] __alloc_pages_nodemask+0x6b1/0x71d
[128413.784001] [] ? __cmpxchg_double_slab+0x2c/0x9e
[128413.784001] [] ? __cmpxchg_double_slab+0x2c/0x9e
[128413.784001] [] alloc_pages_current+0xbe/0xd8
[128413.784001] [] alloc_slab_page+0x1c/0x4d
[128413.784001] [] new_slab+0x80/0x1cb
[128413.784001] [] __slab_alloc+0x292/0x3b4
[128413.784001] [] ? list_del+0xe/0x2d
[128413.784001] [] ? scsi_pool_alloc_command+0x29/0x68
[128413.784001] [] ? __cmpxchg_double_slab+0x2c/0x9e
[128413.784001] [] ? scsi_pool_alloc_command+0x29/0x68
[128413.784001] [] kmem_cache_alloc+0x5b/0xf8
[128413.784001] [] scsi_pool_alloc_command+0x29/0x68
[128413.784001] [] scsi_host_alloc_command+0x1f/0x6b
[128413.784001] [] __scsi_get_command+0x1b/0x96
[128413.784001] [] scsi_get_command+0x3d/0xa5
[128413.784001] [] scsi_get_cmd_from_req+0x25/0x58
[128413.784001] [] scsi_setup_fs_cmnd+0x5d/0x8c
[128413.784001] [] sd_prep_fn+0x2e3/0xbd5
[128413.784001] [] ? cfq_dispatch_requests+0x777/0x8e7
[128413.784001] [] blk_peek_request+0xd4/0x1d2
[128413.784001] [] scsi_request_fn+0x7d/0x409
[128413.784001] [] __blk_run_queue+0x1b/0x1d
[128413.784001] [] cfq_insert_request+0x4e5/0x517
[128413.784001] [] __elv_add_request+0x194/0x1df
[128413.784001] [] __make_request+0x26e/0x2ab
[128413.784001] [] generic_make_request+0x27d/0x312
[128413.784001] [] ? bvec_alloc_bs+0xae/0xcc
[128413.784001] [] submit_bio+0xde/0xfd
[128413.784001] [] __vdev_disk_physio+0x265/0x2cb [zfs]
[128413.784001] [] vdev_disk_io_start+0xbd/0xd5 [zfs]
[128413.784001] [] zio_vdev_io_start+0x218/0x22a [zfs]
[128413.784001] [] zio_execute+0xb6/0xde [zfs]
[128413.784001] [] vdev_queue_io_done+0x8a/0xb0 [zfs]
[128413.784001] [] zio_vdev_io_done+0x6f/0x15b [zfs]
[128413.784001] [] zio_execute+0xb6/0xde [zfs]
[128413.784001] [] taskq_thread+0x286/0x2b7 [spl]
[128413.784001] [] ? try_to_wake_up+0x1c8/0x1c8
[128413.784001] [] ? __taskq_create+0x32a/0x32a [spl]
[128413.784001] [] kthread+0x84/0x8c
[128413.784001] [] kernel_thread_helper+0x4/0x10
[128413.784001] [] ? kthread_worker_fn+0x148/0x148
[128413.784001] [] ? gs_change+0x13/0x13
[128413.784001] Mem-Info:
[128413.784001] Node 0 DMA per-cpu:
[128413.784001] CPU 0: hi: 0, btch: 1 usd: 0
[128413.784001] CPU 1: hi: 0, btch: 1 usd: 0
[128413.784001] CPU 2: hi: 0, btch: 1 usd: 0
[128413.784001] Node 0 DMA32 per-cpu:
[128413.784001] CPU 0: hi: 186, btch: 31 usd: 55
[128413.784001] CPU 1: hi: 186, btch: 31 usd: 0
[128413.784001] CPU 2: hi: 186, btch: 31 usd: 0
[128413.784001] Node 0 Normal per-cpu:
[128413.784001] CPU 0: hi: 186, btch: 31 usd: 0
[128413.784001] CPU 1: hi: 186, btch: 31 usd: 0
[128413.784001] CPU 2: hi: 186, btch: 31 usd: 0
[128413.784001] active_anon:1940 inactive_anon:5943 isolated_anon:0
[128413.784001] active_file:11258 inactive_file:11204 isolated_file:0
[128413.784001] unevictable:11435 dirty:13 writeback:0 unstable:0
[128413.784001] free:3307 slab_reclaimable:16837 slab_unreclaimable:71224
[128413.784001] mapped:2610 shmem:2 pagetables:930 bounce:0
[128413.784001] Node 0 DMA free:13072kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15644kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[128413.784001] lowmem_reserve[]: 0 3471 3975 3975
[128413.784001] Node 0 DMA32 free:156kB min:14252kB low:17812kB high:21376kB active_anon:6004kB inactive_anon:21876kB active_file:44964kB inactive_file:44532kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3555136kB mlocked:0kB dirty:40kB writeback:0kB mapped:10420kB shmem:8kB slab_reclaimable:42340kB slab_unreclaimable:243380kB kernel_stack:784kB pagetables:752kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:58 all_unreclaimable? no
[128413.784001] lowmem_reserve[]: 0 0 504 504
[128413.784001] Node 0 Normal free:0kB min:2068kB low:2584kB high:3100kB active_anon:1756kB inactive_anon:1896kB active_file:68kB inactive_file:284kB unevictable:45740kB isolated(anon):0kB isolated(file):0kB present:516096kB mlocked:0kB dirty:12kB writeback:0kB mapped:20kB shmem:0kB slab_reclaimable:25008kB slab_unreclaimable:41516kB kernel_stack:2040kB pagetables:2968kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

jeremysanders · 2011-11-28T12:22:53Z

System crashed while being written to with a 2GB arc limit. I'm afraid there's nothing in the logs and the traceback went off the top of the screen.

behlendorf · 2011-11-28T19:05:18Z

This may be related to issue #466 which has to do with the arc meta limit not being as strictly enforced when using a 3.1 kernel. There will have to be a patch made for this before tagged rc7.

jeremysanders · 2011-11-29T09:43:31Z

Thanks - I'll try downgrading the kernel. Keep this open if it is useful to track the issue. Is it useful to get more debugging information?

behlendorf · 2011-11-29T16:54:49Z

If your able to recreate this issue regularly with a 3.1 kernel, I'd love to know if you still see it with say a 3.0 kernel.

jeremysanders · 2011-12-02T12:58:01Z

We're running 2.6.40.4 on the system now. I think this is a redhat invention to work around buggy programs and it is really 3.0.4. It has been pretty stable for the last few days, whereas the 3.1 kernel crashed almost immediately and twice.

behlendorf · 2011-12-02T17:13:50Z

Thanks for the update.

ryao · 2012-04-19T04:25:37Z

@jeremysanders What patches were applied to your 3.1.1 kernel? I recall running ZFS with Linux 3.1.1 a few months ago and I had no issues.

jeremysanders · 2012-04-19T08:52:42Z

This was just a standard Fedora 16 kernel, which doesn't ship with many patches. I haven't tried zfs on a recent Fedora 16 3.3.1 kernel.

behlendorf · 2012-04-19T23:11:09Z

I'd hold off updating to 3.3.1 it currently does build there yet.

cwedgwood · 2012-09-03T22:50:02Z

v3.4.10 something similar

moderately low write pressure on a small machine with 24GB ram

[53276.315170] z_wr_iss/10: page allocation failure: order:5, mode:0x4030
[53276.321762] Pid: 3580, comm: z_wr_iss/10 Tainted: G      D    O 3.4.10-cw3 #6
[53276.329005] Call Trace:
[53276.331498]  [<ffffffff810c06b3>] warn_alloc_failed+0x10f/0x123
[53276.337532]  [<ffffffff814d8ed6>] ? __alloc_pages_direct_compact+0xab/0x192
[53276.344576]  [<ffffffff810a5431>] ? trace_hardirqs_on+0x9/0x20
[53276.350452]  [<ffffffff810c359d>] __alloc_pages_nodemask+0x6f9/0x7b7
[53276.356879]  [<ffffffff810eefb1>] alloc_pages_current+0xc3/0xe0
[53276.362893]  [<ffffffff810bffd2>] __get_free_pages+0xe/0x3b
[53276.368575]  [<ffffffff810f54ab>] kmalloc_order_trace+0x2b/0x5b
[53276.374572]  [<ffffffff810f7807>] __kmalloc+0x35/0xeb
[53276.379655]  [<ffffffffa005145c>] spl_kmem_cache_alloc+0x548/0xa9d [spl]
[53276.386418]  [<ffffffff8105506b>] ? abort_exclusive_wait+0x8f/0x8f
[53276.392748]  [<ffffffffa01bb65d>] zio_buf_alloc+0x22/0x24 [zfs]
[53276.398798]  [<ffffffffa01be111>] zio_vdev_io_start+0xd9/0x222 [zfs]
[53276.405225]  [<ffffffffa01bcff4>] zio_nowait+0xdc/0x103 [zfs]
[53276.411039]  [<ffffffffa018ece7>] vdev_mirror_io_start+0x30a/0x327 [zfs]
[53276.417804]  [<ffffffffa018e664>] ? vdev_config_sync+0x137/0x137 [zfs]
[53276.424390]  [<ffffffffa01834b5>] ? spa_config_enter+0xc4/0xde [zfs]
[53276.430796]  [<ffffffffa01be087>] zio_vdev_io_start+0x4f/0x222 [zfs]
[53276.437214]  [<ffffffffa01bb14d>] zio_execute+0xb8/0xe0 [zfs]
[53276.443009]  [<ffffffffa00531a2>] taskq_thread+0x2aa/0x4d2 [spl]
[53276.449070]  [<ffffffff8105e46c>] ? finish_task_switch+0x82/0xad
[53276.455120]  [<ffffffff81062757>] ? try_to_wake_up+0x1d4/0x1d4
[53276.460996]  [<ffffffffa0052ef8>] ? task_done+0x110/0x110 [spl]
[53276.466943]  [<ffffffff8105496a>] kthread+0x8b/0x93
[53276.471850]  [<ffffffff814e8b34>] kernel_thread_helper+0x4/0x10
[53276.477840]  [<ffffffff814e125d>] ? retint_restore_args+0x13/0x13
[53276.484198]  [<ffffffff810548df>] ? kthread_worker_fn+0x149/0x149
[53276.490384]  [<ffffffff814e8b30>] ? gs_change+0x13/0x13

in fact i have several of these:

# dmesg|grep page.alloc
[53276.315170] z_wr_iss/10: page allocation failure: order:5, mode:0x4030
[53431.111266] z_wr_iss/8: page allocation failure: order:5, mode:0x4030
[53431.117772] z_wr_iss/3: page allocation failure: order:5, mode:0x4030
[53431.118030] z_wr_iss/0: page allocation failure: order:5, mode:0x4030
[53431.118147] z_wr_iss/2: page allocation failure: order:5, mode:0x4030
[53431.118228] z_wr_iss/4: page allocation failure: order:5, mode:0x4030
[53431.118348] z_wr_iss/9: page allocation failure: order:5, mode:0x4030
[53431.137908] z_wr_iss/1: page allocation failure: order:5, mode:0x4030

order-5 seems reasonable on a machine with 19GB free (assuming there was memory pressure to cause some of that to get freed)

MemTotal:       24751384 kB
MemFree:          328676 kB
Buffers:            1572 kB
Cached:         19402352 kB
...
Slab:             513604 kB

behlendorf · 2012-09-04T19:43:47Z

This is a side effect of the emergency objects which were added to the slab, commit openzfs/spl@e2dcc6e .

The order 5 allocation is large, but not unreasonably so. The reason the allocation fails is because it was issued as KM_PUSHPAGE which prevents the direct reclaim path from attempting to issue I/O in order to satisfy the allocation. It will be internally retried by the SPL and eventually succeed because kswapd (which can do I/O) will clear enough space in memory for it.

What we might want to do is suppress the warning for the zio_buf_alloc() since this isn't really harmful or totally unexpected. It's just unfortunate and something we need to live with until we do the work described in issue #75

cwedgwood · 2012-09-04T20:35:52Z

I upped min_free' and I still get quite a few of these. Maybe 10s per hour in some cases (it's not clear why more people don't though).

I would probably prefer we did suppress them when not harmful just to avoid polluting dmesg/kernel.log for now.

behlendorf · 2012-09-05T02:29:15Z

@cwedgwood When you get a chance can you please confirm that the above patch silences those allocation failures you were seeing. Incidentally, the original failures reported in the bug should also be resolved in master.

cwedgwood · 2012-09-05T17:24:46Z

I saw some problems spewing out the serial console when writing lots of small files (rsync into zpl). That wedged things up to the extent kern.log got nothing and I had to ipmi reset.

sys_write ... zpl_write -> dmu_request_arcbuf .... zio_data_buf_alloc -> kmalloc

[ 2829.418388] rsync: page allocation failure: order:5, mode:0x4030
[ 2829.418395] Pid: 6345, comm: rsync Tainted: G           O 3.4.10-cw3 #6
[ 2829.418398] Call Trace:
[ 2829.418417]  [<ffffffff810c06b3>] warn_alloc_failed+0x10f/0x123
[ 2829.418427]  [<ffffffff814d8ed6>] ? __alloc_pages_direct_compact+0xab/0x192
[ 2829.418439]  [<ffffffff810a5431>] ? trace_hardirqs_on+0x9/0x20
[ 2829.418447]  [<ffffffff810c359d>] __alloc_pages_nodemask+0x6f9/0x7b7
[ 2829.418459]  [<ffffffff81064b21>] ? __dequeue_entity+0x33/0x38
[ 2829.418468]  [<ffffffff810eefb1>] alloc_pages_current+0xc3/0xe0
[ 2829.418476]  [<ffffffff810bffd2>] __get_free_pages+0xe/0x3b
[ 2829.418487]  [<ffffffff810f54ab>] kmalloc_order_trace+0x2b/0x5b
[ 2829.418494]  [<ffffffff810f7807>] __kmalloc+0x35/0xeb
[ 2829.418515]  [<ffffffffa002145c>] spl_kmem_cache_alloc+0x548/0xa9d [spl]
[ 2829.418528]  [<ffffffffa001f3a0>] ? kmalloc_nofail+0x2c/0x3e [spl]
[ 2829.418538]  [<ffffffff8105506b>] ? abort_exclusive_wait+0x8f/0x8f
[ 2829.418607]  [<ffffffffa01c1b53>] zio_data_buf_alloc+0x22/0x24 [zfs]
[ 2829.418628]  [<ffffffffa0147cbd>] arc_get_data_buf+0x1be/0x3d5 [zfs]
[ 2829.418648]  [<ffffffffa0148544>] arc_buf_alloc+0xa6/0xd1 [zfs]
[ 2829.418668]  [<ffffffffa014858d>] arc_loan_buf+0x1e/0x3c [zfs]
[ 2829.418692]  [<ffffffffa015243e>] dmu_request_arcbuf+0x3d/0x44 [zfs]
[ 2829.418734]  [<ffffffffa01b8d45>] zfs_write+0x49d/0xaa0 [zfs]
[ 2829.418740]  [<ffffffff810a5413>] ? trace_hardirqs_on_caller+0xf/0x24
[ 2829.418748]  [<ffffffff813e9ae9>] ? sock_aio_read+0x1f/0x2a
[ 2829.418792]  [<ffffffffa01c7132>] zpl_write_common+0x52/0x6a [zfs]
[ 2829.418805]  [<ffffffffa0028acc>] ? crhold+0x1b/0x1e [spl]
[ 2829.418848]  [<ffffffffa01c71a8>] zpl_write+0x5e/0x82 [zfs]
[ 2829.418853]  [<ffffffff81101b1a>] vfs_write+0x98/0xda
[ 2829.418858]  [<ffffffff810a5413>] ? trace_hardirqs_on_caller+0xf/0x24
[ 2829.418863]  [<ffffffff81101ce6>] sys_write+0x43/0x70
[ 2829.418870]  [<ffffffff814e77e9>] system_call_fastpath+0x16/0x1b

Perhaps this should be consider a separate issue?

behlendorf · 2012-09-05T18:21:37Z

@cwedgwood This is pretty much the same issue, see commit d7f163e which actually disables these warning for both these call paths. One of for ARC meta data and one is for ARC data.

Buffers for the ARC are normally backed by the SPL virtual slab. However, if memory is low, AND no slab objects are available, AND a new slab cannot be quickly constructed a new emergency object will be directly allocated. These objects can be as large as order 5 on a system with 4k pages. And because they are allocated with KM_PUSHPAGE, to avoid a potential deadlock, they are not allowed to initiate I/O to satisfy the allocation. This can result in the occasional allocation failure. However, since these allocations are allowed to block and perform operations such as memory compaction they will eventually succeed. Since this is not unexpected (just unlikely) behavior this patch disables the warning for the allocation failure. Signed-off-by: Brian Behlendorf <[email protected]> Issue #465

behlendorf · 2012-09-06T19:54:44Z

Closing issue. These problems should now be resolved in master.

The below excerpt of a backtrace is from a ztest failure when running ZoL's ztest. /openzfs#453 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#454 0x00007f03c806106e in vdev_queue_io (zio=zio@entry=0x7f0350003de0) at ../../module/zfs/vdev_queue.c:747 /openzfs#455 0x00007f03c80818c1 in zio_vdev_io_start (zio=0x7f0350003de0) at ../../module/zfs/zio.c:2659 /openzfs#456 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003de0) at ../../module/zfs/zio.c:1399 /openzfs#457 zio_nowait (zio=0x7f0350003de0) at ../../module/zfs/zio.c:1456 /openzfs#458 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350003a10) at ../../module/zfs/vdev_mirror.c:374 /openzfs#459 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003a10) at ../../module/zfs/zio.c:1399 /openzfs#460 zio_nowait (zio=0x7f0350003a10) at ../../module/zfs/zio.c:1456 /openzfs#461 0x00007f03c806464c in vdev_raidz_io_start (zio=0x7f0350003380) at ../../module/zfs/vdev_raidz.c:1607 /openzfs#462 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003380) at ../../module/zfs/zio.c:1399 /openzfs#463 zio_nowait (zio=0x7f0350003380) at ../../module/zfs/zio.c:1456 /openzfs#464 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350002fb0) at ../../module/zfs/vdev_mirror.c:374 /openzfs#465 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002fb0) at ../../module/zfs/zio.c:1399 /openzfs#466 zio_nowait (zio=0x7f0350002fb0) at ../../module/zfs/zio.c:1456 /openzfs#467 0x00007f03c805ed43 in vdev_mirror_io_done (zio=0x7f033957ebf0) at ../../module/zfs/vdev_mirror.c:499 /openzfs#468 0x00007f03c807a0c0 in zio_vdev_io_done (zio=0x7f033957ebf0) at ../../module/zfs/zio.c:2707 /openzfs#469 0x00007f03c808285b in __zio_execute (zio=0x7f033957ebf0) at ../../module/zfs/zio.c:1399 /openzfs#470 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f0390001330, pio=0x7f033957ebf0) at ../../module/zfs/zio.c:547 /openzfs#471 zio_done (zio=0x7f0390001330) at ../../module/zfs/zio.c:3278 /openzfs#472 0x00007f03c808285b in __zio_execute (zio=0x7f0390001330) at ../../module/zfs/zio.c:1399 /openzfs#473 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4013a00, pio=0x7f0390001330) at ../../module/zfs/zio.c:547 /openzfs#474 zio_done (zio=0x7f03b4013a00) at ../../module/zfs/zio.c:3278 /openzfs#475 0x00007f03c808285b in __zio_execute (zio=0x7f03b4013a00) at ../../module/zfs/zio.c:1399 /openzfs#476 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4014210, pio=0x7f03b4013a00) at ../../module/zfs/zio.c:547 /openzfs#477 zio_done (zio=0x7f03b4014210) at ../../module/zfs/zio.c:3278 /openzfs#478 0x00007f03c808285b in __zio_execute (zio=0x7f03b4014210) at ../../module/zfs/zio.c:1399 /openzfs#479 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4014620, pio=0x7f03b4014210) at ../../module/zfs/zio.c:547 /openzfs#480 zio_done (zio=0x7f03b4014620) at ../../module/zfs/zio.c:3278 /openzfs#481 0x00007f03c807a6d3 in __zio_execute (zio=0x7f03b4014620) at ../../module/zfs/zio.c:1399 /openzfs#482 zio_execute (zio=zio@entry=0x7f03b4014620) at ../../module/zfs/zio.c:1337 /openzfs#483 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#484 0x00007f03c806106e in vdev_queue_io (zio=zio@entry=0x7f0350002be0) at ../../module/zfs/vdev_queue.c:747 /openzfs#485 0x00007f03c80818c1 in zio_vdev_io_start (zio=0x7f0350002be0) at ../../module/zfs/zio.c:2659 /openzfs#486 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002be0) at ../../module/zfs/zio.c:1399 /openzfs#487 zio_nowait (zio=0x7f0350002be0) at ../../module/zfs/zio.c:1456 /openzfs#488 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350002810) at ../../module/zfs/vdev_mirror.c:374 /openzfs#489 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002810) at ../../module/zfs/zio.c:1399 /openzfs#490 zio_nowait (zio=0x7f0350002810) at ../../module/zfs/zio.c:1456 /openzfs#491 0x00007f03c8064593 in vdev_raidz_io_start (zio=0x7f0350001270) at ../../module/zfs/vdev_raidz.c:1591 /openzfs#492 0x00007f03c807f243 in __zio_execute (zio=0x7f0350001270) at ../../module/zfs/zio.c:1399 /openzfs#493 zio_nowait (zio=0x7f0350001270) at ../../module/zfs/zio.c:1456 /openzfs#494 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350001e60) at ../../module/zfs/vdev_mirror.c:374 /openzfs#495 0x00007f03c807f243 in __zio_execute (zio=0x7f0350001e60) at ../../module/zfs/zio.c:1399 /openzfs#496 zio_nowait (zio=0x7f0350001e60) at ../../module/zfs/zio.c:1456 /openzfs#497 0x00007f03c805ed43 in vdev_mirror_io_done (zio=0x7f033a0c39c0) at ../../module/zfs/vdev_mirror.c:499 /openzfs#498 0x00007f03c807a0c0 in zio_vdev_io_done (zio=0x7f033a0c39c0) at ../../module/zfs/zio.c:2707 /openzfs#499 0x00007f03c808285b in __zio_execute (zio=0x7f033a0c39c0) at ../../module/zfs/zio.c:1399 /openzfs#500 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03a8003c00, pio=0x7f033a0c39c0) at ../../module/zfs/zio.c:547 /openzfs#501 zio_done (zio=0x7f03a8003c00) at ../../module/zfs/zio.c:3278 /openzfs#502 0x00007f03c808285b in __zio_execute (zio=0x7f03a8003c00) at ../../module/zfs/zio.c:1399 /openzfs#503 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800c400, pio=0x7f03a8003c00) at ../../module/zfs/zio.c:547 /openzfs#504 zio_done (zio=0x7f038800c400) at ../../module/zfs/zio.c:3278 /openzfs#505 0x00007f03c808285b in __zio_execute (zio=0x7f038800c400) at ../../module/zfs/zio.c:1399 /openzfs#506 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800da00, pio=0x7f038800c400) at ../../module/zfs/zio.c:547 /openzfs#507 zio_done (zio=0x7f038800da00) at ../../module/zfs/zio.c:3278 /openzfs#508 0x00007f03c808285b in __zio_execute (zio=0x7f038800da00) at ../../module/zfs/zio.c:1399 /openzfs#509 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800fd80, pio=0x7f038800da00) at ../../module/zfs/zio.c:547 /openzfs#510 zio_done (zio=0x7f038800fd80) at ../../module/zfs/zio.c:3278 /openzfs#511 0x00007f03c807a6d3 in __zio_execute (zio=0x7f038800fd80) at ../../module/zfs/zio.c:1399 /openzfs#512 zio_execute (zio=zio@entry=0x7f038800fd80) at ../../module/zfs/zio.c:1337 /openzfs#513 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#514 0x00007f03c806119d in vdev_queue_io_done (zio=zio@entry=0x7f03a0010950) at ../../module/zfs/vdev_queue.c:775 /openzfs#515 0x00007f03c807a0e8 in zio_vdev_io_done (zio=0x7f03a0010950) at ../../module/zfs/zio.c:2686 /openzfs#516 0x00007f03c807a6d3 in __zio_execute (zio=0x7f03a0010950) at ../../module/zfs/zio.c:1399 /openzfs#517 zio_execute (zio=0x7f03a0010950) at ../../module/zfs/zio.c:1337 /openzfs#518 0x00007f03c7fcd0c4 in taskq_thread (arg=0x966d50) at ../../lib/libzpool/taskq.c:215 /openzfs#519 0x00007f03c7fc7937 in zk_thread_helper (arg=0x967e90) at ../../lib/libzpool/kernel.c:135 /openzfs#520 0x00007f03c78890a3 in start_thread (arg=0x7f03c2703700) at pthread_create.c:309 /openzfs#521 0x00007f03c75c50fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 The backtrace was an infinite loop of `vdev_queue_io_to_issue()` invoking `zio_execute()` until it overran the stack. vdev_queue_io_to_issue() will ony invoke `zio_execute()` on raidz vdevs when aggregation I/Os are generated to improve aggregation continuity. These I/Os do not trigger any writes. However, it appears that they can be generated in such a way that they recurse infinitely upon return to `vdev_queue_io_to_issue()`. As a consequence, we see the number of parents by 1 each time the recursion returns to `vdev_raidz_io_start()`. Signed-off-by: Richard Yao <[email protected]>

The below excerpt of a backtrace is from a ztest failure when running ZoL's ztest. /openzfs#453 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#454 0x00007f03c806106e in vdev_queue_io (zio=zio@entry=0x7f0350003de0) at ../../module/zfs/vdev_queue.c:747 /openzfs#455 0x00007f03c80818c1 in zio_vdev_io_start (zio=0x7f0350003de0) at ../../module/zfs/zio.c:2659 /openzfs#456 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003de0) at ../../module/zfs/zio.c:1399 /openzfs#457 zio_nowait (zio=0x7f0350003de0) at ../../module/zfs/zio.c:1456 /openzfs#458 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350003a10) at ../../module/zfs/vdev_mirror.c:374 /openzfs#459 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003a10) at ../../module/zfs/zio.c:1399 /openzfs#460 zio_nowait (zio=0x7f0350003a10) at ../../module/zfs/zio.c:1456 /openzfs#461 0x00007f03c806464c in vdev_raidz_io_start (zio=0x7f0350003380) at ../../module/zfs/vdev_raidz.c:1607 /openzfs#462 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003380) at ../../module/zfs/zio.c:1399 /openzfs#463 zio_nowait (zio=0x7f0350003380) at ../../module/zfs/zio.c:1456 /openzfs#464 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350002fb0) at ../../module/zfs/vdev_mirror.c:374 /openzfs#465 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002fb0) at ../../module/zfs/zio.c:1399 /openzfs#466 zio_nowait (zio=0x7f0350002fb0) at ../../module/zfs/zio.c:1456 /openzfs#467 0x00007f03c805ed43 in vdev_mirror_io_done (zio=0x7f033957ebf0) at ../../module/zfs/vdev_mirror.c:499 /openzfs#468 0x00007f03c807a0c0 in zio_vdev_io_done (zio=0x7f033957ebf0) at ../../module/zfs/zio.c:2707 /openzfs#469 0x00007f03c808285b in __zio_execute (zio=0x7f033957ebf0) at ../../module/zfs/zio.c:1399 /openzfs#470 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f0390001330, pio=0x7f033957ebf0) at ../../module/zfs/zio.c:547 /openzfs#471 zio_done (zio=0x7f0390001330) at ../../module/zfs/zio.c:3278 /openzfs#472 0x00007f03c808285b in __zio_execute (zio=0x7f0390001330) at ../../module/zfs/zio.c:1399 /openzfs#473 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4013a00, pio=0x7f0390001330) at ../../module/zfs/zio.c:547 /openzfs#474 zio_done (zio=0x7f03b4013a00) at ../../module/zfs/zio.c:3278 /openzfs#475 0x00007f03c808285b in __zio_execute (zio=0x7f03b4013a00) at ../../module/zfs/zio.c:1399 /openzfs#476 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4014210, pio=0x7f03b4013a00) at ../../module/zfs/zio.c:547 /openzfs#477 zio_done (zio=0x7f03b4014210) at ../../module/zfs/zio.c:3278 /openzfs#478 0x00007f03c808285b in __zio_execute (zio=0x7f03b4014210) at ../../module/zfs/zio.c:1399 /openzfs#479 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4014620, pio=0x7f03b4014210) at ../../module/zfs/zio.c:547 /openzfs#480 zio_done (zio=0x7f03b4014620) at ../../module/zfs/zio.c:3278 /openzfs#481 0x00007f03c807a6d3 in __zio_execute (zio=0x7f03b4014620) at ../../module/zfs/zio.c:1399 /openzfs#482 zio_execute (zio=zio@entry=0x7f03b4014620) at ../../module/zfs/zio.c:1337 /openzfs#483 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#484 0x00007f03c806106e in vdev_queue_io (zio=zio@entry=0x7f0350002be0) at ../../module/zfs/vdev_queue.c:747 /openzfs#485 0x00007f03c80818c1 in zio_vdev_io_start (zio=0x7f0350002be0) at ../../module/zfs/zio.c:2659 /openzfs#486 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002be0) at ../../module/zfs/zio.c:1399 /openzfs#487 zio_nowait (zio=0x7f0350002be0) at ../../module/zfs/zio.c:1456 /openzfs#488 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350002810) at ../../module/zfs/vdev_mirror.c:374 /openzfs#489 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002810) at ../../module/zfs/zio.c:1399 /openzfs#490 zio_nowait (zio=0x7f0350002810) at ../../module/zfs/zio.c:1456 /openzfs#491 0x00007f03c8064593 in vdev_raidz_io_start (zio=0x7f0350001270) at ../../module/zfs/vdev_raidz.c:1591 /openzfs#492 0x00007f03c807f243 in __zio_execute (zio=0x7f0350001270) at ../../module/zfs/zio.c:1399 /openzfs#493 zio_nowait (zio=0x7f0350001270) at ../../module/zfs/zio.c:1456 /openzfs#494 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350001e60) at ../../module/zfs/vdev_mirror.c:374 /openzfs#495 0x00007f03c807f243 in __zio_execute (zio=0x7f0350001e60) at ../../module/zfs/zio.c:1399 /openzfs#496 zio_nowait (zio=0x7f0350001e60) at ../../module/zfs/zio.c:1456 /openzfs#497 0x00007f03c805ed43 in vdev_mirror_io_done (zio=0x7f033a0c39c0) at ../../module/zfs/vdev_mirror.c:499 /openzfs#498 0x00007f03c807a0c0 in zio_vdev_io_done (zio=0x7f033a0c39c0) at ../../module/zfs/zio.c:2707 /openzfs#499 0x00007f03c808285b in __zio_execute (zio=0x7f033a0c39c0) at ../../module/zfs/zio.c:1399 /openzfs#500 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03a8003c00, pio=0x7f033a0c39c0) at ../../module/zfs/zio.c:547 /openzfs#501 zio_done (zio=0x7f03a8003c00) at ../../module/zfs/zio.c:3278 /openzfs#502 0x00007f03c808285b in __zio_execute (zio=0x7f03a8003c00) at ../../module/zfs/zio.c:1399 /openzfs#503 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800c400, pio=0x7f03a8003c00) at ../../module/zfs/zio.c:547 /openzfs#504 zio_done (zio=0x7f038800c400) at ../../module/zfs/zio.c:3278 /openzfs#505 0x00007f03c808285b in __zio_execute (zio=0x7f038800c400) at ../../module/zfs/zio.c:1399 /openzfs#506 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800da00, pio=0x7f038800c400) at ../../module/zfs/zio.c:547 /openzfs#507 zio_done (zio=0x7f038800da00) at ../../module/zfs/zio.c:3278 /openzfs#508 0x00007f03c808285b in __zio_execute (zio=0x7f038800da00) at ../../module/zfs/zio.c:1399 /openzfs#509 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800fd80, pio=0x7f038800da00) at ../../module/zfs/zio.c:547 /openzfs#510 zio_done (zio=0x7f038800fd80) at ../../module/zfs/zio.c:3278 /openzfs#511 0x00007f03c807a6d3 in __zio_execute (zio=0x7f038800fd80) at ../../module/zfs/zio.c:1399 /openzfs#512 zio_execute (zio=zio@entry=0x7f038800fd80) at ../../module/zfs/zio.c:1337 /openzfs#513 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#514 0x00007f03c806119d in vdev_queue_io_done (zio=zio@entry=0x7f03a0010950) at ../../module/zfs/vdev_queue.c:775 /openzfs#515 0x00007f03c807a0e8 in zio_vdev_io_done (zio=0x7f03a0010950) at ../../module/zfs/zio.c:2686 /openzfs#516 0x00007f03c807a6d3 in __zio_execute (zio=0x7f03a0010950) at ../../module/zfs/zio.c:1399 /openzfs#517 zio_execute (zio=0x7f03a0010950) at ../../module/zfs/zio.c:1337 /openzfs#518 0x00007f03c7fcd0c4 in taskq_thread (arg=0x966d50) at ../../lib/libzpool/taskq.c:215 /openzfs#519 0x00007f03c7fc7937 in zk_thread_helper (arg=0x967e90) at ../../lib/libzpool/kernel.c:135 /openzfs#520 0x00007f03c78890a3 in start_thread (arg=0x7f03c2703700) at pthread_create.c:309 /openzfs#521 0x00007f03c75c50fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 The backtrace was an infinite loop of `vdev_queue_io_to_issue()` invoking `zio_execute()` until it overran the stack. vdev_queue_io_to_issue() will ony invoke `zio_execute()` on raidz vdevs when aggregation I/Os are generated to improve aggregation continuity. These I/Os do not trigger any writes. However, it appears that they can be generated in such a way that they recurse infinitely upon return to `vdev_queue_io_to_issue()`. Signed-off-by: Richard Yao <[email protected]>

The below excerpt of a backtrace is from a ztest failure when running ZoL's ztest: /openzfs#453 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#454 0x00007f03c806106e in vdev_queue_io (zio=zio@entry=0x7f0350003de0) at ../../module/zfs/vdev_queue.c:747 /openzfs#455 0x00007f03c80818c1 in zio_vdev_io_start (zio=0x7f0350003de0) at ../../module/zfs/zio.c:2659 /openzfs#456 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003de0) at ../../module/zfs/zio.c:1399 /openzfs#457 zio_nowait (zio=0x7f0350003de0) at ../../module/zfs/zio.c:1456 /openzfs#458 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350003a10) at ../../module/zfs/vdev_mirror.c:374 /openzfs#459 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003a10) at ../../module/zfs/zio.c:1399 /openzfs#460 zio_nowait (zio=0x7f0350003a10) at ../../module/zfs/zio.c:1456 /openzfs#461 0x00007f03c806464c in vdev_raidz_io_start (zio=0x7f0350003380) at ../../module/zfs/vdev_raidz.c:1607 /openzfs#462 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003380) at ../../module/zfs/zio.c:1399 /openzfs#463 zio_nowait (zio=0x7f0350003380) at ../../module/zfs/zio.c:1456 /openzfs#464 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350002fb0) at ../../module/zfs/vdev_mirror.c:374 /openzfs#465 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002fb0) at ../../module/zfs/zio.c:1399 /openzfs#466 zio_nowait (zio=0x7f0350002fb0) at ../../module/zfs/zio.c:1456 /openzfs#467 0x00007f03c805ed43 in vdev_mirror_io_done (zio=0x7f033957ebf0) at ../../module/zfs/vdev_mirror.c:499 /openzfs#468 0x00007f03c807a0c0 in zio_vdev_io_done (zio=0x7f033957ebf0) at ../../module/zfs/zio.c:2707 /openzfs#469 0x00007f03c808285b in __zio_execute (zio=0x7f033957ebf0) at ../../module/zfs/zio.c:1399 /openzfs#470 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f0390001330, pio=0x7f033957ebf0) at ../../module/zfs/zio.c:547 /openzfs#471 zio_done (zio=0x7f0390001330) at ../../module/zfs/zio.c:3278 /openzfs#472 0x00007f03c808285b in __zio_execute (zio=0x7f0390001330) at ../../module/zfs/zio.c:1399 /openzfs#473 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4013a00, pio=0x7f0390001330) at ../../module/zfs/zio.c:547 /openzfs#474 zio_done (zio=0x7f03b4013a00) at ../../module/zfs/zio.c:3278 /openzfs#475 0x00007f03c808285b in __zio_execute (zio=0x7f03b4013a00) at ../../module/zfs/zio.c:1399 /openzfs#476 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4014210, pio=0x7f03b4013a00) at ../../module/zfs/zio.c:547 /openzfs#477 zio_done (zio=0x7f03b4014210) at ../../module/zfs/zio.c:3278 /openzfs#478 0x00007f03c808285b in __zio_execute (zio=0x7f03b4014210) at ../../module/zfs/zio.c:1399 /openzfs#479 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4014620, pio=0x7f03b4014210) at ../../module/zfs/zio.c:547 /openzfs#480 zio_done (zio=0x7f03b4014620) at ../../module/zfs/zio.c:3278 /openzfs#481 0x00007f03c807a6d3 in __zio_execute (zio=0x7f03b4014620) at ../../module/zfs/zio.c:1399 /openzfs#482 zio_execute (zio=zio@entry=0x7f03b4014620) at ../../module/zfs/zio.c:1337 /openzfs#483 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#484 0x00007f03c806106e in vdev_queue_io (zio=zio@entry=0x7f0350002be0) at ../../module/zfs/vdev_queue.c:747 /openzfs#485 0x00007f03c80818c1 in zio_vdev_io_start (zio=0x7f0350002be0) at ../../module/zfs/zio.c:2659 /openzfs#486 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002be0) at ../../module/zfs/zio.c:1399 /openzfs#487 zio_nowait (zio=0x7f0350002be0) at ../../module/zfs/zio.c:1456 /openzfs#488 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350002810) at ../../module/zfs/vdev_mirror.c:374 /openzfs#489 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002810) at ../../module/zfs/zio.c:1399 /openzfs#490 zio_nowait (zio=0x7f0350002810) at ../../module/zfs/zio.c:1456 /openzfs#491 0x00007f03c8064593 in vdev_raidz_io_start (zio=0x7f0350001270) at ../../module/zfs/vdev_raidz.c:1591 /openzfs#492 0x00007f03c807f243 in __zio_execute (zio=0x7f0350001270) at ../../module/zfs/zio.c:1399 /openzfs#493 zio_nowait (zio=0x7f0350001270) at ../../module/zfs/zio.c:1456 /openzfs#494 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350001e60) at ../../module/zfs/vdev_mirror.c:374 /openzfs#495 0x00007f03c807f243 in __zio_execute (zio=0x7f0350001e60) at ../../module/zfs/zio.c:1399 /openzfs#496 zio_nowait (zio=0x7f0350001e60) at ../../module/zfs/zio.c:1456 /openzfs#497 0x00007f03c805ed43 in vdev_mirror_io_done (zio=0x7f033a0c39c0) at ../../module/zfs/vdev_mirror.c:499 /openzfs#498 0x00007f03c807a0c0 in zio_vdev_io_done (zio=0x7f033a0c39c0) at ../../module/zfs/zio.c:2707 /openzfs#499 0x00007f03c808285b in __zio_execute (zio=0x7f033a0c39c0) at ../../module/zfs/zio.c:1399 /openzfs#500 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03a8003c00, pio=0x7f033a0c39c0) at ../../module/zfs/zio.c:547 /openzfs#501 zio_done (zio=0x7f03a8003c00) at ../../module/zfs/zio.c:3278 /openzfs#502 0x00007f03c808285b in __zio_execute (zio=0x7f03a8003c00) at ../../module/zfs/zio.c:1399 /openzfs#503 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800c400, pio=0x7f03a8003c00) at ../../module/zfs/zio.c:547 /openzfs#504 zio_done (zio=0x7f038800c400) at ../../module/zfs/zio.c:3278 /openzfs#505 0x00007f03c808285b in __zio_execute (zio=0x7f038800c400) at ../../module/zfs/zio.c:1399 /openzfs#506 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800da00, pio=0x7f038800c400) at ../../module/zfs/zio.c:547 /openzfs#507 zio_done (zio=0x7f038800da00) at ../../module/zfs/zio.c:3278 /openzfs#508 0x00007f03c808285b in __zio_execute (zio=0x7f038800da00) at ../../module/zfs/zio.c:1399 /openzfs#509 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800fd80, pio=0x7f038800da00) at ../../module/zfs/zio.c:547 /openzfs#510 zio_done (zio=0x7f038800fd80) at ../../module/zfs/zio.c:3278 /openzfs#511 0x00007f03c807a6d3 in __zio_execute (zio=0x7f038800fd80) at ../../module/zfs/zio.c:1399 /openzfs#512 zio_execute (zio=zio@entry=0x7f038800fd80) at ../../module/zfs/zio.c:1337 /openzfs#513 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#514 0x00007f03c806119d in vdev_queue_io_done (zio=zio@entry=0x7f03a0010950) at ../../module/zfs/vdev_queue.c:775 /openzfs#515 0x00007f03c807a0e8 in zio_vdev_io_done (zio=0x7f03a0010950) at ../../module/zfs/zio.c:2686 /openzfs#516 0x00007f03c807a6d3 in __zio_execute (zio=0x7f03a0010950) at ../../module/zfs/zio.c:1399 /openzfs#517 zio_execute (zio=0x7f03a0010950) at ../../module/zfs/zio.c:1337 /openzfs#518 0x00007f03c7fcd0c4 in taskq_thread (arg=0x966d50) at ../../lib/libzpool/taskq.c:215 /openzfs#519 0x00007f03c7fc7937 in zk_thread_helper (arg=0x967e90) at ../../lib/libzpool/kernel.c:135 /openzfs#520 0x00007f03c78890a3 in start_thread (arg=0x7f03c2703700) at pthread_create.c:309 /openzfs#521 0x00007f03c75c50fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 This occurred when ztest was simulating a scrub under heavy I/O load. Under those circumstances, it was possible for a mix of noop I/Os for aggregation continuity and the I/O elevator to generate arbitrarily deep recursion. This patch modifies ZFS to propapage a recursion counter inside the zio_t objects such that IOs will be redispatched upon reaching a given recursion depth. We can detect long call chains and dispatch to another ZIO taskq. We cut in-line when we do this to minimize the potential for taskq exhaustion that can prevent a zio from notifying its parent. Signed-off-by: Richard Yao <[email protected]>

behlendorf closed this as completed Sep 6, 2012

ryao mentioned this issue Oct 9, 2014

vdev_raidz_io_start: Ignore empty writes #2778

Closed

ryao mentioned this issue Oct 10, 2014

Redispatch ZIOs in deep call graphs #2786

Closed

pcd1193182 pushed a commit to pcd1193182/zfs that referenced this issue Jun 27, 2022

Use "-d" when ZETTACACHE_DEVICES is empty (openzfs#465)

d768a75

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

page allocation failure #465

page allocation failure #465

jeremysanders commented Nov 26, 2011

jeremysanders commented Nov 28, 2011

behlendorf commented Nov 28, 2011

jeremysanders commented Nov 29, 2011

behlendorf commented Nov 29, 2011

jeremysanders commented Dec 2, 2011

behlendorf commented Dec 2, 2011

ryao commented Apr 19, 2012

jeremysanders commented Apr 19, 2012

behlendorf commented Apr 19, 2012

cwedgwood commented Sep 3, 2012

behlendorf commented Sep 4, 2012

cwedgwood commented Sep 4, 2012

behlendorf commented Sep 5, 2012

cwedgwood commented Sep 5, 2012

behlendorf commented Sep 5, 2012

behlendorf commented Sep 6, 2012

page allocation failure #465

page allocation failure #465

Comments

jeremysanders commented Nov 26, 2011

jeremysanders commented Nov 28, 2011

behlendorf commented Nov 28, 2011

jeremysanders commented Nov 29, 2011

behlendorf commented Nov 29, 2011

jeremysanders commented Dec 2, 2011

behlendorf commented Dec 2, 2011

ryao commented Apr 19, 2012

jeremysanders commented Apr 19, 2012

behlendorf commented Apr 19, 2012

cwedgwood commented Sep 3, 2012

behlendorf commented Sep 4, 2012

cwedgwood commented Sep 4, 2012

behlendorf commented Sep 5, 2012

cwedgwood commented Sep 5, 2012

behlendorf commented Sep 5, 2012

behlendorf commented Sep 6, 2012