Multiple messages INFO: task txg_sync:... blocked for more than 120 seconds.

[arieleizenberg]

System information

Ubuntu 22.04 running 6.2.0-1018-aws #18~22.04.1-Ubuntu.
zfs-2.1.5-1ubuntu6~22.04.2
zfs-kmod-2.1.9-2ubuntu1.1
on AWS u-12tb1.112xlarge x86_64 instance.

Describe the problem you're observing

Our dmesg log contains multiple instances of:

[552261.064408] INFO: task txg_sync:8100 blocked for more than 120 seconds.
[552261.067253]       Tainted: P           O       6.2.0-1018-aws #18~22.04.1-Ubuntu
[552261.070383] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[552261.073647] task:txg_sync        state:D stack:0     pid:8100  ppid:2      flags:0x00004000
[552261.073654] Call Trace:
[552261.073657]  <TASK>
[552261.073663]  __schedule+0x2b4/0x5e0
[552261.073678]  schedule+0x5d/0x100
[552261.073683]  taskq_wait+0x70/0xa0 [spl]
[552261.073719]  ? __pfx_autoremove_wake_function+0x10/0x10
[552261.073736]  dmu_objset_sync+0x2d7/0x490 [zfs]
[552261.074587]  dsl_dataset_sync+0x67/0x300 [zfs]
[552261.074773]  dsl_pool_sync+0x9b/0x410 [zfs]
[552261.074918]  ? __mod_timer+0x293/0x440
[552261.074926]  ? __kmalloc_node+0x54/0x140
[552261.074937]  spa_sync_iterate_to_convergence+0xe0/0x200 [zfs]
[552261.075098]  spa_sync+0x2dc/0x5c0 [zfs]
[552261.075258]  txg_sync_thread+0x225/0x290 [zfs]
[552261.075443]  ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
[552261.075656]  ? __pfx_thread_generic_wrapper+0x10/0x10 [spl]
[552261.075673]  thread_generic_wrapper+0x61/0x80 [spl]
[552261.075688]  kthread+0xca/0xf0
[552261.075694]  ? __pfx_kthread+0x10/0x10
[552261.075696]  ret_from_fork+0x29/0x50
[552261.075702]  </TASK>
[552381.896348] INFO: task txg_sync:8100 blocked for more than 241 seconds.
[552381.899208]       Tainted: P           O       6.2.0-1018-aws #18~22.04.1-Ubuntu
[552381.902352] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[552381.905673] task:txg_sync        state:D stack:0     pid:8100  ppid:2      flags:0x00004000
[552381.905680] Call Trace:
[552381.905683]  <TASK>
[552381.905689]  __schedule+0x2b4/0x5e0
[552381.905706]  schedule+0x5d/0x100
[552381.905712]  taskq_wait+0x70/0xa0 [spl]
[552381.905750]  ? __pfx_autoremove_wake_function+0x10/0x10
[552381.905767]  dmu_objset_sync+0x2d7/0x490 [zfs]
[552381.906321]  dsl_dataset_sync+0x67/0x300 [zfs]
[552381.906493]  dsl_pool_sync+0x9b/0x410 [zfs]
[552381.906693]  ? __mod_timer+0x293/0x440
[552381.906701]  ? __kmalloc_node+0x54/0x140
[552381.906710]  spa_sync_iterate_to_convergence+0xe0/0x200 [zfs]
[552381.906868]  spa_sync+0x2dc/0x5c0 [zfs]
[552381.907024]  txg_sync_thread+0x225/0x290 [zfs]
[552381.907185]  ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
[552381.907444]  ? __pfx_thread_generic_wrapper+0x10/0x10 [spl]
[552381.907462]  thread_generic_wrapper+0x61/0x80 [spl]
[552381.907478]  kthread+0xca/0xf0
[552381.907484]  ? __pfx_kthread+0x10/0x10
[552381.907486]  ret_from_fork+0x29/0x50
[552381.907495]  </TASK>

[rincebrain]

Commonly, that message comes after something broke earlier with a more angry message like VERIFY or PANIC or BUG, could you check and see if something like that is there?

[arieleizenberg]

There is none, here is the full message:
dmesg1.txt