dbuf_evict / txg_sync hang #6880
Milestone
Comments
|
@cwedgwood unfortunately it's not much fun. If you have If this is a new issue, do you have a feeling for roughly when it was introduced? |
|
new to me ... i reluctantly upgraded recently from what i found to be quite stable |
|
@cwedgwood definitely let us know if you hit this again. |
|
@behlendorf i hit this again, both times with 'heavy' zvol access what's unclear to me is why other people are not seeing this as i'm sure the use case of running VMs over zvols is very common kernel 4.14.x zfs from master ba88ef6bbdcfbf4cb6edfdfceb2d7e5a5e36c72b |
@cwedgwood you may not be the only one. It looks like this is related to 6952, and you're seeing the stacks since According to the profiling in #6952 there's a significant amount of lock contention evicting from the dbuf cache when it's filled with many small 8k blocks from zvols. And your backtraces suggest that While we run this down you could make the following safel change which should avoid the problem. Normally, when the last process to have a zvol open closes it all of the cached buffers backing it are immediately dropped. That's taking a long time of your system and isn't strictly needed, the buffers will be evicted from the ARC as needed anyway. diff --git a/module/zfs/zvol.c b/module/zfs/zvol.c
index 6ea8224..9949e92 100644
--- a/module/zfs/zvol.c
+++ b/module/zfs/zvol.c
@@ -1211,7 +1211,6 @@ zvol_shutdown_zv(zvol_state_t *zv)
if (dsl_dataset_is_dirty(dmu_objset_ds(zv->zv_objset)) &&
!(zv->zv_flags & ZVOL_RDONLY))
txg_wait_synced(dmu_objset_pool(zv->zv_objset), 0);
- (void) dmu_objset_evict_dbufs(zv->zv_objset);
}
/* |
|
@behlendorf i will make the suggested change later today that said, the zvols here are not opened/closed in my case but open (for days at a time) the last report i made of this - things were hung for many hours before i caught it, attempting to nuke processes/etc does not help and it required a hard (ipmi) reboot usually |
|
@behlendorf is there any reason when making this change i should or should not update to current |
|
@cwedgwood then I misunderstood, I didn't realize the entire system was deadlocked on this. So removing the |
|
@behlendorf reading over this now i see your comment wrt to dmu_objset_evict_dbufs likely came out of a stack trace where i killed qemu ... but that was after the hang ... i don't have a clean understanding of why things hang in the first place, at one point i thought it was because of a bad disk in l2arc but i'm running without that these days so i can't place blame there is there some way i can build the kernel to know who owns a lock should this happen again? (without completely crushing the performance) also, is there a tag or something in github to remind me to close this if it doesn't show up in 90 days or so? |
|
@cwedgwood it's only been 50 days, but if you haven't seen this again do you mind is we close this one out? |
|
@behlendorf the bug is still there, just now i got the trace below it's definitely related to qemu/kvm closing/opening zvol's and i'm going to retest with your suggested change above as i think that helped last i had it i think i can probably reproduce this fairly (happens within a day usually) if you have some suggestions on what else to try |
|
@cwedgwood in the past, we've been able to debug the zvol evict ARC buffers on close() by simply doing dd to a zvol. When dd calls close(), the zvol_shutdown_zv() function described above kicks in. In VMs, this would occur if a VM that uses a zvol is stopped. Hopefully you'll be able to reproduce without having to wait a few days before stopping a VM. |
behlendorf
added
the
Type: Performance
13 tasks
|
@cwedgwood @richardelling PR #7273 may also help with this issue. This is a port of OpenZFS openzfs/openzfs#564 which increases the default dbuf cache size. This can improve performance by keeping more indirect blocks in the dbuf cache which is particularly important when the block size is small. You can see if this helps without patching anything by increasing the |
|
@behlendorf i saw the upstream issue, it's unclear to me how a performance fix will prevent what's a hard lock/failure in this case |
|
Today we can already tune the dbuf cache size in sysfs, so it is easy to test. However, I don't think the size is where the bug lies, though increasing the size might make it easier to trigger the bug when compressed ARC is in play. |
|
@cwedgwood yes of course, I was thinking this was solely a performance issue again. Reading back through the issue again it's pretty clear that in fact things are blocking on the |
behlendorf
removed
the
Type: Performance
|
Have received a similar lock on a machine that receives very frequent compressed recv for 4 datasets (not zvols) using znapzend. Using 0.7.8 release. Full syslog is lengthy but starts the same way. Received a similar lock on 0.7.7 a few weeks back, but on z_zvol. |
|
Seeing something similiar with 0.7.5-1ubuntu13 - happens after a few days of heavy buildbot activity on the machine. |
|
I have yet to have the behavior come back after upgrading from 0.7.8 to 0.7.9. Hope it stays that way. EDIT: Nevermind, it's back again, same trace as above. |
13 tasks
|
we may have had a |
|
The issue is still present in 0.7.8. However, the auto-rebooter got to the machine before I could collect meaningful debug info. |
|
looks like we may need to backport #7602 to 0.7+ |
|
Yes it looks that way, I've added #7602 to the list for 0.7.13. |
|
@behlendorf you're confident this is the last fix needed here? if that is the case i will look at switch my tree over and test |
|
@cwedgwood yes, as far as we're aware. |
|
@behlendorf this looks to be resolved using a more recent version from git |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
linux 4.9.61
zfs d4a72f2 (origin/master as of a few hours ago)
@behlendorf if this happens again (i'm going to reboot i guess) how ideally do i find out who is holding the mutex?
The text was updated successfully, but these errors were encountered: