Kernel panic while running bonnie++ test on ZFS with recordsize 1M #14732
Comments
|
That stack trace is you triggering an OOM from the ext4 code, not a kernel panic. ZFS's memory usage there doesn't seem absurd, from the output, but there seems to be a lot of RAM not covered by the output there if it's got 32 GIB in total. Could you share more of the output, ideally the first 200 or so lines after it starts to go off the rails? e: also In theory, I think the OOM messages should print somewhere around when they start how much they're trying to find, so that'd be a hint. |
|
No problem - here is file attached. It's a dmesg dump created by kdump-tools.
|
|
That's odd - the first thing it mentions is running out of RAM trying to fork(), essentially, unless I'm really bad at kernel code, which should be a very small allocation, comparatively... There's also almost no binaries running. That's very strange indeed. I'll try to reproduce this later when I'm home with my nice testbeds. |
|
As an addition to previous info I've compiled several ZFS versions from release source tarballs and tested it and it seems that this bug first "appeared" in version 2.1.6 - because on versions 2.1.2-2.1.5 test runs without any issues but on 2.1.6 (and on 2.1.7 and 2.1.8 as well) it crashes the same way as on 2.1.9. Maybe it can help to find the root cause... |
|
My wild guess would have been an interaction with #12746 but that was 2.1.5. Simple enough to bisect assuming it reproduces readily, I suppose. |
|
A bisection would be great. I personally would suspect #13794 as the direct origin of the symptoms, as it changes how ZFS frees up memory. If so, that may mean that there is some other underlying memory accounting/nonfreeing/fragmentation issue and that this just reveals it… |
|
Returning back here from very long session of testing various builds. Sorry for giving (as I've detected later) wrong info about where bug is probably first appeared - more deeper testing with bonnie++ n passes detected another picture. First I've built several release tarball versions and detected after tests that the latest "clean" version is 2.0.5, 2.0.6 is broken already. Second - I've got a sequence of commits between that versions (zfs-2.0.5...zfs-2.0.6) and found (through simple test driven) a commit which leads to crashes, here it is: 7a41ef2 |
|
Even 2.0.3 version crashing from time to time - so it's really unclear which change is root cause. In addition I see strange output while getting info from kernel crash dump - |
|
Is there a version after which this is much more easily triggered? For example, it is true it happens ~quickly on 2.1.6+, and much more rarely on 2.0.3-2.1.5? I strongly suspect multiple issues at play. That negative cache value is funny — I guess some tabulation race condition from being under memory pressure and constantly dropping caches and things. |
Yeah, on 2.1.9 this issue triggered quickly after 1-2 tests and in addition not only with bonnie++ - for example this fio test trigger it as well: cd /testpool && fio --name rw --rw rw --size 2G |
|
In addition I was able to reproduce this issue even on VMware virtual machine. I've prepared an OVF with all development tools required to compile a ZFS release and prepared ZFS pool with required settings and script to execute the test, kdump-tools is configured as well to produce a crash dump: https://drive.google.com/drive/folders/10lb9pjK23BHiezq08NveaRhJ0bddw5XK |
|
After further several tests I've found one more thing: this issue stops reproducing if I change logbias setting from "throughput" to "latency". Let's write here initial settings from issue description:
With this settings we have a kernel panic after testing, and after just apply one setting below tests are going fine even with 1M recordsize:
I've maid a quick search in source code for 2.1.9 version and found that only one method behavior depends on this setting actually and it's "zfs_log_write" - here it is: https://github.com/openzfs/zfs/blob/zfs-2.1.9/module/zfs/zfs_log.c#L532 P.S. Right now I've compiled ZFS packages from 2.1.9 tag with only one change in https://github.com/openzfs/zfs/blob/zfs-2.1.9/module/zfs/zfs_log.c#L586 - changed |
|
That change will cause an out of bounds memory writes on synchronous writes less than immediate_sz in size on line 595, so is not what you want. The fact that allocating a larger itx fixes the issue for wr_indirect records suggests an issue with that codepath writing beyond the record (when no additional length is allocated), but I don’t immediately see the bug that would cause that. |
I've got your point but in the case of writing beyond the record we're usually get an access violation error or something similar while in this case I get kernel panic due to out of memory - usually it happens when code allocates large amount of memory but not free it, correct me if I'm wrong pls. |
|
Well, one possibility is there aren’t any OOB writes and that your patch works by making the itx allocations larger, helping prevent memory fragmentation. If you are going to keep using your patch, I would change it to just unconditionally be len (rather than len or 0), to avoid an issue with small synchronous writes. it would be great for a maintainer to chime in here since you’ve found a great reproducer. |
System information
Describe the problem you're observing
System getting kernel panic everytime when running simple bonnie++ test on ZFS filesystem (not zvol) with recordsize set to 1M (512K affected as well). If setting recordsize to 256K or lower test is running fine. This test was performed on bare metal hardware (Intel i3 12 gen, 32GB RAM), but I was able to reproduce it on VMware virtual machine as well.
Describe how to reproduce the problem
apt-get update && apt-get -y dist-upgrade);/etc/modprobe.d/zfs.conf;zpool create -o ashift=12 testpool raidz2 /dev/disk/by-id/<disk1_name> /dev/disk/by-id/<disk8_name>;zfs set atime=off relatime=on xattr=sa dnodesize=auto logbias=throughput sync=standard dedup=off recordsize=1M compression=off testpool;useradd frank && cd /testpool && mkdir bon && chmod 777 bon;bonnie++ -d /testpool/bon/ -x 1 -u frank.Include any warning/errors/backtraces from the system logs
ZFS pool config (last characters obfuscated):
Disk info (all 8 HDD are idetical):
Dmesg trace (last 180 lines):
The text was updated successfully, but these errors were encountered: