A reproducible testcase for ZFS hanging on IO error. #7469
Comments
|
Would it be possible to try this on 0.7.8? 0.6.5.11 is rather dated and a multitude of updates have occurred since it was released. |
|
I've also tested this on upcoming Ubuntu 18.04 LTS, which has 0.7.5 -- https://packages.ubuntu.com/bionic/zfs-dkms -- and it shows the same behavior. |
|
I've updated the issue description with more information. |
|
Thanks for pointing that out. To try to get around that, I've now set up a block device, which gets mapped via virtio into VM. I used that as the underlying device for device mapper to set up error mapping instead of loop. The result is the same, so I'm not sure if ZFS also has a scheduling issue with device mapper. These show up in kernel log after a while: |
|
The original author's test is mainly showing the ZFS will almost never return an EIO to the userland process when faced with a pool having no redundancy and underlying vdevs which repeatedly, themselves return EIOs to the zio layer. I ran the author's test manually and what happens is that when the bad section of the vdev is hit, self-healing comes into play and everything works just fine. Eventually, the pool fills to the point where that's no longer possible and the pool will likely become DEGRADED. If there's a bug here, it's that the EIOs should eventually be propagated back to the userland process. Finally, I'll note that this all has nothing to do with I suppose this would be similar to the conditions under which some file systems (ext4) automatically engage a read-only mode. |
|
This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions. |
and others
System information
Describe the problem you're observing
This is a pure software test case for reliably giving ZFS block device error by using "error" mapping target from device mapper. This is done to test how a file system reacts to device error when issuing fsync, as there are differences between different file systems.
The linked script creates a 200MB file to be used as a loop device, sets up a loop device, then sets up device mapper to throw error on access to a range of 5KB somewhere in the block device. Zpool is created on the resulting device mapper block device, thus giving a file system which will reliably hit an error if you write enough data to it.
A tool then issues writes and fsyncs until the file system is full or hits another error.
The issue that this tool is testing is that on some file systems fsync returns success after previous fsync error even though the previous data was not persisted successfully, effectively losing data. More information about fsync losing data is available on Stack Overflow -- http://stackoverflow.com/q/42434872/398670 and PostgreSQL mailing list -- https://www.postgresql.org/message-id/CAMsr+YHh+5Oq4xziwwoEfhoTZgr07vdGG+hu=1adXx59aTeaoQ@mail.gmail.com
ZFS does not lose data, but hangs the reproducing tools.
On ZFS the tool hangs on fsync call, which I assume is hanging on error from underlying device mapper block device, getting error either on write or on read. After hitting that, z_wr_iss kernel process will be using most of the CPU, load increases significantly and the system becomes somewhat unresponsive.
I was running this in a VM, with freshly installed Ubuntu (server release, both 17.10.1 and 18.04), so this is not a hardware issue. Rather it is a simulation of hardware issue. The question is what is expected of ZFS to do when it hits a bad (unreadable and unwritable) sector, but making system unresponsive is probably not desired.
Describe how to reproduce the problem
The tool for writing and fsyncing and a script around it to automate setting up loopback, device mapper with error target and zpool is available here:
https://github.com/zejn/scrapcode/tree/master/testcases/fsync-error-clear
To run it you need to have docker installed then run
FSTYPE=zfs makein the fsync-error-clear folder.Or, if you don't want docker, copy fsync_test_setup.sh into standalone and run
./fsync_test_standalone.sh zfs "" "" "keepopen". You will probably want to do that in a VM since the machine will become mostly unusable.Include any warning/errors/backtraces from the system logs
There are txg_sync hangs in the logs.
The text was updated successfully, but these errors were encountered: