zfs still appears to respond incorrectly to failed writes

[rincebrain]

On 2.6.32-24-generic, Ubuntu AMD64, with latest Git as of right now (commit b5c254b...d5fd), I've experienced a fascinating behavior which occurred in slightly different conditions before.

I've got 7 disks in 7 zpools, named disk1 through disk7.

I have 7 zvols, unoriginally named d1 through d7, composed of the entirety of space available on disk[1-7], respectively.

I've assembled these 7 zvols into an MD RAID-6, made a filesystem on it, and began writing data.

I pulled disk 2 physically. Unfortunately, it appears the controller these disks are plugged into is poor at detecting such things, and only reports failed writes, not that the device is really gone.

zfs, in turn, appears to block forever on expecting writes to succeed to the device eventually, and thus doesn't mark the disk as faulted, thus not marking the pool as ..., thus ... zvol as ..., thus not informing md, thus causing all of the above to block...forever.

Reinserting the disk doesn't do the correct thing, presumably because the controller doesn't notice. It's quite a dumb controller, and I agree it's not entirely representative of what a real, modern controller will do (it's an Adaptec AHA-2940W - isn't old kit wonderful{, ly inadequate}), but I was using it to test prior to attempting such a configuration with modern drives on a modern SATA controller. That said, I don't think the correct behavior is for ZFS to block forever in kernelspace and never give up, and thereby cause anything based on the zvol to cry horrible tears of blood as it, too, blocks forever.

Shortly to be attached is the complete dmesg of the system in question. (the ata hard resets are me using a horrible bash script to force a manual reprobe of all SCSI busses on the machine, which caused no change).

[rincebrain]

Oh, right, I never figured out if this had any form of attachment.

http://rincebrain.org/lol_zfs.log

[behlendorf]

Thanks for the bug report and testing. I'm hopeful that in the next few months we'll be able to devote some time to improving the situation so this sort of thing is handled better. I suspect we may need to do some device driver work to get this exactly right so your mileage may vary if your using different hardware than us. But we'll do what we can.

As for adding attachments, the thing to do on GitHub I think is to create a public gist http://gist.github.com and then simply link it from the bug report.

[behlendorf]

Rincebrain, the latest git code should handle various failure modes better. I haven't tested the case you describe but there is now a zfault.sh regression suite which does validate several of the most common failure modes. Everything is working well for me, but there are still a few false positive in the test suite.

[behlendorf]

Rincebrain I'm closing this issue because I've recently done quite a bit of failure testing and it's working well for me. I didn't try your exact setup but if you continue to have issues with the latest release please open a new bug with the details.