0.6.5.1 - I/O timeout during disk spin up

[speedst3r]

I have an 8-disk raidz2 vdev where the component disks are put in standby after being idle for 15 minutes. With 0.6.5.1 it appears that something has changed that causes I/O requests to timeout while waiting for the drives to spin up, with the following in the syslog:

Sep 30 11:45:41 sook kernel: [ 1604.793964] sd 0:0:4:0: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Sep 30 11:45:41 sook kernel: [ 1604.793969] sd 0:0:4:0: [sdf] CDB:
Sep 30 11:45:41 sook kernel: [ 1604.793972] Read(16): 88 00 00 00 00 00 9c 0e 65 c8 00 00 00 08 00 00
Sep 30 11:45:41 sook kernel: [ 1604.793984] blk_update_request: I/O error, dev sdf, sector 2618189256

For each recurrence the component device differs as does the sector (i.e. it's not a faulty drive), and sometimes errors for two different devices are logged in the same request. Once the error occurs the system is inaccessible (ping replies, but other services open a connection with zero data) and needs to be reset. Individual drives can be woken up with dd but when attempting to access the filesystems on the pool the I/O error occurs.

Downgraded to 0.6.4.2 and this issues does not occur.

Steps to reproduce:

1. Put disks in vdev in standby (e.g. hdparm -y /dev/sd[b-i])
2. Attempt to access a filesystem in that pool
3. Observe errors

System is running Ubuntu 14.04, kernel 3.13.0-61 (tried -62, -63, -65 and the vivid enablement kernel with no difference).

[mountassir]

I think this is similar to #3785, a fix will be in the 0.6.5.2 release.

[kernelOfTruth]

yes, which should be today or tomorrow according to @behlendorf

@speedst3r

please try #3785 (comment)

If you're able to reproduce this could you try setting the module option spl_taskq_thread_dynamic=0 at boot time and seeing if this resolves the issue.

[behlendorf]

Yes, this should be addressed in 0.6.5.2 which is available now. It'll take a little while for it to work its way in to the distributions but you can always download it directly. Definitely let us know if it does or doesn't resolve the issue.

https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.6.5.2

[speedst3r]

Thanks all, I only checked the open issues - not closed!

[drozdu]

I am still experiencing this error in ZFS version 0.6.5.3, linux arch, kernel 4.2.5.
Using IBM ServeRAID M1015 in IT mode (SAS2008).

gru 25 01:27:49 nas kernel: sd 6:0:0:0: [sdf] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x00
gru 25 01:27:49 nas kernel: sd 6:0:0:0: [sdf] CDB: opcode=0x2a 2a 00 65 23 23 28 00 00 08 00
gru 25 01:27:49 nas kernel: blk_update_request: I/O error, dev sdf, sector 1696801576
gru 25 01:27:49 nas kernel: sd 6:0:2:0: [sdh] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x00
gru 25 01:27:49 nas kernel: sd 6:0:2:0: [sdh] CDB: opcode=0x2a 2a 00 65 23 23 28 00 00 08 00
gru 25 01:27:49 nas kernel: blk_update_request: I/O error, dev sdh, sector 1696801576
gru 25 01:27:49 nas kernel: sd 6:0:3:0: [sdi] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x00
gru 25 01:27:49 nas kernel: sd 6:0:3:0: [sdi] CDB: opcode=0x2a 2a 00 65 23 23 30 00 00 08 00
gru 25 01:27:49 nas kernel: blk_update_request: I/O error, dev sdi, sector 1696801584
gru 25 01:28:02 nas kernel: sd 6:0:1:0: [sdg] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x00
gru 25 01:28:02 nas kernel: sd 6:0:1:0: [sdg] CDB: opcode=0x2a 2a 00 88 1a 2c b8 00 00 08 00
gru 25 01:28:02 nas kernel: blk_update_request: I/O error, dev sdg, sector 2283416760
gru 25 01:28:02 nas zed[21198]: eid=426 class=delay pool=dysk1
gru 25 01:28:02 nas zed[21200]: eid=427 class=io pool=dysk1
gru 25 01:28:02 nas zed[21223]: eid=428 class=delay pool=dysk1
gru 25 01:28:02 nas zed[21225]: eid=429 class=io pool=dysk1
gru 25 01:28:02 nas zed[21248]: eid=430 class=delay pool=dysk1
gru 25 01:28:02 nas zed[21250]: eid=431 class=io pool=dysk1
gru 25 01:28:02 nas zed[21273]: eid=432 class=io pool=dysk1
gru 25 01:28:02 nas zed[21296]: eid=433 class=delay pool=dysk1
gru 25 01:28:02 nas zed[21298]: eid=434 class=io pool=dysk1