Reduce timeout on disks

[Rudd-O]

For every disk device that ZFS manages:

On Fri, 2011-11-25 at 11:12 +0100, e-t172 wrote:
On 2011-11-25 11:02, Fajar A. Nugraha wrote:

For zfs to work properly:

• a drive needs to be report quickly that it failed
• zfs needs to know that the drive failed, and kick it out of the pool
  (possibly activating hot spare, if available)

The problem is some drives (your HDD might be among those) will retry
for a LOOOONG time reading/writing to a bad sector before if it gives
up. And linux's driver will also spend lots of time retrying.

Maybe reducing /dev/block/*/device/timeout would alleviate this issue?

It'd be appropriate in case the device is just being... uhh... quite literally retarded. Five seconds? Ten seconds? Should be enough in contrast to the minutes- or hours-long timeouts.

Donno if this will fix the "kernel retries reads" problem (which fundamentally we need to solve in the linux kernel), but this might help.

I'd advise to adjust the timeout even in the case of disk partitions being used rather than whole disks. Remember that most of us who use zfs-on-root have no choice but to use ZFS on disk partitions because of /boot.

[stevecs]

Just to add here in addition to the mail-list so as to have a single thread.

The timeout values in /sys/block/${SDDEVICE}/device/timeout are for command timeouts NOT drive timeouts.

Yes, this does help (I've been fighting this issue here as the ST2000DL003's do not have TLER/EER/EEC codes so will take up to MINUTES to recover for various types of errors. (one reason why I mentioned previously that these are not good drives to pick up, and instead the SV35's or others that support TLER would be needed even with software raids).

Anyway, setting the command timeouts to a more manageable limit (default is 30 seconds on current kernels) something like 7 seconds DOES help, in that the system will not hang on I/O for long periods, however will cause other issues (i.e. you'll have to re-issue the command or flag it as an error, etc depending on what was issuing that command).

With ZFS this will most likely show up as data corruption issues during a scrub operation. Which if you don't have multiples on the same vdev and in the same spots you can recover if you have the redundancy.

[Rudd-O]

Yeah I don't think we can do much about TLER.

What we want is this: instead of paralyzing the array for minutes at a time, whenever a drive does not respond in X seconds, kick it off the array -- and, during this single-disk-paralysis time, whatever other disks that can continue servicing the pending requests should continue doing so. X should be 10 because TLER is commonly 7 (which means that the drive is guaranteed to respond in 7 seconds).

The command timeout I think would help with that. I am not sure, though.

[Rudd-O]

Command timeouts will most likely NOT show as data corruption issues. To the extent that I know, drives freezing past their command timeouts do not make read or write operations return EIO. But, if I am wrong, then command timeouts might cause IO errors which ZFS will handle nicely. However, if that is the case, and ZFS detects EIO after 10 seconds waiting on an IO operation, then ZFS should conclude that the drive has frozen and kick it off the array, marking it as FAULTED (that's the point -- if the drive is freezing, it is faulted).

[Rudd-O]

Check ref: http://en.wikipedia.org/wiki/Time-Limited_Error_Recovery#Software_Raid

[stevecs]

They do, just tried:

Here's an example of a drive that is having problems on my system here. This is with a short device timeout (5 seconds), the device driver is showing an i/o error to that device, the timeout lets the system only 'hang' for 5 seconds which is much better than the default of 30.

[27642.969486] end_request: I/O error, dev sdp, sector 1798141632

However due to the error, you get the following when doing a scrub: (/dev/sdp is c02t0d15)

      raidz2-15   ONLINE       0     0     0
        c04t0d15  ONLINE       0     0     0
        c03t0d15  ONLINE       0     0     0
        c02t0d15  ONLINE       0     0     0  (repairing)
        c86t0d15  ONLINE       0     0     0
        c85t0d15  ONLINE       0     0     0
        c84t0d15  ONLINE       0     0     0

-----Original Message-----
From: Rudd-O [mailto:[email protected]]
Sent: Tuesday, November 29, 2011 01:28 PM
To: 'stevecs'
Subject: Re: [zfs] Reduce timeout on disks (#471)

Command timeouts will most likely NOT show as data corruption issues. To the extent that I know, drives freezing past their command timeouts do not make read or write operations return EIO. But, if I am wrong, then command timeouts might cause IO errors which ZFS will handle nicely. However, if that is the case, and ZFS detects EIO after 10 seconds waiting on an IO operation, then ZFS should conclude that the drive has frozen and kick it off the array.

---

Reply to this email directly or view it on GitHub:
#471 (comment)

[Stoatwblr]

Blue Sky thinking:

Timeout errors on individual drives are most likely to be bad sector related - and it'd be good for ZFS to detect which sectors (or blocks of sectors) are generating the timeouts and generate a few 0x00 writes to the sector.

This will trigger a drive to properly test pending bad sectors and remap or recover them as needed. Following the writes, a retest of the sectors can check to see if the timeouts are recurring. and if things are back to normal the bad area can be rebuilt using raidz or mirror data.

(NOTE: "Enterprise" drives spend far less time trying to recover data than standard ones because the design assumption is that they're in a RAID array - this means they MUST NOT be used in solo operation due to increased data loss risk.)

Going further, ZFS could actively test disks in an array using some variant of badblocks, trigger repairs etc Right now a drive has to be taken out of service to do tests, which is time consuming and (because there are humans in the loop) error prone no matter how careful any individual is.

[Rudd-O]

All ZFS has to do is:

If the correct data is available on other drives, fetch it after a quick timeout (1 second?) and deliver it to userspace immediately, marking the slow drive as "unresponsive".
If the correct data is not available in other drives, pause (just as it does today).

Then, after ~15 seconds of the drive being "unresponsive" (not delivering data for any read requests), ZFS must assume that the sector returned "bad data" (preferably not reporting it as "bad data" or "CKSUM", but as "unresponsive drive at sector XXXXX").

At this point, if possible, corrective action kicks in. IF the correct data is available (RAID1, RAIDZ):

• ZFS must first temporarily "isolate" the drive from the pool (so as not to destroy performance),
• then ZFS must (possibly) reset the drive (possibly because write requests won't go in until the drive has recovered),
• then ZFS must attempt to write the CORRECT data to said sector

All of the above actions should be performed in a separate "thread", parallel to the pool I/O.

If the attempt to do so succeeds, then ZFS can "de-isolate" the drive from the pool, and then resilver the drive using its usual algorithm, until the drive has caught up with the current state of the pool. Then the drive is "good" again (as we can mostly safely assume the sector reallocation algorithms kicked in).

But if enough of those attempts fail (two?), then the drive must be taken offline and the reason ("drive unresponsive / drive bad") should be noted in the event log. I think there is a WRITE column in zpool status.

You will note how this algorithm works with both enterprise and consumer drives, by allowing enterprise drives' fast "oh shit, I have a problem" response to come back soon enough, and repairing consumer drives faster, while minimizing the performance impact of the overall pool I/O in both cases.

This is a bug for upstream to handle, I think.

[stevecs]

Only thing to be careful of here is that I am seeing in my testing a lot of these timeouts/errors due to firmware incompatibilities with various sas expanders. So we have to be careful or we could end up dropping lots (hundreds) of drives.

Probably need some check so as to /not/ drop if it's the last drive for data integrity in a vdev, or provide an option to just increment the counters for a particular drive and not do the drop (let the admin figure it out as to what the problem is). Probably even make it so that not-dropping (but incrementing counters) would be the default (error on the cautious side).

For example here I have two backplanes that are having issues and in 24 hours I have thrown over 3,000 errors on each (yes, it's causing I/O to be dead slow) but still available.

[Stoatwblr]

PLEASE NOTE: a lot of consumer drives can have timeouts set, which kind of renders the problem moot.

smartctl -l scterc,70,70 {device}

the man page says:

scterc[,READTIME,WRITETIME] - [ATA only] [NEW EXPERIMENTAL SMARTCTL FEATURE] prints values and descriptions of the SCT Error Recovery Control set‐tings. These are equivalent to TLER (as used by Western Digital), CCTL (as used by Samsung and Hitachi) and ERC (as used by Seagate). READTIME and WRITETIME arguments (deciseconds) set the specified values. Values of 0 disable the feature, other values less than 65 are probably not supported. For RAID configurations, this is typically set to 70,70 deciseconds.

I've verified this works on Samsung spinpoint F5 devices.

It would still be good for ZFS to note the timeout and attempt to trigger drive sector remapping during a quiet period.

[Stoatwblr]

More bluesky:

If ZFS can attempt to set the timeouts on array drives, then it can make a note of which ones don't support scterc and in turn treat them to the longer-winded approach if there are timeouts.

[behlendorf]

A reasonable approach to handle this has been proposed in #2387.

[behlendorf]

Closing. The retry behavior in modern Linux kernels has improved and efforts to add improved logic to ZFS has been problematic. Users who require lower timeouts should set the standard /dev/block/*/device/timeout tunable.

[dioni21]

I reached this issue while searching for a solution to a problem.

I have a disk that is probably near to fail, but still responds. No kernel error is logged, neither while a surface scan, nor during a scrub. But scrub find some errors. Also, trying to read every file (rsync or zfs send) finds even more errors (100 times more than scrub). Now doing a ddrescue copy I noticed that it sometimes freezes for some seconds (around 10s?), but continues without any error.

This disk was mirrored to another one that has failed, and I had to take off. Now I want to save my data. If this is some kind of low ZFS timeout, how can I make this higher, just to make a full copy?

Currently using ZoL from git master, last compiled middle February, when most commits were for ZTS and FreeBSD source integration.