allow control of I/O priority class of zpool scrub

[krichter722]

Starting a scrub with zpool scrub currently doesn't allow any control of the I/O class (see man ionice for details), because the (sub)command returns immediately and delegates to kernel(?)/internal control (at least something that doesn't allow the application of ionice).

My suggestion is to either provide a dummy process running in the foreground or being put in the background or not based on an option of zpool scrub which is responsive to ionice and delegates to the kernel threads(?)/internal control when passed as argument to the -p option of ionice. An alternative would be an option of zpool scrub imitating the behavior of ionice, e.g. zpool scrub --renice -c <class> -n <level> <poolname> or - the concept of ionice doesn't have to be copied - just zpool scrub --renice <0-9> <poolname>.

The default value should be low or at least between lowest and highest because some systems just freeze under the heavy load of a scrub. If anything, a setting for the default value can be added in a configuration file.

[rlaager]

ZFS has its own I/O scheduler. First off, make sure you're setting the Linux I/O scheduler to noop on your ZFS disks. ZFS will do this automatically if you are using "wholedisk".

Secondly, ZFS has some relevant tunables. See also the comments in #944.