Potential txg_sync deadlock from sleeping in arc.c while holding a hash_lock

[ryao]

The backtraces in #3835 make it look like something is holding the hash lock, which the txg_sync thread needs. The code for arc_read() is such that it is possible for us to sleep because we call kmem_zalloc(sizeof (arc_callback_t), KM_SLEEP) while holding the hash_lock. I do not know offhand if this would print to dmesg (as not all sleeps do), but this is not the only instance of a sleep while holding the hash_lock. arc_buf_clone() is also called under the hash_lock and it does a kmem_cache_alloc(buf_cache, KM_PUSHPAGE), which admittedly, is not quite as bad, but it illustrates that we have other potential places where we can sleep. Given that the hash_lock is just supposed to protect insertion, I think we could switch to finer grained locking.

I will be working on this more as I find time. I am opining this issue for opinions from others regarding this.

[gmelikov]

#3835 is resolved now, i think we can close this issue.