Set TASKQ_DYNAMIC for kmem and system taskqs

Add the TASKQ_DYNAMIC flag to the kmem_cache and system taskqs
to reduce the number of idle threads on the system.  Additional
threads will be created on demand up to the previous maximum
thread counts.  This should have minimal, if any, impact on
performance.

This makes the system taskq consistent with illumos which is
always created as a dynamic taskq with up to 64 threads.

The task limits for the kmem_cache have been increased to avoid
any unnessisary throttling and to keep a larger reserve of
task_t structures on the free list.

Signed-off-by: Brian Behlendorf <[email protected]>

module/spl/spl-kmem-cache.c

@@ -1725,7 +1725,9 @@ spl_kmem_cache_init(void)
 	init_rwsem(&spl_kmem_cache_sem);
 	INIT_LIST_HEAD(&spl_kmem_cache_list);
 	spl_kmem_cache_taskq = taskq_create("spl_kmem_cache",
-	    spl_kmem_cache_kmem_threads, maxclsyspri, 1, 32, TASKQ_PREPOPULATE);
+	    spl_kmem_cache_kmem_threads, maxclsyspri,
+	    spl_kmem_cache_kmem_threads * 8, INT_MAX,
+	    TASKQ_PREPOPULATE | TASKQ_DYNAMIC);
 	spl_register_shrinker(&spl_kmem_cache_shrinker);
 
 	return (0);

module/spl/spl-taskq.c

@@ -987,10 +987,10 @@ EXPORT_SYMBOL(taskq_destroy);
 int
 spl_taskq_init(void)
 {
-	/* Solaris creates a dynamic taskq of up to 64 threads, however in
-	 * a Linux environment 1 thread per-core is usually about right */
-	system_taskq = taskq_create("spl_system_taskq", num_online_cpus(),
-				    minclsyspri, 4, 512, TASKQ_PREPOPULATE);
+	int threads = MAX(num_online_cpus(), 64);
+
+	system_taskq = taskq_create("spl_system_taskq", threads,
+	    minclsyspri, threads, INT_MAX, TASKQ_PREPOPULATE | TASKQ_DYNAMIC);
 	if (system_taskq == NULL)
 		return (1);