Wait for all znodes to be released before tearing down the superblock

By the time we're tearing down our superblock the VFS has started releasing
all our inodes/znodes. Some of this work may have been handed off to our
iput taskq so we need to wait for that work to complete. However the iput
from the taskq can itself result in additional work being added to the
taskq:

dsl_pool_iput_taskq
 iput
  iput_final
   evict
    destroy_inode
     zpl_inode_destroy
      zfs_inode_destroy
       zfs_iput_async(ZTOI(zp->z_xattr_parent))
        taskq_dispatch(dsl_pool_iput_taskq..., iput, ...)

Let's wait until all our znodes have been released.

Signed-off-by: Chris Dunlop <[email protected]>
Closes openzfs#3281

module/zfs/zfs_vfsops.c

@@ -1130,15 +1130,25 @@ EXPORT_SYMBOL(zfs_sb_prune);
 int
 zfs_sb_teardown(zfs_sb_t *zsb, boolean_t unmounting)
 {
-	znode_t	*zp;
-
 	/*
 	 * If someone has not already unmounted this file system,
 	 * drain the iput_taskq to ensure all active references to the
 	 * zfs_sb_t have been handled only then can it be safely destroyed.
 	 */
-	if (zsb->z_os)
+	if (zsb->z_os) {
 		taskq_wait(dsl_pool_iput_taskq(dmu_objset_pool(zsb->z_os)));
+		/*
+		 * Iputs run from the taskq may add the parents of dir-based
+		 * xattrs to the taskq: wait for these
+		 *
+		 * We can safely read z_nr_znodes without locking because the
+		 * VFS has already blocked operations which add to the
+		 * z_all_znodes list and increment z_nr_znodes.
+		 */
+		if (zsb->z_nr_znodes > 0)
+			taskq_wait(dsl_pool_iput_taskq(dmu_objset_pool(
+			    zsb->z_os)));
+	}
 
 	rrw_enter(&zsb->z_teardown_lock, RW_WRITER, FTAG);
 
@@ -1175,27 +1185,13 @@ zfs_sb_teardown(zfs_sb_t *zsb, boolean_t unmounting)
 		return (SET_ERROR(EIO));
 	}
 
-	/*
-	 * At this point there are no VFS ops active, and any new VFS ops
-	 * will fail with EIO since we have z_teardown_lock for writer (only
-	 * relevant for forced unmount).
-	 *
-	 * Release all holds on dbufs.
-	 */
-	mutex_enter(&zsb->z_znodes_lock);
-	for (zp = list_head(&zsb->z_all_znodes); zp != NULL;
-	    zp = list_next(&zsb->z_all_znodes, zp)) {
-		if (zp->z_sa_hdl)
-			zfs_znode_dmu_fini(zp);
-	}
-	mutex_exit(&zsb->z_znodes_lock);
-
 	/*
 	 * If we are unmounting, set the unmounted flag and let new VFS ops
 	 * unblock.  zfs_inactive will have the unmounted behavior, and all
 	 * other VFS ops will fail with EIO.
 	 */
 	if (unmounting) {
+		ASSERT(list_empty(&zsb->z_all_znodes));
 		zsb->z_unmounted = B_TRUE;
 		rrw_exit(&zsb->z_teardown_lock, FTAG);
 		rw_exit(&zsb->z_teardown_inactive_lock);