Fix zpl_mount() deadlock

[behlendorf]

Description

Commit 93b43af inadvertently introduced the following scenario which can result in a deadlock. This issue was most easily reproduced by LXD containers using a ZFS storage backend but should be reproducible under any workload which is frequently mounting and unmounting.

-- THREAD A --
spa_sync()
  spa_sync_upgrades()
    rrw_enter(&dp->dp_config_rwlock, RW_WRITER, FTAG); <- Waiting on B

-- THREAD B --
mount_fs()
  zpl_mount()
    zpl_mount_impl()
      dmu_objset_hold()
        dmu_objset_hold_flags()
          dsl_pool_hold()
            dsl_pool_config_enter()
              rrw_enter(&dp->dp_config_rwlock, RW_READER, tag);
    sget()
      sget_userns()
        grab_super()
          down_write(&s->s_umount); <- Waiting on C

-- THREAD C --
cleanup_mnt()
  deactivate_super()
    down_write(&s->s_umount);
    deactivate_locked_super()
      zpl_kill_sb()
        kill_anon_super()
          generic_shutdown_super()
            sync_filesystem()
              zpl_sync_fs()
                zfs_sync()
                  zil_commit()
                    txg_wait_synced() <- Waiting on A

Motivation and Context

Resolve issue #7691. @ColinIanKing @sforshee can you please review this proposed fix.

How Has This Been Tested?

Locally built and verified the relevant test cases still pass. Unfortunately, I wasn't able to reproduce the issue original issue so I can't verify this does completely resolve it. However, based on the analysis above it should.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the ZFS on Linux code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commit messages are properly formatted and contain Signed-off-by.
• Change has been approved by a ZFS on Linux member.

[ColinIanKing]

I've tried this fix with lxd creating in parallel 64, then 96, then 128 containers and also deleting these without any lockups. Without the fix, I was able to trigger the lockups. Fix looks good to me. Thanks!

[alek-p]

LGTM, thanks for the detailed write-up on the 3 threads involved.

[vazir]

Any idea if this patch is going to be adopted to Ubuntu 18.04 anytime soon?

[ColinIanKing]

We have a 3 week release cadence on the kernel for patch integration, testing and release, so it should be landing in the next 3 week cycle.

[vazir]

I hit this bug in slightly different circumstances. Invoking "lxc ls" from inside the ZFS mounted container, lead for me to the same behavior everyone here described. So i than had to power-off the host to restore it. That may be in parallel with manual container space mount via "zfs mount" before starting the container and not unmounting.