Unable to export rpool before first reboot

[reefland]

@rlaager, I have the following issue with the Ubuntu 20.04 Root on ZFS HOWTO:

I've been using the ZFS on Root for Ubuntu for several years since 18.04 LTS. I've even replicated much of it as ansible playbook. I completed step updates based on guide to work with 20.04, then 20.04.1, now I'm testing with 20.04.3 and having a problem.

Following: https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2020.04%20Root%20on%20ZFS.html#overview

I use a VirtualBox environment to practice / refine the steps before attempting on bare metal. I'm testing with three 20GB VDI files which provide for a 3 disk Legacy Boot, ZFS Native Encryption, Raidz1 setup, as per guide mdm is being used for encrypted swap.

VirtualBox boots the Ubuntu Desktop 20.04.3 ISO image and I follow the steps provided in the guide. I have started from scratch several times and keep having the same issue:

I get to Section 5, GRUB Installation, Step 10 ... I'm unable to export the pools. Everything previous to this goes pretty smooth.

My troubleshooting trying to determine why the pool busy:

I'm not in a path used by rpool:

# pwd
/home/ansible

Everything appears to unmounted cleanly:

# lsof /mnt
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/999/gvfs
      Output information may be incomplete.

# lsof | grep "/mnt"   
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/999/gvfs
      Output information may be incomplete.
    
# ls -l /mnt
total 0

# mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}'

No swap is used at this point:

# cat /proc/swaps
Filename                                Type            Size            Used            Priority

# free -mh
              total        used        free      shared  buff/cache   available
Mem:          7.8Gi       2.6Gi       3.2Gi       425Mi       2.0Gi       4.5Gi
Swap:            0B          0B          0B

The pool itself is in good shape:

# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool    53G  1.56G  51.4G        -         -     0%     2%  1.00x    ONLINE  /mnt

# zpool status -v
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME                                             STATE     READ WRITE CKSUM
        rpool                                            ONLINE       0     0     0
          raidz1-0                                       ONLINE       0     0     0
            ata-VBOX_HARDDISK_VB195fbc7a-08e09291-part4  ONLINE       0     0     0
            ata-VBOX_HARDDISK_VB8b7fa809-6ae551c5-part4  ONLINE       0     0     0
            ata-VBOX_HARDDISK_VB52366d20-43e72e2a-part4  ONLINE       0     0     0

errors: No known data errors

# zfs list
NAME                                               USED  AVAIL     REFER  MOUNTPOINT
rpool                                             1.04G  33.2G      234K  /mnt
rpool/ROOT                                        1.03G  33.2G      234K  none
rpool/ROOT/ubuntu_p029lm                          1.03G  33.2G      984M  /mnt
rpool/ROOT/ubuntu_p029lm/srv                       234K  33.2G      234K  /mnt/srv
rpool/ROOT/ubuntu_p029lm/usr                       629K  33.2G      234K  /mnt/usr
rpool/ROOT/ubuntu_p029lm/usr/local                 394K  33.2G      394K  /mnt/usr/local
rpool/ROOT/ubuntu_p029lm/var                      74.2M  33.2G      234K  /mnt/var
rpool/ROOT/ubuntu_p029lm/var/games                 234K  33.2G      234K  /mnt/var/games
rpool/ROOT/ubuntu_p029lm/var/lib                  72.2M  33.2G     1.50M  /mnt/var/lib
rpool/ROOT/ubuntu_p029lm/var/lib/AccountsService   234K  33.2G      234K  /mnt/var/lib/AccountsService
rpool/ROOT/ubuntu_p029lm/var/lib/NetworkManager    234K  33.2G      234K  /mnt/var/lib/NetworkManager
rpool/ROOT/ubuntu_p029lm/var/lib/apt              51.1M  33.2G     51.1M  /mnt/var/lib/apt
rpool/ROOT/ubuntu_p029lm/var/lib/dpkg             19.1M  33.2G     19.1M  /mnt/var/lib/dpkg
rpool/ROOT/ubuntu_p029lm/var/log                   480K  33.2G      480K  /mnt/var/log
rpool/ROOT/ubuntu_p029lm/var/mail                  234K  33.2G      234K  /mnt/var/mail
rpool/ROOT/ubuntu_p029lm/var/snap                  234K  33.2G      234K  /mnt/var/snap
rpool/ROOT/ubuntu_p029lm/var/spool                 330K  33.2G      330K  /mnt/var/spool
rpool/ROOT/ubuntu_p029lm/var/www                   234K  33.2G      234K  /mnt/var/www
rpool/USERDATA                                     480K  33.2G      234K  /mnt
rpool/USERDATA/root_p029lm                         245K  33.2G      245K  /mnt/root

Unable to export the pool:

# zpool export -a
cannot export 'rpool': pool is busy

# zpool export -f rpool
cannot export 'rpool': pool is busy

If I ignore this issue and attempt to reboot, the rpool will not be imported as its says it was used by another system. My understanding is that is normal if you are unable to export cleanly. Thus I'm trying to solve this.

[Z3TA]

I have the same problem. Can't export the pool because rpool is busy. Then can't import rpool because pool was previously in use from another system.

I think that "pool is busy" is a bug in openzfs itself because nothing was mounted.

Editing /etc/default/grub and adding zfsforce=yes to GRUB_CMDLINE_LINUX_DEFAULT makes it possible to import the rpool though.

[DrGeek]

Launch those commands before rebooting it will work 👍

mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' |xargs -i{} umount -lf {}
umount /mnt/*
umount -l /mnt
grep /mnt /proc/*/mounts | tac| awk '{print $3}' FS="/" | uniq | xargs -i{} kill -9 {}
zpool export -a
reboot -f

[reefland]

@DrGeek

Just took an environment and ran all the steps to try your suggestion. I still got the rpool is busy message, unable to export rpool and upon reboot failed to import rpool and get dropped at the (initramfs) prompt. It's not hard to recover from here, but just can not find what has that rpool busy.

[reefland]

@DrGeek - see if this sparks another idea.

I've implemented the ZFS 20.04 how-to as an Ansible role. The entire process works great minus this rpool busy issue. I've applied your suggestion in Ansible and got some additional output from it.

Note the umount: /mnt/boot: target is busy.

Relevant Ansible Task:

 - name: Unmount chroot file systems
    shell: |
      mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
      umount /mnt/*
      umount -l /mnt
      grep /mnt /proc*/mounts | tac | awk '{print $3}' FS="/" | uniq | xargs -i{} kill -9 {}

  - name: Export all ZFS Pools
    command: 
      "zpool export -a"

  tags:
    - unmount_chroot

The process fails on the zpool export 100% of the time as follows:

changed: [testlinux.rich-durso.us] => {
    "changed": true,
    "cmd": "mount | grep -v zfs | tac | awk '/\\/mnt/ {print $3}' | xargs -i{} umount -lf {}\numount /mnt/*\numount -l /mnt\ngrep /mnt /proc*/mounts | tac | awk '{print $3}' FS=\"/\" | uniq | xargs -i{} kill -9 {}\n",
    "delta": "0:00:00.256326",
    "end": "2022-03-23 17:54:37.031167",
    "rc": 0,
    "start": "2022-03-23 17:54:36.774841"
}

STDERR:

umount: /mnt/bin: not mounted.
umount: /mnt/boot: target is busy.
umount: /mnt/dev: not mounted.
umount: /mnt/etc: not mounted.
umount: /mnt/home: not mounted.
umount: /mnt/lib: not mounted.
umount: /mnt/lib32: not mounted.
umount: /mnt/lib64: not mounted.
umount: /mnt/libx32: not mounted.
umount: /mnt/media: not mounted.
umount: /mnt/mnt: not mounted.
umount: /mnt/opt: not mounted.
umount: /mnt/proc: not mounted.
umount: /mnt/run: not mounted.
umount: /mnt/sbin: not mounted.
umount: /mnt/sys: not mounted.
umount: /mnt/tmp: not mounted.
umount: /mnt/usr: not mounted.
umount: /mnt/var: not mounted.

TASK [zfs_on_root : Export all ZFS Pools] *****************************************************************************************************************************************************************************************************************************
task path: /home/rich/Sync/rich/ansible/roles/zfs_on_root/tasks/15-5_unmount_chroot.yml:16
fatal: [testlinux]: FAILED! => {
    "changed": true,
    "cmd": [
        "zpool",
        "export",
        "-a"
    ],
    "delta": "0:00:05.176945",
    "end": "2022-03-23 17:54:42.413881",
    "rc": 1,
    "start": "2022-03-23 17:54:37.236936"
}

STDERR:

cannot export 'rpool': pool is busy

MSG:

non-zero return code

If I re-run just this task, buy its tag still fails, much less output:

changed: [testlinux.rich-durso.us] => {
    "changed": true,
    "cmd": "mount | grep -v zfs | tac | awk '/\\/mnt/ {print $3}' | xargs -i{} umount -lf {}\numount /mnt/*\numount -l /mnt\ngrep /mnt /proc*/mounts | tac | awk '{print $3}' FS=\"/\" | uniq | xargs -i{} kill -9 {}\n",
    "delta": "0:00:00.005469",
    "end": "2022-03-23 18:06:59.411353",
    "rc": 0,
    "start": "2022-03-23 18:06:59.405884"
}

STDERR:

umount: /mnt/*: no mount point specified.
umount: /mnt: not mounted.

TASK [zfs_on_root : Export all ZFS Pools] *****************************************************************************************************************************************************************************************************************************
task path: /home/rich/Sync/rich/ansible/roles/zfs_on_root/tasks/15-5_unmount_chroot.yml:16
fatal: [testlinux: FAILED! => {
    "changed": true,
    "cmd": [
        "zpool",
        "export",
        "-a"
    ],
    "delta": "0:00:00.045487",
    "end": "2022-03-23 18:06:59.637535",
    "rc": 1,
    "start": "2022-03-23 18:06:59.592048"
}

STDERR:

cannot export 'rpool': pool is busy

MSG:

non-zero return code

[djacu]

@reefland I just ran into a similar issue with the NixOS Root on ZFS installation guide. In the last steps before exporting the zpools it says to run:

umount /mnt/boot/efis/*
umount /mnt/boot/efi

The first line runs fine but the second fails, but not because of the pool being busy but because nothing was mounted there. The problem is that all the pools were still mounted. I am running all this from a script and was not even getting the cannot export 'rpool': pool is busy messages and wasn't aware anything was wrong until I tried booting and the boot pool failed to be imported similar to what you experienced. I tried many things but the changing the previous umount commands to the following fixed it for me.

umount /mnt/boot/efis/*
umount -R /mnt

I was able to export my zpools and reboot into the installed OS.