zpool remove - order of disks problem

[kolczykm]

Hello I have found strange behavior of zpool remove. Sometimes it works correctly - removes spare disks and some times not. I trying to remove 3 spare disks from pool and it working correctly but when I change order of disks it returning error that one of disks isn't present in pool. Let me explain how it looks step by step. I have following example zpool:

zpool status
  pool: Zpool1
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    Zpool1      ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sdb     ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
      mirror-1  ONLINE       0     0     0
        sde     ONLINE       0     0     0
        sdf     ONLINE       0     0     0
        sdg     ONLINE       0     0     0
      mirror-2  ONLINE       0     0     0
        sdh     ONLINE       0     0     0
        sdi     ONLINE       0     0     0
        sdj     ONLINE       0     0     0
    logs
      mirror-3  ONLINE       0     0     0
        sdk     ONLINE       0     0     0
        sdl     ONLINE       0     0     0
        sdm     ONLINE       0     0     0
    spares
      sds       AVAIL   
      sdq       AVAIL   
      sdo       AVAIL   
      sdn       AVAIL   
      sdp       AVAIL   

errors: No known data errors

When I try to remove disks sdo, sdn, sdp and to do that I call command:

zpool remove Zpool1 sdo sdn sdp

It works perfectly correct, those spare disks are removed. But when I have those disks back in pool and try to remove them again but I use different order of disks than in previous command it fails. Command fails when I try to remove spare disks using following command:

 zpool remove Zpool1 sdn sdp sdo
cannot remove sdo: no such device in pool

Command returns 1 and disk sdo isn't removed from pool it's still there, disks sdn and sdp are removed. When I run after previous command:

zpool remove Zpool1 sdo

This one correctly removes sdo from pool.

Remove command has also problems when I call it following way:

zpool remove Zpool1 sdp sdn sdo
cannot remove sdn: no such device in pool
cannot remove sdo: no such device in pool

In that case it can't remove disks sdn and sdo.

It looks like when disk order is different than that in zpool status then remove command fails to remove disks that are listed earlier in status if disks that are listed later are removed before those that are listed listed earlier.

[behlendorf]

@kolczykm Thanks for the detailed bug report and test case! It sounds like you've found a subtle bug, we'll dig in to it time permitting.

[loli10K]

This seems to fixed by b145e23, closing.