Unable to promote spare to replace failed drive; spare busy.

[dpickford]

I have a raidz2 volume with log/cache and 2 spares running on 3.2.0-4-amd64 kernel on Debian 7.

I lost a drive (these 3TB seagates die like flies) and after it rebuilt onto the spare, I'm unable to replace the offline drive, and get a drive busy error.

zpool replace storage ata-ST3000DM001-9YN166_S1F0F33C ata-ST3000VN000-1HJ166_W73036ML
cannot replace ata-ST3000DM001-9YN166_S1F0F33C with ata-ST3000VN000-1HJ166_W73036ML: ata-ST3000VN000-1HJ166_W73036ML is busy

I've actually replaced half a dozen of these drives using the same method over the last couple of years (not kidding about these seagates) and have not had this issue before.

I've tried -f, and manually offlinining and removing the drives, and even went ahead and rebooted to see if that would help with no avail.

Any suggestions would be appreciated.

Here are the package versions:

ii zfs-dkms 0.6.3-1wheezy all Native ZFS filesystem kernelmodules for Linux
ii zfsonlinux 4 all archive.zfsonlinux.org trust package
ii zfsutils 0.6.3-1wheezy amd64 command-line tools to manage ZFS filesystems

Here is the output of zpool status.

pool: storage
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: resilvered 2.22T in 35h20m with 0 errors on Thu Mar 5 03:50:06 2015
config:

NAME                                                  STATE     READ WRITE CKSUM
storage                                               DEGRADED     0     0     0
  raidz2-0                                            DEGRADED     0     0     0
    ata-ST3000DM001-1CH166_W1F2VS3E                   ONLINE       0     0     0
    ata-ST3000DM001-1CH166_W1F2WPQE                   ONLINE       0     0     0
    ata-ST3000DM001-1CH166_W1F5LBTD                   ONLINE       0     0     0
    ata-ST3000DM001-1CH166_Z1F4B0EN                   ONLINE       0     0     0
    ata-ST3000DM001-1ER166_Z5005PG2                   ONLINE       0     0     0
    spare-5                                           DEGRADED     0     0     0
      ata-ST3000DM001-9YN166_S1F0F33C                 OFFLINE      0     0     0
      ata-ST3000VN000-1HJ166_W73036ML                 ONLINE       0     0     0
    ata-ST3000VN000-1HJ166_W7300NAW                   ONLINE       0     0     0
    ata-ST3000VN000-1H4167_Z301S4AY                   ONLINE       0     0     0
logs
  mirror-1                                            ONLINE       0     0     0
    ata-INTEL_SSDSA2CW120G3_CVPR111004B0120LGN-part3  ONLINE       0     0     0
    ata-INTEL_SSDSA2CW120G3_CVPR111201W2120LGN-part3  ONLINE       0     0     0
cache
  ata-INTEL_SSDSA2CW120G3_CVPR111004B0120LGN-part4    ONLINE       0     0     0
  ata-INTEL_SSDSA2CW120G3_CVPR111201W2120LGN-part4    ONLINE       0     0     0
spares
  ata-ST3000VN000-1HJ166_W73036ML                     INUSE     currently in use
  ata-ST3000VN000-1HJ166_W7301YYX                     AVAIL   

[dpickford]

It's been a couple of days since I posted this. I'm happy to move this to a mailing list etc. if that is a better forum. I'm happy to run any additional tests/diagnostics if needed.

Thanks.

[behlendorf]

@dpickford if you haven't already you may want to move this to the mailing list for a faster reply. You may have found a bug here so I don't mind leaving this open, but it will take us a little while before we can look in to it.

[dpickford]

Okay, can do. I'll keep this updated with any learnings from the mailing list. Thx.

[behlendorf]

Closing. Issue was moved to the mailing list long ago.