cannot upgrade zfs from 0.7.10 to 0.7.11 while resilvering pool #7909

artee666 · 2018-09-15T07:28:01Z

System information

Distribution Name Archlinux
Distribution Version latest
Linux Kernel 4.14.69-1-lts
Architecture x64
ZFS Version 0.7.10-1
SPL Version 0.7.10-1

Describe the problem you're observing

I have pool with one disk (DISK1) and attached another disk (DISK2) to it. It started to resilver. Compiled zfs 0.7.11 and spl 0.7.11 and rebooted. The pool couldn't be imported because one device was missing. Tried to import it manually by issuing zpool import -f -d /dev/disk/by-id, but the output was that the DISK1 (with all the original data) had corrupted data. Going back to zfs 0.7.10 fixed this for me and now the pool continues to resilver.

artee666 · 2018-09-15T07:32:35Z

This is from journalctl for that boot:
Sep 15 08:35:49 rakef systemd[1]: Starting Import ZFS pools by cache file...
Sep 15 08:35:50 rakef kernel: WARNING: can't open objset 68, error 6
Sep 15 08:35:50 rakef kernel: WARNING: can't open objset 51, error 6
Sep 15 08:35:51 rakef zpool[443]: cannot import 'tank': one or more devices is currently unavailable
Sep 15 08:35:51 rakef systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 08:35:51 rakef systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
Sep 15 08:35:51 rakef systemd[1]: Failed to start Import ZFS pools by cache file.

artee666 · 2018-09-15T08:43:03Z

After resilvering and booting into kernel 4.18.7-arch1-1-ARCH with zfs 0.7.11:

zpool status -v
pool: tank
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: resilvered 743G in 3h7m with 0 errors on Sat Sep 15 10:10:55 2018
config:

NAME                        STATE     READ WRITE CKSUM
tank                        DEGRADED     0     0     0
  mirror-0                  DEGRADED     0     0     0
    11982558819847001828    FAULTED      0     0     0  was /dev/disk/by-id/wwn-0x50014ee2baaf6436-part1
    wwn-0x50014ee2baaf64b1  ONLINE       0     0     0

errors: No known data errors

zdb:
tank:
version: 5000
name: 'tank'
state: 0
txg: 5989
pool_guid: 15804811324617297113
errata: 0
hostid: 8323329
hostname: 'rakef'
com.delphix:has_per_vdev_zaps
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 15804811324617297113
create_txg: 4
children[0]:
type: 'mirror'
id: 0
guid: 17041492188469128553
whole_disk: 0
metaslab_array: 256
metaslab_shift: 35
ashift: 12
asize: 4000771735552
is_log: 0
create_txg: 4
com.delphix:vdev_zap_top: 130
children[0]:
type: 'disk'
id: 0
guid: 11982558819847001828
path: '/dev/disk/by-id/wwn-0x50014ee2baaf6436-part1'
devid: 'ata-WDC_WD40EFRX-68N32N0_WD-WCC7K1HKX86A-part1'
phys_path: 'pci-0000:00:1f.2-ata-1'
whole_disk: 1
not_present: 1
DTL: 4
create_txg: 4
com.delphix:vdev_zap_leaf: 129
children[1]:
type: 'disk'
id: 1
guid: 7626111177690474345
path: '/dev/disk/by-id/wwn-0x50014ee2baaf64b1-part1'
devid: 'ata-WDC_WD40EFRX-68N32N0_WD-WCC7K1HKXN01-part1'
phys_path: 'pci-0000:00:1f.5-ata-1'
whole_disk: 1
DTL: 194
create_txg: 4
com.delphix:vdev_zap_leaf: 315
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data

GregorKopka · 2018-09-15T08:44:23Z

As it looks the WARNING: can't open objset _x_, error _y_ can (at least the string "can't open objset" seems to be nowhere else in the code) only be emitted from
https://github.com/zfsonlinux/zfs/blob/492f64e941e3d6b947d1cc387a1a380c0c738b09/module/zfs/zil.c#L814
or
https://github.com/zfsonlinux/zfs/blob/492f64e941e3d6b947d1cc387a1a380c0c738b09/module/zfs/zil.c#L919
but I have no idea how this could be triggered - looking at the changeset zfs-0.7.10...zfs-0.7.11 I see no obvious reason.

So it's likely that this is just a another case of a stale /etc/zpool.cache getting in the way (again), have you tried to move it out of the way prior a manual import?

artee666 · 2018-09-15T08:52:57Z

I've moved /etc/zfs/zpool.cache to /etc/zfs/zpool.cache.old and the output of zpool import is:
pool: tank
id: 15804811324617297113
state: DEGRADED
status: One or more devices contains corrupted data.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
config:

tank                        DEGRADED
  mirror-0                  DEGRADED
    11982558819847001828    FAULTED  corrupted data
    wwn-0x50014ee2baaf64b1  ONLINE

Edit: Tried both with lts and latest kernel (4.14.69-1-lts, 4.18.7-arch1-1-ARCH)

artee666 · 2018-09-15T09:08:40Z

Tried bringing the device offline and online and seems like it is ok now:

zpool offline tank wwn-0x50014ee2baaf6436

zpool status

pool: tank
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: resilvered 743G in 3h7m with 0 errors on Sat Sep 15 10:10:55 2018
config:

NAME                        STATE     READ WRITE CKSUM
tank                        DEGRADED     0     0     0
  mirror-0                  DEGRADED     0     0     0
    11982558819847001828    OFFLINE      0     0     0  was /dev/disk/by-id/wwn-0x50014ee2baaf6436-part1
    wwn-0x50014ee2baaf64b1  ONLINE       0     0     0

zpool online tank wwn-0x50014ee2baaf6436

zpool status

pool: tank
state: ONLINE
scan: resilvered 34.6M in 0h0m with 0 errors on Sat Sep 15 11:07:03 2018
config:

NAME                        STATE     READ WRITE CKSUM
tank                        ONLINE       0     0     0
  mirror-0                  ONLINE       0     0     0
    wwn-0x50014ee2baaf6436  ONLINE       0     0     0
    wwn-0x50014ee2baaf64b1  ONLINE       0     0     0

errors: No known data errors

artee666 · 2018-09-15T09:18:35Z

And after reboot with lts kernel and zfs 0.7.11:

pool: tank
id: 15804811324617297113
state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: http://zfsonlinux.org/msg/ZFS-8000-72
config:

tank                        FAULTED  corrupted data
  mirror-0                  ONLINE
    wwn-0x50014ee2baaf6436  ONLINE
    wwn-0x50014ee2baaf64b1  ONLINE

On lts kernel and 0.7.10 the pool mounts fine (zpool status):
pool: tank
state: ONLINE
scan: resilvered 34.6M in 0h0m with 0 errors on Sat Sep 15 11:07:03 2018
config:

NAME                        STATE     READ WRITE CKSUM
tank                        ONLINE       0     0     0
  mirror-0                  ONLINE       0     0     0
    wwn-0x50014ee2baaf6436  ONLINE       0     0     0
    wwn-0x50014ee2baaf64b1  ONLINE       0     0     0

errors: No known data errors

GregorKopka · 2018-09-15T09:56:50Z

Make sure that the zpool.cache in your 0.7.11 initramfs is current (=from after the pool had been cleanly imported by a 0.7.11 zfs version).
As it's a root pool make sure that all device nodes for the pool are present before the zpool import happens, possibly your systemd decided to import the pool before udev was done creating all of them.

Or do as I did on my systems and hack zpool.cache out of the initramfs: either by giving the zpool import $bootpool a -d /dev or by keeping zpool.cache from being copied into the initramfs in the first place.

Both will force zpool import to do a real import instead of loading potential garbage from a stale file.

You might need to add -f to the zpool import in the initrd script to make it ignore the 'used in another system' (which stems from root pools never being cleanly exported on shutdown/reboot since / can't be unmounted).

artee666 · 2018-09-15T15:17:29Z

It is not root pool luckily... Just pool for /home. I've made sure to delete /etc/zfs/zpool.cache before creating initramfs with 0.7.11 (and I've removed copying of zpool.cache as well, just to be sure).

After reboot, the pool was not imported and I couldn't import it by hand due to corrupt metadata:

pool: tank
id: 15804811324617297113
state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: http://zfsonlinux.org/msg/ZFS-8000-72
config:

tank                        FAULTED  corrupted data
  mirror-0                  ONLINE
    wwn-0x50014ee2baaf6436  ONLINE
    wwn-0x50014ee2baaf64b1  ONLINE

On zfs 0.7.10 the pool works just fine (although had 2 files with permanent errors after scrubbing).

artee666 · 2018-09-15T15:39:23Z

The same is happening with zfs-linux-lts (version 0.7.11) package from https://github.com/archzfs/archzfs/wiki

[root@rakef ~]# zpool import -f -d /dev/disk/by-id/ tank
cannot import 'tank': I/O error
Destroy and re-create the pool from
a backup source.

artee666 · 2018-09-15T15:56:16Z

Couldn't this be affected by the fact that I'm using the whole device and created the pool under 0.7.10? And in module/zfs/vdev_disk.c there was code for 0.7.10:

if (wholedisk) {
...
uint64_t used = EFI_MIN_RESV_SIZE + NEW_START_BLOCK +
PARTITION_END_ALIGNMENT;
...
}

GregorKopka · 2018-09-15T18:34:21Z

@shartse (original commit) or @tonyhutter (having reverted it) might have a clue why 0.7.11 wouldn't import your 0.7.10 pool - I don't really have one. The 0.7.10 bug should only matter when creating the partitions, it was (as far as I got it) in the partitions created too big (beyond the end of the device).

Could you post sgdisk -p <affected disk> (or similar that lists the partition table and amount of drive sectors), just in case?

artee666 · 2018-09-15T23:55:10Z

Disk /dev/sdb: 7814037168 sectors, 3.6 TiB
Model: WDC WD40EFRX-68N
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 83D1687A-7D80-374D-AE7E-12979C0422A8
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 3693 sectors (1.8 MiB)

Number Start (sector) End (sector) Size Code Name
1 2048 7814019071 3.6 TiB BF01 zfs-01ad21b8d8ea1023
9 7814019072 7814035455 8.0 MiB BF07

GregorKopka · 2018-09-16T00:43:43Z

Looks good to me, apart from the protective partition no longer being at the absolute end of the drive.

Could you try to ln -s the -part1 device nodes of both the drives of the defective pool to an empty directory and then point the 0.7.11 version of zpool import (without specifying a pool name) toward that directory using -d?

artee666 · 2018-09-16T05:11:14Z

[root@rakef ~]# zpool import -d /pool/
pool: tank
id: 15804811324617297113
state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: http://zfsonlinux.org/msg/ZFS-8000-72
config:

tank                        FAULTED  corrupted data
  mirror-0                  ONLINE
    wwn-0x50014ee2baaf6436  ONLINE
    wwn-0x50014ee2baaf64b1  ONLINE

rincebrain · 2018-09-16T05:26:58Z

Does the pool still function on 0.7.10, or is it rejected that way for both 0.7.10 and 0.7.11 now?

I would guess that the reverted logic changes in 2a16d4c might have written an incorrect set of size values that work enough for the flawed calculations in it to do the right thing, but then when something without that flawed logic tries to open it, it decides rightfully that those values are wearing flaming underwear on their heads.

On 0.7.11, zdb -e -vv tank might eventually say something useful about why it's insane, or comparing the values from that on 0.7.10 and 0.7.11. (There are probably more precise ways to do this, but it's not my forte, so I'm iterating on the logic.)

This might also be getting bitten by #7443 or #7184 .

artee666 · 2018-09-16T05:58:28Z

On 0.7.10 seems to be fine:
[root@rakef ~]# zpool import
pool: tank
id: 15804811324617297113
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

tank                        ONLINE
  mirror-0                  ONLINE
    wwn-0x50014ee2baaf6436  ONLINE
    wwn-0x50014ee2baaf64b1  ONLINE

The only difference between zdb -e -vv is that on 0.7.10 it contains hostid and hostname and complains: zdb: can't open 'tank': File exists where on 0.7.11 it complains: zdb: can't open 'tank': Input/output error

artee666 · 2018-09-16T06:57:17Z

I ran strace zfs import on both 0.7.10 and 0.7.11 and on 0.7.10:
it opens /dev/zfs and runs
ioctl(3, _IOC(0, 0x5a, 0x5, 0), 0x7ffde460d160) = -1 ENOENT (No such file or directory)
ioctl(3, _IOC(0, 0x5a, 0x6, 0), 0x7ffde460d200) = 0
ioctl(3, _IOC(0, 0x5a, 0x3f, 0), 0x7ffde460d4f0) = -1 EPERM (Operation not permitted)
and closes the filehandle.

On 0.7.11 it opens /dev/zfs and:
ioctl(3, _IOC(0, 0x5a, 0x4, 0), 0x7ffd233a6060) = 0
ioctl(3, _IOC(0, 0x5a, 0x5, 0), 0x7ffd233a5ec0) = -1 ENOENT (No such file or directory)
ioctl(3, _IOC(0, 0x5a, 0x6, 0), 0x7ffd233a5f60) = 0
ioctl(3, _IOC(0, 0x5a, 0x2, 0), 0x7ffd233a5750) = -1 EIO (Input/output error)

tonyhutter · 2018-09-17T20:50:04Z

@artee666 I'm sorry you're running into problems importing your pool. Just so I'm clear, was this the sequence of events?:

Created a pool (do you remember which version you created it in?)
Attached DISK2 while running 0.7.10.
DISK2 resilvered 100% correctly.
After the disk was resilvered, you upgraded to 0.7.11.
DISK2 was reported missing

rincebrain · 2018-09-17T22:15:09Z

@tonyhutter AIUI the sequence is

Single-disk pool predates 0.7.10
While running 0.7.10, zpool attach pool disk1 disk2
Before resilvering finished, installed 0.7.11 SPL+ZFS and rebooted into it
fire

Between openzfs/zfs#7909,openzfs/zfs#7899, openzfs/zfs#7906 and http://list.zfsonlinux.org/pipermail/zfs-discuss/2018-September/032318.html, it seems like 0.7.10 should be clearly marked as "bad".

tonyhutter · 2018-09-21T00:53:38Z

I tried reproducing your error on some 8TB disks w/4k sectors (ashift=12), but no luck. Perhaps my disk geometry doesn't hit the issue. Here's the steps I did:

Created one-disk pool with 0.7.8.
Wrote 20GB to disk
Loaded 0.7.10
Attached a new disk to make a mirrored pair.
Rebooted when resilver was ~8% done.
Loaded 0.7.11
Imported pool, and watched it resilver correctly.
For sanity, ran a scrub and saw no errors.

My only suggestion is to detatch the disk, wipe the label, and then try re-attaching it while the pool is running 0.7.11.

artee666 · 2018-09-21T08:41:58Z

I think I've created the one disk pool with 0.7.10, wrote there data, attached another disk to make mirrored pair and rebooted into 0.7.11 while still resilvering. I've reinstalled my machine and now I'm running ubuntu with 0.7.5. So far so good :-)

behlendorf · 2018-10-12T22:02:56Z

Closing. 0.7.10 was marked a dud, and 0.7.11 released with the fix. Only disks added/replaced when running 0.7.10 with certain geometries where effected.

rincebrain mentioned this issue Sep 18, 2018

Mark 0.7.10 as on fire zfsonlinux/zfsonlinux.github.com#39

Merged

behlendorf closed this as completed Oct 12, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cannot upgrade zfs from 0.7.10 to 0.7.11 while resilvering pool #7909

cannot upgrade zfs from 0.7.10 to 0.7.11 while resilvering pool #7909

artee666 commented Sep 15, 2018

artee666 commented Sep 15, 2018

artee666 commented Sep 15, 2018

GregorKopka commented Sep 15, 2018

artee666 commented Sep 15, 2018 •

edited

Loading

artee666 commented Sep 15, 2018

artee666 commented Sep 15, 2018

GregorKopka commented Sep 15, 2018

artee666 commented Sep 15, 2018

artee666 commented Sep 15, 2018

artee666 commented Sep 15, 2018

GregorKopka commented Sep 15, 2018

artee666 commented Sep 15, 2018

GregorKopka commented Sep 16, 2018 •

edited

Loading

artee666 commented Sep 16, 2018

rincebrain commented Sep 16, 2018

artee666 commented Sep 16, 2018

artee666 commented Sep 16, 2018

tonyhutter commented Sep 17, 2018

rincebrain commented Sep 17, 2018

tonyhutter commented Sep 21, 2018

artee666 commented Sep 21, 2018

behlendorf commented Oct 12, 2018

cannot upgrade zfs from 0.7.10 to 0.7.11 while resilvering pool #7909

cannot upgrade zfs from 0.7.10 to 0.7.11 while resilvering pool #7909

Comments

artee666 commented Sep 15, 2018

System information

Describe the problem you're observing

artee666 commented Sep 15, 2018

artee666 commented Sep 15, 2018

GregorKopka commented Sep 15, 2018

artee666 commented Sep 15, 2018 • edited Loading

artee666 commented Sep 15, 2018

zpool offline tank wwn-0x50014ee2baaf6436

zpool status

zpool online tank wwn-0x50014ee2baaf6436

zpool status

artee666 commented Sep 15, 2018

GregorKopka commented Sep 15, 2018

artee666 commented Sep 15, 2018

artee666 commented Sep 15, 2018

artee666 commented Sep 15, 2018

GregorKopka commented Sep 15, 2018

artee666 commented Sep 15, 2018

GregorKopka commented Sep 16, 2018 • edited Loading

artee666 commented Sep 16, 2018

rincebrain commented Sep 16, 2018

artee666 commented Sep 16, 2018

artee666 commented Sep 16, 2018

tonyhutter commented Sep 17, 2018

rincebrain commented Sep 17, 2018

tonyhutter commented Sep 21, 2018

artee666 commented Sep 21, 2018

behlendorf commented Oct 12, 2018

artee666 commented Sep 15, 2018 •

edited

Loading

GregorKopka commented Sep 16, 2018 •

edited

Loading