ZFS replay transaction error 5, dataset omega/ROOT/precise, seq 0x2, txtype 9

[ejdonegan]

About 8 months ago my Ubuntu 11.04 system running native ZFS started displaying this error message.

(snip from a dmesg when I booted the system today)
.
.
[ 5.154614] SPL: Loaded module v0.6.0.76-rc10
[ 5.154998] zunicode: module license 'CDDL' taints kernel.
[ 5.155041] Disabling lock debugging due to kernel taint
[ 10.536384] ZFS: Loaded module v0.6.0.76-rc10, ZFS pool version 28, ZFS filesystem version 5
[ 10.541113] SPL: using hostid 0x007f0100
[ 11.150051] SPLError: 763:0:(spl-err.c:67:vcmn_err()) WARNING: ZFS replay transaction error 5, dataset delta/ROOT/natty-1, seq 0x4634, txtype 9
.
.

I lived with it at the time. It wasn't more than an annoyance, and I thought it was because I had been playing with a ZIL device (closest thing that matched the issue was a ZIL bug introduced and then fixed). Sometimes my system would display the message and boot, but more recently, it would drop to initramfs prompt after throwing an error:

(hand-written from the initramfs shell)
.
.
Begin: Setting mountpount=/ on ZFS file system delta/ROOT/natty-1 ... done
Begin: Mounting ZFS filesystem delta/ROOT/natty-1 ...

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [] mutex_lock +0x21/0x50
PGD:11b1c6067 PUD 122e06067 PMD0
Oops: 0002 [#1] SMP
last sysfs file: /sys/devices/virtual/bdi/zfs-1/uevent
CPU1
Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlibdeflate {... there are a bunch of modules - list truncated}
Pid: 752, comn:exe Tainted:P 2.6.38-15-generic #64-Ubuntu /DG41RQ
RIP: 0010 [] [] mutex lock +0x21/0x50
RSP: 0018:ffff88011b1e3888 EFLAGS: 0001 0246

At this point I would hard-power off the system, turn it back on, and it usually (75%) booted. Lately, it would take two, three, four times of power cycling...

Booting off LiveCD and doing a zpool scrub always came up with no errors. The drives were mirrored so I felt confident that the bits were good - just at bootup it was not happy. I did not need to reinstall the modules or other libs. I kept up-to-date with the zfs/daily updates, but the issue always remained.

I recently bought an SSD and did a clean install of Ubuntu 12.04 on a new SSD device. Things went fairly well following the native ZFS install guide (I'll make another ticket later about some things I ran into with the installation). I had some graphics-related driver issues and had to hard-reset the system.

Now when I boot this 12.04 install, I see:
.
.
[ 1.978494] ZFS: Loaded module v0.6.0.71-rc10, ZFS pool version 28, ZFS filesystem version 5
[ 2.188731] SPL: using hostid 0x007f0101
[ 2.443662] SPLError: 602:0:(spl-err.c:67:vcmn_err()) WARNING: ZFS replay transaction error 5, dataset omega/ROOT/precise, seq 0x2, txtype 9
.
.

The boot continues and the OS starts. I scrub the zpool (was not booted off the LiveCD though) - no errors detected.

I don't want to go through the disk issues again! I'm not mirrored on the new drive yet. I do have the other two disks attached so I can import in the old zpool (manually).

There may be two issues going on, but I'm not sure if there is a direct relation between them:

• the 'unable to handle kernel NULL pointer dereference at ...(null).
• the ZFS replay transaction error 5 WARNING messages

I've given you a bunch of symptoms but no backing data. What do you need from me to help chase this down?

-E

[dajhorn]

This is likely fixed in patch openzfs/zfs#970 for bug openzfs/zfs#933. The daily builds should get it this week.

[ejdonegan]

I'll test when released and update here.

[dajhorn]

The patch is in the 0.6.0.80 daily release, and it will be published to the stable PPA this week. Please reopen this ticket if the error persists.

Closed-by: 49d3979