Filesystem can not be mounted: Input/output error #13521
Comments
|
Some notes that may help:
|
|
After attempting
|
|
@tcaputi hope you don't mind the tag, apologies if you do. I just know you've been through this one before. |
|
Well, you're running a 2+ year old version with known bugs (not that the latest version of native encryption doesn't also have a host of known bugs nobody's fixed), so perhaps you should consider not doing that. That said, my wild guess would be that it's the unfixed bug where incremental receive will sometimes happily do |
rincebrain
added
Component: Send/Recv
|
@rincebrain Thank you! So it looks like in Ubuntu 20.04 I can update from
A few questions:
Just trying to figure out the best way to get up and running again under Ubuntu 20.04. Thanks! |
|
The current series of releases is 2.1.x, you are on 0.8.x. Upgrading from 0.8.3-mumble13 to 0.8.3-mumble14 was not my suggestion, though you could always try reporting your issue to Ubuntu's bugtracker in that case, as I would be pretty astonished if 0.8.x saw any fixes or releases ever again here. Upgrading things won't fix this problem at this point. If I'm right about what happened, it wouldn't have even prevented them before, since that bug is unfixed. One could write code to rekey the individual datasets forcibly ignoring that you don't have the old key loaded, there's even the framework for it to exist already, it's just not written anywhere (that I'm aware of); I have an incomplete version that doesn't work on every key type at the moment, but haven't gone back to figure out what's missing in a bit. |
|
@rincebrain Well that doesn't sound promising then. What's your recommended upgrade methodology to 2.1.x for Ubuntu 20.04 that wouldn't involve me losing access to data on the main server? They're supposed to still be supporting 20.04, but it sounds like zfs is woefully out of date there. I assume by "write code" you don't mean use the Is there any way I can get this to work without starting from scratch? Such as leveraging the first snap that is mountable in combination with an upgrade (from.. somewhere)? This is many TB of data separated by many states, so I can't easily do full sends across the internet. |
|
From this reddit thread it looks like I could upgrade without issue, and from this one there appears to be a maintained PPA. So if I went that route, upgraded on both systems, would it be possible to roll back to a working snapshot (if I have one) on the backup server, then |
|
I believe there's a popular PPA; you could also build packages from source; that's just general advice though, if I'm correct about the problem, it's still unfixed now too, so it wouldn't have helped. I don't, no. If you can mount some old snapshots, though, but not newer ones, on the same dataset, I don't think it can be the bug I'm describing, because I don't think there's any sort of old key history kept, so they should all be equally unusable. One thing you could try, potentially, assuming it's one of the encryption bugs that has been fixed instead, would be to upgrade to 2.1.4 using the PPA or building your own packages or w/e, import the pool, see if you can mount things, and if not, you could try doing a raw send of the encrypted dataset and receiving it anew on the same pool somewhere else, e.g. |
|
Ahh yes. I can mount the original snapshots, just like in the first thread I linked. Every incremental snapshot that was sent after won't mount, but the first will. And any pool that only has that first snapshot also mounts as expected. A big concern I have right now is if the main server reboots and I lose access to my data altogether. |
|
Btw, this is what I mean by they are different from a key perspective: Main server: Backup server: The backup server was inheriting from I.E.:
Would the backup server get confused without |
|
Yes, that is certainly quite confused, but I think I'm surprised that it can still mount the old one, as like I said, changing the wrapping key or no, the actual key that encrypts the data has remained the same, so you should still need the same unlock for both cases, AFAIK. |
|
Ok, thanks. If you were in my boat, what steps would you take? |
|
I'd probably try the send/recv thing I mentioned above, and if that didn't work, go hackily patch the code to override that property value in that specific case to be what you know to be correct and/or finish the WIP code for forcibly resetting the property. I'm taking a break for a bit after I spent a bunch of time today hacking on something complicated and then someone pointed out I was being foolish and there was a much simpler solution, but if you remind me later I'll see if I can't make that branch that's incomplete for fixing this complete and then you can try it. |
Yes, I agree. This looks more like the user accounting MAC issue which was fixed in #12981 a couple of months ago. So I'd say a recent ZFS version (2.1.4 or git@master) should be able to mount the datasets, at least it's worth a try. OTOH if it's really an issue with overwriting the wrapping key, it would be good to see the exact sequence of commands which got you into this situation. Unfortunately, from the description you gave I can't deduce the exact commands you issued. I've a couple of reproducers for key issues which I can compare your commands against. With "mount the snapshot" you mean |
|
I think #12981 only means you could send/recv and then unlock the resulting recv, not mount the existing dataset, though I think writing a zhack command to trigger the same behavior without a recv might be reasonably doable... |
|
I will give some of this a shot, and also try to recreate the issue before (and after) I update. If I need to fully re-send, should I be using raw, or re-encrypting at destination? |
|
Re-encrypting at destination would avoid problems with change-key propagating surprisingly, but there are other bugs which only require receiving raw no matter what (hi #11679), so pick your poison, IMO. |
|
@rincebrain Honestly I can't tell, didn't look very thoroughly, sorry. This issue just reminded me of an issue a had some time ago where I couldn't mount a raw send/recvd encrypted dataset (without key manipulation) while I could mount an older snapshot. It was due to the MAC of the accounting object failing and it did show the @Fmstrat Depends on your use case. If you want to do raw incremental sends, re-encrypting isn't an option. If you do raw sends, I'd strongly encourage you to not manipulate the keys on the receiving side since there are some edge cases where you could end up with inaccessible received datasets. If the send communication channel is trusted OTOH, you can re-encrypt plain incremental sends. |
|
And of course what @rincebrain just wrote. |
|
@AttilaFueloep i wonder if the root cause of that is actually related to the #11679 panic, which it turns out is caused by the accounting code's dirtying every object and syncing out trick causing inconsistent state in memory and a NULL deref if someone tries to read the dnode at the wrong time |
|
@AttilaFueloep what exactly do you mean by "manipulate the keys"? I'm trying to understand if this would help my current situation and if so, how. |
|
Ok, before I try anything else, I have this fully reproducible in # Make server A, create snapshot 1, then make that pool read-only
truncate -s 1G /tmp/tmpa.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmpa /tmp/tmpa.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmpa/storage tmpa/storage
zfs create -o mountpoint=/mnt/tmpa/storage/test tmpa/storage/test
echo 1 > /mnt/tmpa/storage/test/testfile.txt
zfs snap -r tmpa/storage@1
zfs set readonly=on tmpa/storage/test
# Make server B, send from A to B
truncate -s 1G /tmp/tmpb.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmpb /tmp/tmpb.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmpb/storage tmpb/storage
zfs send -w tmpa/storage/test@1 |pv -Wbraft |zfs recv tmpb/storage/test
# Set server B writable, change file, send to A
zfs set readonly=off tmpb/storage/test
zfs set mountpoint=/mnt/tmpb/storage/test tmpb/storage/test
zfs set keylocation=file:///etc/zfskey tmpb/storage/test
zfs load-key tmpb/storage/test
zfs mount tmpb/storage/test
echo 2 > /mnt/tmpb/storage/test/testfile.txt
zfs snap tmpb/storage/test@2
zfs send -w -I tmpb/storage/test@1 tmpb/storage/test@2 |pv -Wbraft |zfs recv tmpa/storage/test
# Unmount and try to remount A. Failure
zfs unmount tmpa/storage/test
zfs mount tmpa/storage/test
# Cleanup
zpool destroy tmpa
zpool destroy tmpb
rm /tmp/tmpa.img
rm /tmp/tmpb.imgGiven this, would you recommend me try anything else before I upgrade? |
|
I did the above on a spare laptop with Ubuntu 20.04 on it, then upgraded to this version from the PPA: However I'm still unable to mount, so no luck there. I then tried re-sending the incremental in hopes I could salvage: But still no luck on mounting afterwards. For the heck of it, I reran the above test from scratch in 2.1.4 and lo and behold, the issue exists there, too. |
|
So far the only way I can get this to work is by re-sending the entire pools from scratch and setting them up with their own encryption roots: # Make server A, create snapshot 1, then make that pool read-only
truncate -s 1G /tmp/tmp2a.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmp2a /tmp/tmp2a.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmp2a/storage tmp2a/storage
zfs create -o mountpoint=/mnt/tmp2a/storage/test tmp2a/storage/test
echo 1 > /mnt/tmp2a/storage/test/testfile.txt
zfs snap -r tmp2a/storage@1
zfs set readonly=on tmp2a/storage/test
# Make server B, send from A to B
truncate -s 1G /tmp/tmp2b.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmp2b /tmp/tmp2b.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmp2b/storage tmp2b/storage
zfs send -w tmp2a/storage/test@1 |pv -Wbraft |zfs recv tmp2b/storage/test
# Set server B writable, change file, send to A
zfs set readonly=off tmp2b/storage/test
zfs set mountpoint=/mnt/tmp2b/storage/test tmp2b/storage/test
zfs set keylocation=file:///etc/zfskey tmp2b/storage/test
zfs load-key tmp2b/storage/test
zfs mount tmp2b/storage/test
echo 2 > /mnt/tmp2b/storage/test/testfile.txt
zfs snap tmp2b/storage/test@2
zfs send -w -I tmp2b/storage/test@1 tmp2b/storage/test@2 |pv -Wbraft |zfs recv tmp2a/storage/test
# Unmount and try to remount A. Failure
zfs unmount tmp2a/storage/test
zfs mount tmp2a/storage/test
# Destroy server A test pool and resend from scratch
zfs destroy -r tmp2a/storage/test
zfs send -w tmp2b/storage/test@2 |pv -Wbraft |zfs recv tmp2a/storage/test
zfs set readonly=on tmp2a/storage/test
zfs set mountpoint=/mnt/tmp2a/storage/test tmp2a/storage/test
zfs set keylocation=file:///etc/zfskey tmp2a/storage/test
zfs load-key tmp2a/storage/test
zfs mount tmp2a/storage/test
# Make a change on server B and send
echo 3 > /mnt/tmp2b/storage/test/testfile.txt
zfs snap tmp2b/storage/test@3
zfs send -w -I tmp2b/storage/test@2 tmp2b/storage/test@3 |pv -Wbraft |zfs recv tmp2a/storage/test
# Unmount and try to remount A. Success
zfs unmount tmp2a/storage/test
zfs mount tmp2a/storage/test
# Cleanup
zpool destroy tmp2a
zpool destroy tmp2b
rm /tmp/tmp2a.img
rm /tmp/tmp2b.img |
|
One more test. The error also occurs if I send the entire pool, instead of just "test": # Make server A, create snapshot 1, then make that pool read-only
truncate -s 1G /tmp/tmp2a.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmp2a /tmp/tmp2a.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmp2a/storage tmp2a/storage
zfs create -o mountpoint=/mnt/tmp2a/storage/test tmp2a/storage/test
echo 1 > /mnt/tmp2a/storage/test/testfile.txt
zfs snap -r tmp2a/storage@1
zfs set readonly=on tmp2a/storage/test
# Make server B, send from A to B
truncate -s 1G /tmp/tmp2b.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmp2b /tmp/tmp2b.img
# -=> First difference, send the entire pool recursively
#zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmp2b/storage tmp2b/storage
#zfs send -w tmp2a/storage/test@1 |pv -Wbraft |zfs recv tmp2b/storage/test
zfs send -wR tmp2a/storage@1 |pv -Wbraft |zfs recv tmp2b/storage
# Set server B writable, change file, send to A
zfs set readonly=off tmp2b/storage/test
zfs set mountpoint=/mnt/tmp2b/storage/test tmp2b/storage/test
# -=> Next difference, key location and load keys for storage
#zfs set keylocation=file:///etc/zfskey tmp2b/storage/test
#zfs load-key tmp2b/storage/test
zfs set keylocation=file:///etc/zfskey tmp2b/storage
zfs load-key tmp2b/storage
zfs mount tmp2b/storage/test
echo 2 > /mnt/tmp2b/storage/test/testfile.txt
zfs snap tmp2b/storage/test@2
zfs send -w -I tmp2b/storage/test@1 tmp2b/storage/test@2 |pv -Wbraft |zfs recv tmp2a/storage/test
# Unmount and try to remount A. Failure
zfs unmount tmp2a/storage/test
zfs mount tmp2a/storage/test
# Cleanup
zpool destroy tmp2a
zpool destroy tmp2b
rm /tmp/tmp2a.img
rm /tmp/tmp2b.img |
|
Sorry for the flood, just trying to eliminate dead-ends. Using recursive snap doesn't seem to be the issue. It seems to be related to when the encryption root exists in # Make server A, create snapshot 1, then make that pool read-only
truncate -s 1G /tmp/tmp2a.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmp2a /tmp/tmp2a.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmp2a/storage tmp2a/storage
zfs create -o mountpoint=/mnt/tmp2a/storage/test tmp2a/storage/test
echo 1 > /mnt/tmp2a/storage/test/testfile.txt
# -=> Just snap test
#zfs snap -r tmp2a/storage@1
zfs snap tmp2a/storage/test@1
zfs set readonly=on tmp2a/storage/test
# Make server B, send from A to B
truncate -s 1G /tmp/tmp2b.img
zpool create -o ashift=12 -O atime=off -O mountpoint=none tmp2b /tmp/tmp2b.img
zfs create -o encryption=on -o keyformat=hex -o keylocation=file:///etc/zfskey -o compression=lz4 -o mountpoint=/mnt/tmp2b/storage tmp2b/storage
zfs send -w tmp2a/storage/test@1 |pv -Wbraft |zfs recv tmp2b/storage/test
# Set server B writable, change file, send to A
zfs set readonly=off tmp2b/storage/test
zfs set mountpoint=/mnt/tmp2b/storage/test tmp2b/storage/test
zfs set keylocation=file:///etc/zfskey tmp2b/storage/test
zfs load-key tmp2b/storage/test
zfs mount tmp2b/storage/test
echo 2 > /mnt/tmp2b/storage/test/testfile.txt
zfs snap tmp2b/storage/test@2
zfs send -w -I tmp2b/storage/test@1 tmp2b/storage/test@2 |pv -Wbraft |zfs recv tmp2a/storage/test
# Unmount and try to remount A. Failure
zfs unmount tmp2a/storage/test
zfs mount tmp2a/storage/test
# Cleanup
zpool destroy tmp2a
zpool destroy tmp2b
rm /tmp/tmp2a.img
rm /tmp/tmp2b.img |
|
@rincebrain Your last comment there was an interesting read. Look like the integration of encryption with the quota accounting still has some rough edges. Maybe I can say more after replaying the given reproducers. |
Well, running |
|
Nice to see some reproducers. It's quite late here already, I'll try to digest your input and reproduce it on a somewhat current master tomorrow. |
|
Thanks! As a note, I have tried the below with no success, too. I'll hold off today and doing any sends since we're talking large amounts of data here, just in case as you dive in you find a way I can recover. # zfs change-key -o keyformat=hex -o keylocation=file:///etc/zfskey tmpa/storage/test
# zfs mount tmpa/storage/test
filesystem 'tmpa/storage/test' can not be mounted: Input/output error
cannot mount 'tmpa/storage/test': Invalid argument |
|
I also just tried zfs change-key -i tmpb/storage/test
zfs send -w -I tmpb/storage/test@1 tmpb/storage/test@2 |pv -Wbraft |zfs recv tmpa/storage/test
zfs mount tmpa/storage/test
echo 3 > /mnt/tmpb/storage/test/testfile.txt
zfs snap tmpb/storage/test@3
zfs send -w -I tmpb/storage/test@2 tmpb/storage/test@3 |pv -Wbraft |zfs recv tmpa/storage/test
zfs mount tmpa/storage/test
zfs send -w -I tmpb/storage/test@1 tmpb/storage/test@3 |pv -Wbraft |zfs recv tmpa/storage/test
zfs mount tmpa/storage/test |
If you find where the rough edge is, let me know, but it's very strange that stubbing it out like that ameliorates the problem in #11679 - my guess is, having done some debug printing, I noticed that it dirties and syncs out the same dnode multiple times per syncing context of the same txg, and I would not be surprised if some of the async callback stuff doesn't properly lock or assumes multiple people won't be touching it at once, but I haven't gotten back to looking at why. Now, back to this thread... @Fmstrat you were running the 2.0.2 kernel modules, which is before the workarounds we suggested might help were implemented - you probably needed to reboot or manually unload and reload the kernel modules to get them to 2.1.4 after upgrading from the PPA. And yes, changing the key on something that wasn't originally an encryptionroot and then sending it back incrementally reproduces this pretty readily. Unfortunately, the amount of people-time working on encryption bugs relative to the number of people reporting them is pretty small, and I can't speak for anyone but myself, but I've been focusing on #11679 because it's been averaging a new report every 10 days or so. |
|
@rincebrain Oh geez, I did reboot, but maybe that PPA isn't fully up-to-date or I need a kernel upgrade first. Running a full system update on that laptop now to see. EDIT Nvm, I realized with the PPA I have to manually install |
|
@rincebrain So it looks like that's what's in the PPA. The guides for Ubuntu says to use the packages. Is there a guide anywhere for getting running with the latest release? |
|
The PPA is shipping modules with the version string 2.1.4-0york0~20.04 - note the lack of ubuntu anywhere in it. So you're running old modules that presumably came with the OS install. I would have expected the workaround documented here to be unnecessary with those packages, but perhaps not - give it a try? What does |
|
Well, @AttilaFueloep, please disregard my tests. With the 2.1.4 module, the failures now work. I have one other machine I can try an upgrade test on, to see if upgrading to 2.1.4 fixes things right off. |
|
That's great. I'm curious if a 2.1.4 module will be able to mount the datasets not mountable by 2.0.2 or if you need to resend your data to fix this. As a side note, you may want to look at my comment in #12123 and the issues cited there for more detail why not to change keys on raw received datasets. |
|
@AttilaFueloep It would not mount after the upgrade and reboot. However, I made a new snap of |
|
If you give me a day or two, I'd be curious if I could cut a zhack command
to trigger recalculating that like a send/recv does.
If not, nbd, it's not like this is hard to reproduce, it's just nicer to
test with actual examples once people are sure the data is safe.
…On Tue, May 31, 2022, 10:31 AM Ben Curtis ***@***.***> wrote:
@AttilaFueloep <https://github.com/AttilaFueloep> It would not mount
after the upgrade and reboot. However, I made a new snap of
tmpb/storage/test, sent it, and then tmpa/storage/test mounted! So I'm
going to run some data-integrity tests to make sure previous data works and
see what happens with previous snaps/removing/etc before I call that a win.
—
Reply to this email directly, view it on GitHub
<#13521 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABUI7PCSN4QZ7GHZUNPZUTVMYPD3ANCNFSM5XIHFSQA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@rincebrain I'd be happy to give it a shot. I've just completed testing on the sends. All data in my test cases remains intact, so it's the method I'll use for my production data. Basically I ran through scenarios of adding files, clearing snapshots, etc, and checked to make sure all files remained and were accessible in the latest snapshot that would mount. |
|
I'm still a bit worried about the encryption root change you are doing by running |
|
@AttilaFueloep Thanks for the tip. I'm not actually doing a NOTE: I can't mount the old snapshots still, but the latest one does. |
|
You will probably find an idea I had in another thread about this useful. Or maybe not, it might not be the same bug. We'll see. |
|
This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions. |
System information
Describe the problem you're observing
After a reboot, ZFS volumes aren't mounting, some do, some don't.
The issue with the snapshot above was not visible at boot. It only came up after I tried to mount it. I was able to mount the original snapshot (non-incremental) with
mount -t zfs s/storage/n/[email protected] /mnt/testbut I don't have original snapshots for every volume.Have I lost this data or is this recoverable? What works vs what doesn't seems eerily similar to #8103
Describe how to reproduce the problem
This happened after the filesystem locked up the server. Commands accessing the zpool stopped responding, and required a forced reboot (kill -9 didn't work).
The text was updated successfully, but these errors were encountered: