zfs send -R from FreeNAS9 and receive on a ZoL result in hard lock #2602
Milestone
Comments
|
Tried again, and yet again I woke up to a hard locked kernel... And now the 'import' (WITH What could cause a simple send+receive to fuck up the pool so badly that it becomes impossible to import!? |
|
Screenshot from failure.
|
|
@FransUrbo The only thing which will cause a hard lockup is if a cpu is spinning in a function with interrupts disabled. If you're seeing this on every import can you get the backtrace from the console. In particular, the chunk of the dump which includes the You could also try disabling the nmi watchdog. This might allow the system to boot as long as the code we're spinning in does eventually terminate. We just need to identify what the block of code is. This could perhaps be done by running |
|
@FransUrbo This is a complete and total guess. But pull #2614 is designed to minimize the number of reads in |
It WILL hard lock eventually non the less. But my terminal is to small (and I have not been able to increase the virtual terminal size - can't find a fb module that works on my graphic card). And the SysRQ keys don't work at this point unfortunately.
All datasets have now been synced (for the umpteenth time - have to recreate and resync everything from scratch every time) and it's only the ZVOLs missing. I tried last time to go back to a TXG that just before the ZVOLs, but then I got 'missing device' or something like that... How do I figure out the current TXG? Looking at "zdb | grep -w txg:", I see: Is that the txg I should use if (when) it crashes again, so that I don't have to do everything from scratch again? |
Can't find #2614 (latest is 2611 :). |
|
No |
|
OK, given the stacks I can explain a small part of this. The One possibility is that this is being caused by some old compatibility code in the spl. Can you try reverting commit openzfs/spl@46aa7b3 and see if it resolves the issue. If that doesn't do the trick then I would try rolling back to zfs commit 4dd1893 and see if you can reproduce this issue. I don't think this is a new issue we introduced, but changes after that commit may have made it more likely. |
|
@FransUrbo Actually, give me a few hours and I'll have a patch for the SPL you can test. I just want to test it first. |
Ah, that's the same as #2480 which I already prepared to use (can't remember if I actually was running that PR when I was doing the final sync of the ZVOLs but I think I was). I'm doing it now and the import still gives a stack trace (no hard locks any more since I disabled the NMI watchdog). Trying to go back to the TXG before that sync just gives: Tried with and without #2598. I'm waiting for your spl patch and see if that allows me to import the pool so I don't have to resync everything again... |
|
Using spl/master and going back to 4dd1893 still gives a stack trace at import and using '-T...' still gives |
|
@FransUrbo Can you try openzfs/spl#383. It removes a lot of legacy compatibility code and the non-critical |
|
|
Still down in |
|
Looks different now. This is with openzfs/spl#383 and zfs reverted to 4dd1893. |
|
With zfs/master (and still with openzfs/spl#383), we're back to the previous problem: |
|
With additional, undocumented, options to |
|
And as soon as I run |
|
Running |
|
Ok. The issue looks the same in all those stacks. I'll post a patch tomorrow to dump the pid of the task holding the offending leaf lock. That should shed some light on this. |
|
@behlendorf Regarding |
|
No idea what I'm doing, but a Running the suggested No idea where the |
|
@FransUrbo After enabling debugging I hit the following assert. I haven't had a chance to run down why yet. This hit before we got to the code for the debug patch I was intending to test. The debug patch I was intending to test can be found at behlendo/zfs@f4fd052e631637d13b3e5c26df2b150a364c9548. It should return the pid of the process holding the lock, if there is one. |
|
After trying again - recreating the pool and push all filesystems (worked just fine), I then used my script to send indiviual (the last) snapshot using Woke up this morning with a crashed kernel again :( Script basically only do So it seems even a |
|
Adding a dprintf() to get |
|
I think I found the problem... This dataset ( And I don't have a 'backup' of the metadata for it. I just remember seeing that it was So apparently the FreeNAS9 machine happily sent the ZVOL to the ZoL machine, which after it received it crashed. And now, when it tries to import the pool, it sees this ZVOL, and then borks... |
|
Thinking even harder (and looking/grep'ing at the metadata I've kept about this whole thing, ever since the original crash back in January) I'm starting to remember that all the snapshots for that ZVOL was also broken. How I got it over to the FreeNAS9 machine is still a mystery, but I think that I had to snapshot it there (did a recursive snap from the top) and then pushed that back... |
|
And getting the extra |
|
Commenting out the |
|
@FransUrbo That dataset/zvol is clearly damaged on disk. You could probably patch zap_hash() to return an error instead of hitting the ASSERT. Then just make sure an error gets returned all the way to the ioctl(). This might allow you to import the rest of the pool so you can cleanly destroy this broken dataset. |
|
I tried make it return But shouldn't we investigate this further? A broken ZVOL that cause this much trouble... ? It's obviously not that big of a problem on *BSD... Any information or something else you want me to take before I go on trying to destroy it? |
|
Dang! It seems there was a downside of doing that... The ZVOL is not in the list and a |
|
Not sure if that's because the way I did it:
This gives: |
|
The pool got imported in any case and most (625 out of 640) filesystems is mountable: So returning |
|
Crud! All this messing about eventually lead to |
|
After starting from scratch and initiating the send, I decided to take a nap. Four hours later, I wake up to a machine with a locked CPU. A hard-lock, if not for the Importing the pool went mostly ok, although I got It's not in the list of filesystems/volumes... |
|
Starting the send again, skipping (with my |
|
Looks like that part is related to #1761. |
|
After a couple of crashes, reboots and imports (these now work without any problem) I've started to sync the filesystems. I'm now up to my second crash/hard lock ( Memory (usage) problems? |
|
Yet another crash. This time I got the advice to use |
|
Running with debugging modules: http://bayour.com/misc/screenlog.2.txt This time, only 9 snapshots... |
Closed
|
Using MD+LVM+XFS in the same type of layout as the previous zpool, I managed to copy ALL data without so much as a hickup or warning. Not a pip from the system! Going back to a zpool, I this time decided to use It's no where near finished (~2TB of ~22TB). It's going to take ages, but seems to work. |
|
@FransUrbo you must be running with #2387 which clearly introduced this NULL dereference. The function in question |
|
@behlendorf You mean for the |
|
@FransUrbo the kernel NULL dereference basically means you've halted that process. If it's holding any resources or locks other processes need you system may deadlock. You can certainly continue to limp along just be aware of that. Reverting the vdev timeout patch will resolve the NULL dereference you hit. |
|
Btw, I see that I never changed the url in #2602 (comment). It should have been http://bayour.com/misc/screenlog.7.txt. Removing #2387 and trying again gives http://bayour.com/misc/screenlog.8.txt after a while. |
|
Conclusion must be that this must have been a hardware issue at hart... Borrowed a second machine from my friend (dedicated server), but with my disks. The transfer was successful. It only crashed twice, both times completely unrelated to ZoL. |
|
I'm glad you got it sorted out. Although I was hoping for a more satisfying answer... |
ryao
pushed a commit
to ryao/zfs
that referenced
this issue
Nov 29, 2014
If a non-ZAP object is passed to zap_lockdir() it will be treated as a valid ZAP object. This can result in zap_lockdir() attempting to read what it believes are leaf blocks from invalid disk locations. The SCSI layer will eventually generate errors for these bogus IOs but the caller will hang in zap_get_leaf_byblk(). The good news is that is a situation which can not occur unless the pool has been damaged. The bad news is that there are reports from both FreeBSD and Solaris of damaged pools. Specifically, there are normal files in the filesystem which reference another normal file as their parent. Since pools like this are known to exist the zap_lockdir() function has been updated to verify the type of the object. If a non-ZAP object has been passed it EINVAL will be returned immediately. Signed-off-by: Brian Behlendorf <[email protected]> Issue openzfs#2597 Issue openzfs#2602
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
I have no other information that that. It seems to be completely random and I have not been able to reproduce it.
I first thought it was ryao/zvol (#2484, some information of value might be there, including the only 'screenshot' (literally!) I have), but it happens on an absolutly clean master as well.
First time it crashed (running #2484 and some of my own PRs - see #2484 (comment)), I was able to import the pool, but not run any
zfscommand (such as a simplezfs listwithout hard locking the kernel).Second time, running clean ZoL, it crashed after all datasets and got 20 ZVOLs in.
Third time, running ONLY #2484, it crashed 18 ZVOLs in (all datasets sent just fine before the ZVOLs).
After a reboot and import, I'm now sending ONLY the ZVOLs (all the other data have already been sent successfully). Crashed again, and
zfs list(again!) hangs the kernel after about 20 seconds..Using the recommended #2338 do not help.
The text was updated successfully, but these errors were encountered: