Unable to import pool stored in file #412
Replies: 22 comments 33 replies
-
|
Show your work? One common misconception from Linux is that Maybe place it a folder as well, instead of root. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! My stupidity! I'm still working on a simplified test case, but it looks right now that it is only a display problem in non-root DSs (at least in Powershell hard links are pointing wrongly in Get-Item | Format-List). I'll continue on this. |
Beta Was this translation helpful? Give feedback.
-
|
Can you try to run the following script on some sort of Linux (I used Ubuntu 24) and then copy the image file over to Windows and import it? When I do this I am unable to access DS zfs_windows_test_pool/backups. Moving over a USB device worked like a charm.
Maybe I'm doing something wrong here? |
Beta Was this translation helpful? Give feedback.
-
|
I can totally believe hardlinks are off, not heavily tested. Thanks for the commands, I will give it a go. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the speedy response. Please uncompress the attached file under Windows (should then be 100MB in size) and import it.
Please note that this is currently not yet the hardlink "problem": after import you should see a DS zfs_windows_test_pool/backups which I cannot access by any means: a directory /backup is created but it is empty in my case.
|
Beta Was this translation helpful? Give feedback.
-
|
Sorry, forgot: this is what you should see under Windows:
***@***.***:~# ls -lR /zfs_windows_test
/zfs_windows_test:
total 1
drwxr-xr-x 4 root root 4 Nov 1 11:34 backups
/zfs_windows_test/backups:
total 1
drwxr-xr-x 3 root root 3 Nov 1 11:34 SystemA
drwxr-xr-x 3 root root 3 Nov 1 11:34 SystemB
/zfs_windows_test/backups/SystemA:
total 1
drwxr-xr-x 2 root root 3 Nov 1 11:34 Dir1
/zfs_windows_test/backups/SystemA/Dir1:
total 2205
-rw-r--r-- 2 root root 2277376 Nov 1 11:34 voyager-ext.mp3
/zfs_windows_test/backups/SystemB:
total 1
drwxr-xr-x 2 root root 3 Nov 1 11:34 Dir1
/zfs_windows_test/backups/SystemB/Dir1:
total 2205
-rw-r--r-- 2 root root 2277376 Nov 1 11:34 voyager-ext.mp3
Von: Jorgen Lundman ***@***.***>
Gesendet: Freitag, 1. November 2024 10:27
An: openzfsonwindows/openzfs ***@***.***>
Cc: Erwin53 ***@***.***>; Author ***@***.***>
Betreff: Re: [openzfsonwindows/openzfs] Unable to import pool stored in file (Discussion #412)
I can totally believe hardlinks are off, not heavily tested. Thanks for the commands, I will give it a go.
Alternatively, if you create the pool file of zeros (dd, mkfile etc) of say, 1g in size, and make the issue happen, you can gzip up the image and send it to me as well.
-
Reply to this email directly, view it on GitHub<#412 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJTYOUXE7LXECPOGU74QRL3Z6NCO3AVCNFSM6AAAAABQ6QO2IGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMJSGAZDSMA>.
You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>
|
Beta Was this translation helpful? Give feedback.
-
|
You might need to email me directly at |
Beta Was this translation helpful? Give feedback.
-
|
OK imported and the directory is empty, as expected - let me take a look |
Beta Was this translation helpful? Give feedback.
-
|
are you sure there are supposed to be items inside |
Beta Was this translation helpful? Give feedback.
-
|
Yes, this is exactly the problem:
***@***.***:~# cd /dpool/scratch/
***@***.***:/dpool/scratch# ll
total 1,0K
drwxr-xr-x 2 root root 2 Nov 1 16:01 ./
drwxr-xr-x 4 root root 4 Jun 27 11:39 ../
***@***.***:/dpool/scratch# cp -v /client_backups/zfs_windows_test.img.gz .
'/client_backups/zfs_windows_test.img.gz' -> './zfs_windows_test.img.gz'
***@***.***:/dpool/scratch# gunzip zfs_windows_test.img.gz
***@***.***:/dpool/scratch# ll
total 1,5K
drwxr-xr-x 2 root root 3 Nov 2 07:25 ./
drwxr-xr-x 4 root root 4 Jun 27 11:39 ../
-rw-r--r-- 1 root root 100M Nov 2 07:25 zfs_windows_test.img
***@***.***:/dpool/scratch# zpool import -d .
pool: zfs_windows_test_pool
id: 2567215879436983193
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
zfs_windows_test_pool ONLINE
/dpool/scratch/zfs_windows_test.img ONLINE
***@***.***:/dpool/scratch# zpool import -d . zfs_windows_test_pool
***@***.***:/dpool/scratch# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bkparch 3.62T 2.27T 1.35T - - 3% 62% 1.00x ONLINE -
dpool 10.9T 3.85T 7.05T - - 11% 35% 1.49x ONLINE -
zfs_windows_test_pool 80M 2.77M 77.2M - - 7% 3% 1.00x ONLINE -
***@***.***:/dpool/scratch# zfs list | grep windows
zfs_windows_test_pool 2.77M 37.2M 96K /zfs_windows_test
zfs_windows_test_pool/backups 2.25M 37.2M 2.25M /zfs_windows_test/backups
***@***.***:/dpool/scratch# ll /zfs_windows_test/
total 5,0K
drwxr-xr-x 3 root root 3 Nov 1 10:44 ./
drwxr-xr-x 33 root root 4,0K Nov 1 10:44 ../
drwxr-xr-x 4 root root 4 Nov 1 10:44 backups/
***@***.***:/dpool/scratch# ll -aR /zfs_windows_test/
/zfs_windows_test/:
total 5,0K
drwxr-xr-x 3 root root 3 Nov 1 10:44 ./
drwxr-xr-x 33 root root 4,0K Nov 1 10:44 ../
drwxr-xr-x 4 root root 4 Nov 1 10:44 backups/
/zfs_windows_test/backups:
total 2,0K
drwxr-xr-x 4 root root 4 Nov 1 10:44 ./
drwxr-xr-x 3 root root 3 Nov 1 10:44 ../
drwxr-xr-x 3 root root 3 Nov 1 10:44 SystemA/
drwxr-xr-x 3 root root 3 Nov 1 10:44 SystemB/
/zfs_windows_test/backups/SystemA:
total 1,5K
drwxr-xr-x 3 root root 3 Nov 1 10:44 ./
drwxr-xr-x 4 root root 4 Nov 1 10:44 ../
drwxr-xr-x 2 root root 3 Nov 1 10:44 Dir1/
/zfs_windows_test/backups/SystemA/Dir1:
total 2,2M
drwxr-xr-x 2 root root 3 Nov 1 10:44 ./
drwxr-xr-x 3 root root 3 Nov 1 10:44 ../
-rw-r--r-- 2 root root 2,2M Nov 1 10:44 voyager-ext.mp3
/zfs_windows_test/backups/SystemB:
total 1,5K
drwxr-xr-x 3 root root 3 Nov 1 10:44 ./
drwxr-xr-x 4 root root 4 Nov 1 10:44 ../
drwxr-xr-x 2 root root 3 Nov 1 10:44 Dir1/
/zfs_windows_test/backups/SystemB/Dir1:
total 2,2M
drwxr-xr-x 2 root root 3 Nov 1 10:44 ./
drwxr-xr-x 3 root root 3 Nov 1 10:44 ../
-rw-r--r-- 2 root root 2,2M Nov 1 10:44 voyager-ext.mp3
***@***.***:/dpool/scratch#
Do you have any Linux system where you can try to import the file?
Von: Jorgen Lundman ***@***.***>
Gesendet: Samstag, 2. November 2024 07:23
An: openzfsonwindows/openzfs ***@***.***>
Cc: Erwin53 ***@***.***>; Author ***@***.***>
Betreff: Re: [openzfsonwindows/openzfs] Unable to import pool stored in file (Discussion #412)
are you sure there are supposed to be items inside backups ? zap reports no entries.
-
Reply to this email directly, view it on GitHub<#412 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJTYOUWFXRXIY3IS3GYOO23Z6RVUDAVCNFSM6AAAAABQ6QO2IGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMJSG43DKMY>.
You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>
|
Beta Was this translation helpful? Give feedback.
-
|
Oh I see: OK so when it mounts the lower dataset |
Beta Was this translation helpful? Give feedback.
-
|
Okay! This one ends up being quite amusing; With all my own tests, I have assumed the pool name What's happening here is we mount pool So clearly the code that works out |
Beta Was this translation helpful? Give feedback.
-
|
If you could this build of zpool and zfs: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. I can now see the directory
but the directory is empty (nothing in it: |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Ah yeah ok, it's a bit racy. If import -N and mount one at a time we are ok, but during a normal import mount storm it doesnt quite work out |
Beta Was this translation helpful? Give feedback.
-
|
Oh I hadn't actually recompiled |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, still no change for me: directory is there but empty. |
Beta Was this translation helpful? Give feedback.
-
|
Show me new text output? |
Beta Was this translation helpful? Give feedback.
-
|
Yeah so it looked it up too soon, and got the Volume name before it settled on D: Need to address this race somehow. If you |
Beta Was this translation helpful? Give feedback.
-
|
Yep, I can confirm that manually mounting the DSs fixes the problem.
Von: Jorgen Lundman ***@***.***>
Gesendet: Mittwoch, 6. November 2024 10:54
An: openzfsonwindows/openzfs ***@***.***>
Cc: Erwin53 ***@***.***>; Author ***@***.***>
Betreff: Re: [openzfsonwindows/openzfs] Unable to import pool stored in file (Discussion #412)
sending mountpoint: '\??\//./Volume{870bb693-81b9-4988-8c94-72be365cabd2}//backups'
Yeah so it looked it up too soon, and got the Volume name before it settled on D:
Need to address this race somehow. If you import -N then zfs mount one at a time it should work, lemme work on something
-
Reply to this email directly, view it on GitHub<#412 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJTYOUXG6VHQO3C4WS2BV7DZ7HRKVAVCNFSM6AAAAABQ6QO2IGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMJWGQ3TIOA>.
You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>
|
Beta Was this translation helpful? Give feedback.
-
|
OK, now as I have this workaround to have the data sets being mounted, we can come back to the original problem with the hardlinks.
The target of the link is actually pointing to a non-existend file: it should not read I don't know right now if this is actually causing any problems, but at least it is very confusing. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I just noticed, that the directory is also wrong: it should read |
Beta Was this translation helpful? Give feedback.
-
|
OK so looking at the hardlink problem now. Just to clarify, hardlinks don't point to another name, what happens is it creates a new name entry, that references the same file data as an existing file. But what I think you are saying is that hardlinks need to re-link its parent whenever one is accessed. Perhaps? |
Beta Was this translation helpful? Give feedback.
-
|
Oh wait, I see what you are saying: 
When it creates the full path, it only goes to root of that dataset, not the mountpoint, missing |
Beta Was this translation helpful? Give feedback.
-
|
Yep, looks like this is exactly the problem. I actually don't know how Powershell fiddles out that these are actually hard linked files - you very likely know this better than I do. The only thing I can say is that when you do this on an NTFS volume things look right. |
Beta Was this translation helpful? Give feedback.
-
|
OK, sorry for the delay, can you check this zpool.exe/zfs.exe - it is quite verbose, but I hope I have sorted the race with import/mounts. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot, this looks very promising now: importing the sub DSs works like a charm! |
Beta Was this translation helpful? Give feedback.
-
|
OK took a bit, but could you try rc10 and confirm both issues are fixed? |
Beta Was this translation helpful? Give feedback.
-
|
OK. Where can I get RC10 from? |
Beta Was this translation helpful? Give feedback.
-
|
From rc9 area, it’s in the list just not announced |
Beta Was this translation helpful? Give feedback.
-
|
Right. Now automatic mounting works correctly and I can see all files, but the "Target" column in the Powershell Get-ChildItem output is empty now:
|
Beta Was this translation helpful? Give feedback.
-
|
Hmm looks like I broke it a bit |
Beta Was this translation helpful? Give feedback.
-
|
OK so I am about to undo some of the name work here. Turns out, if I create 2 VHD formatted NTFS, and mount the 2nd under D:/backups, any name queried inside (using FileTest.exe) the lower volume will always return relative path, ie, As for Target column in Get-ChildItem, according to chatgpt: So with hardlinks, we should not see a Target column. But we can use: again, note the relative paths. So we need to check where 2 ZFS mounts differ compared to 2 NTFS mounts. |
Beta Was this translation helpful? Give feedback.
-
|
I tried to reproduce this under Ubuntu 24.04: It's interesting that here the |
Beta Was this translation helpful? Give feedback.
-
|
It knows it is a hardlink as the reference count is above 1, ie in stat, st_nlinks. Be interesting to run FileSpy.exe when PS fetches Target, I would guess it uses FileHardlinkInformation and prints that. Or, it stores an EA/xattr with siblings. |
Beta Was this translation helpful? Give feedback.
-
|
OK, thanks for the details. I have downloaded FileSpy.exe meanwhile, but because every access to my drive |
Beta Was this translation helpful? Give feedback.
-
|
Ah I'll check rc10 and roll a new version tomorrow. |
Beta Was this translation helpful? Give feedback.
-
|
Running rc10 here, and I have no issues, including gci on my driveletter? |
Beta Was this translation helpful? Give feedback.
-
|
I sure love how our PS behave quite differently; E: is ntfs. |
Beta Was this translation helpful? Give feedback.
-
|
If we are curious, fsutil uses So query FileHardlinkInformation, which appears to return FileIDs of the parents, which it then opens to get filenames. |
Beta Was this translation helpful? Give feedback.
-
|
Powershell isn't equal to Powershell: which one are you using? The version that comes already installed with Windows (should be some version 5.x) seems to populate the BTW: the target column, if it is populated, is still missing the top-level directory |
Beta Was this translation helpful? Give feedback.
-
|
Can you run filespy when it is grabbing Target, curious if it is using QUERY(FileHardlinkInformation). No worries if you can't. I just run PS that comes with Win11. |
Beta Was this translation helpful? Give feedback.
-
|
So is rc10 still bad for you? |
Beta Was this translation helpful? Give feedback.
-
|
OK so slight amusing issue. I created BOOM, on and the blasted thing put backups on Highly amusing, but I need to fix that. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for the delay, I was off over the weekend. Here is the FileSpy output -- let me know if you need more. In general I'm fine with RC10 except from the fact that the hard link path is still missing the top-level directory ( |
Beta Was this translation helpful? Give feedback.
-
|
I posted a new rc10, but doesn't add anything to this discussion, apart from the mountpath fixes. I still can't get Target to show on my system, but when comparing Where E: is your pool, and F: and F:\backups are VHD ntfs I mounted and setup the same. That E: only shows one is because we currently do not track hardlinks, but will eventually (See discussion regarding ideas in upstream: openzfs#16763 ) |
Beta Was this translation helpful? Give feedback.
-
|
I have to apologize: I'm very sorry about this Please adjust the filename above as appropriate once you have unzipped the attachment. After you have run that you should see the One again: I'm sorry! |
Beta Was this translation helpful? Give feedback.
-
|
Ah ok, that could just mean that format is wrong in general? Or does the output differ with ntfs? |
Beta Was this translation helpful? Give feedback.
-
|
Well, the format is not actually always wrong: with this setting this column is always printed, irrespective of the filesystem type. On FAT32 it is always empty (no hard links supported), on NTFS it seems to be always correct (at least I have never found an error there), and on ZFS it is partially correct (top-level directory is missing). I think this is really an edge case and I don't know whether it makes sense to waste your time on this (or you might want to put it on the low priority list). Data sets are now mounting correctly and hard links seem to be handled properly: the disk usage is computed correctly in Explorer (Properties / Properties) and Thanks a lot! |
Beta Was this translation helpful? Give feedback.
-
I have some problems with hardlinks in pools copied over from Linux: when trying to prepare a simple test case I thought I keep things simple: create a pool under Linux in an ordinary file, export it, copy it over to windows and import it. Unfortunately this didn't work:
...EFI read OK, max partitions 128part 0: offset 800: len e8e04000: tag: 4 name: 'zfs-18ee08797d6a9501'part 8: offset e8e04800: len 4000: tag: b name: ''cannot import 'D:\zfs_windows_test.img': no such pool availableAm I doing something wrong or is this a permanent restriction?
Erwin
Beta Was this translation helpful? Give feedback.
All reactions