Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"zfs diff" fails with "Unable to determine path or stats for object ...: File exists" on encrypted filesystems #8931

Closed
chungy opened this issue Jun 19, 2019 · 6 comments · Fixed by #9343
Closed

"zfs diff" fails with "Unable to determine path or stats for object ...: File exists" on encrypted filesystems #8931

chungy opened this issue Jun 19, 2019 · 6 comments · Fixed by #9343
Labels
Component: Encryption "native encryption" feature Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@chungy
Copy link
Contributor

chungy commented Jun 19, 2019

System information

Type Version/Name
Distribution Name Arch
Distribution Version N/A
Linux Kernel 5.1.11-arch1-1-ARCH
Architecture x86_64
ZFS Version 0.8.1
SPL Version 0.8.1

Describe the problem you're observing

When running against an encrypted dataset, the zfs diff command fails to complete its task, outputting a message at the end such as:
Unable to determine path or stats for object $objid in $dataset: File exists

Describe how to reproduce the problem

Have an encrypted dataset, key loaded and mounted, with a snapshot and with data changed after the snapshot, run the zfs diff command against the snapshot.

Example:

# zfs create -o mountpoint=/mnt rpool/temp
# cd /mnt/
# echo hello > alice
# zfs snapshot rpool/temp@alice
# echo new > bob
# zfs diff rpool/temp@alice
M	/mnt/
+	/mnt/bob
# zfs create -o encryption=on -o keyformat=passphrase rpool/temp/encr 
Enter passphrase: 
Re-enter passphrase: 
# cd encr/
# echo hello > alice
# zfs snap rpool/temp/encr@alice
# echo new > bob
# zfs diff rpool/temp/encr@alice
Unable to determine path or stats for object 3 in rpool/temp/encr@alice: File exists
@behlendorf behlendorf added Component: Encryption "native encryption" feature Type: Defect Incorrect behavior (e.g. crash, hang) labels Jun 21, 2019
@behlendorf
Copy link
Contributor

@chungy I wasn't able to reproduce this issue using 0.8.1 on Fedora 30 using the reproducer you provided. Are you able to reproduce this consistently on Arch?

$ zfs diff tank/temp/encr@alice
M	/tank/temp/encr/
+	/tank/temp/encr/bob
+	/tank/temp/encr/bob/<xattrdir>
+	/tank/temp/encr/bob/<xattrdir>/security.selinux

@chungy
Copy link
Contributor Author

chungy commented Jun 21, 2019

Yes, it happens consistently on Arch.

@youzhongyang
Copy link
Contributor

While playing with zfs on Clear Linux, I tried the reproduction steps. Not reproduced.

root@clearlinux-dev/mnt/encr # zfs diff tank/temp/encr@alice
+       /mnt/encr/bob
M       /mnt/encr/
root@clearlinux-dev/mnt/encr # zfs version
zfs-0.8.1-1
zfs-kmod-0.8.1-1
root@clearlinux-dev/mnt/encr # uname -a
Linux clearlinux-dev 5.1.12-785.native #1 SMP Wed Jun 19 07:09:10 UTC 2019 x86_64 GNU/Linux

@sempervictus
Copy link
Contributor

Any chance this is another side effect of the slow path use without SIMD? 5.1.1 has the GPL_ONLY FPU export thing going...
There have also been a bunch of changes in Arch userspace recently, and we are now on GCC 9.1, so might be worth rebuilding the userspace side of things. (We use an in-house fork of Arch for servers and workstations, and have successful builds of our 0.7.13 zfs-integrated kernels on GCC 9.1 in testing now. 0.7 because reading the issue tracker makes 0.8 seem untenably dangerous for prod use).

@HankB
Copy link

HankB commented Sep 17, 2019 •
edited
Loading

I believe I have been able to reproduce this on Debian with 0.8.1 installed from the Github repo. It seems to be related to the number of files in a snapshot. I discovered this after installing gnome-core on a relatively bare system and could not view a diff of the files changed (~65K lines in the output on an unencrypted install.) I can recreate using the following commands

root@mlbry03:/home/hbarta# zfs create rpool/home/hbarta/test
root@mlbry03:/home/hbarta# chown hbarta.hbarta test
root@mlbry03:/home/hbarta# zfs snap rpool/home/hbarta/test@create
hbarta@mlbry03:~/test$ for i in {1..100}; do for j in {1..100}; do echo $i $j > ${i}.${j}.txt; done; done
hbarta@mlbry03:~/test$ find .|wc -l
10001
hbarta@mlbry03:~/test$ 
root@mlbry03:/home/hbarta# zfs diff rpool/home/hbarta/test@create|wc -l
1
Unable to determine path or stats for object 3 in rpool/home/hbarta/test@zfs-diff-14831-00000001000ed5dd: File exists
root@mlbry03:/home/hbarta#

This is on a Lenovo Y50 (I7 processor, 16GB RAM and 120GB cheap SATA SSD.) Install is per instructions at https://github.com/zfsonlinux/zfs/wiki/Debian-Buster-Encrypted-Root-on-ZFS, scripted using https://github.com/HankB/Linux_ZFS_Root/blob/master/Debian/install_Debian_to_ZFS_root.sh

Edit.0: I was able to reproduce this on a Virtualbox VM as well.
Versions:

Linux mlbry03 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2 (2019-08-28) x86_64 GNU/Linux
hbarta@mlbry03:~/test$ /sbin/modinfo zfs | grep version
version:        0.8.1-4
srcversion:     FA9BDA7077DD9A40222C4B8
vermagic:       4.19.0-6-amd64 SMP mod_unload modversions 
hbarta@mlbry03:~/test$ 
```

@loli10K loli10K mentioned this issue Sep 22, 2019
Merged
12 tasks
behlendorf pushed a commit that referenced this issue Sep 24, 2019
@loli10K @behlendorf
diff_cb() does not handle large dnodes
d359e99 
Trying to 'zfs diff' a snapshot with large dnodes will incorrectly try
to access its interior slots when dnodesize > sizeof(dnode_phys_t).
This is normally not an issue because the interior slots are
zero-filled, which report_dnode() handles calling
report_free_dnode_range(). However this is not the case for encrypted
large dnodes or filesystem using many SA based xattrs where the extra
data past the legacy dnode size boundary is interpreted as a
dnode_phys_t.

Reviewed-by: Brian Behlendorf <[email protected]>
Reviewed-by: Tom Caputi <[email protected]>
Reviewed-by: Ryan Moeller <[email protected]>
Signed-off-by: loli10K <[email protected]>
Closes #7678 
Closes #8931 
Closes #9343
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Dec 24, 2019
@loli10K @tonyhutter
diff_cb() does not handle large dnodes
e2963a7 
Trying to 'zfs diff' a snapshot with large dnodes will incorrectly try
to access its interior slots when dnodesize > sizeof(dnode_phys_t).
This is normally not an issue because the interior slots are
zero-filled, which report_dnode() handles calling
report_free_dnode_range(). However this is not the case for encrypted
large dnodes or filesystem using many SA based xattrs where the extra
data past the legacy dnode size boundary is interpreted as a
dnode_phys_t.

Reviewed-by: Brian Behlendorf <[email protected]>
Reviewed-by: Tom Caputi <[email protected]>
Reviewed-by: Ryan Moeller <[email protected]>
Signed-off-by: loli10K <[email protected]>
Closes openzfs#7678
Closes openzfs#8931
Closes openzfs#9343
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Dec 27, 2019
@loli10K @tonyhutter
diff_cb() does not handle large dnodes
484a9d4 
Trying to 'zfs diff' a snapshot with large dnodes will incorrectly try
to access its interior slots when dnodesize > sizeof(dnode_phys_t).
This is normally not an issue because the interior slots are
zero-filled, which report_dnode() handles calling
report_free_dnode_range(). However this is not the case for encrypted
large dnodes or filesystem using many SA based xattrs where the extra
data past the legacy dnode size boundary is interpreted as a
dnode_phys_t.

Reviewed-by: Brian Behlendorf <[email protected]>
Reviewed-by: Tom Caputi <[email protected]>
Reviewed-by: Ryan Moeller <[email protected]>
Signed-off-by: loli10K <[email protected]>
Closes openzfs#7678
Closes openzfs#8931
Closes openzfs#9343
tonyhutter pushed a commit that referenced this issue Jan 23, 2020
@loli10K @tonyhutter
diff_cb() does not handle large dnodes
ec5d76e 
Trying to 'zfs diff' a snapshot with large dnodes will incorrectly try
to access its interior slots when dnodesize > sizeof(dnode_phys_t).
This is normally not an issue because the interior slots are
zero-filled, which report_dnode() handles calling
report_free_dnode_range(). However this is not the case for encrypted
large dnodes or filesystem using many SA based xattrs where the extra
data past the legacy dnode size boundary is interpreted as a
dnode_phys_t.

Reviewed-by: Brian Behlendorf <[email protected]>
Reviewed-by: Tom Caputi <[email protected]>
Reviewed-by: Ryan Moeller <[email protected]>
Signed-off-by: loli10K <[email protected]>
Closes #7678
Closes #8931
Closes #9343
@bew-dev
Copy link

bew-dev commented Nov 5, 2022 •
edited
Loading

This is happening to me still

$ zfs diff nas/live/rsnapshot@merged-san-2021-0930 nas/live/rsnapshot@rsnap-weekly-2021-1001 
M       /nas/live/rsnapshot/
Unable to determine path or stats for object 65926 in nas/live/rsnapshot@merged-san-2021-0930: No such file or directory

Its not encrypted (that i know of?), sorry just reread the title. not sure how do dig deeper here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: Encryption "native encryption" feature Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

diff_cb() does not handle encrypted large dnodes loli10K/zfs
6 participants
@behlendorf @chungy @sempervictus @HankB @youzhongyang @bew-dev