centos7 - selinux must be disabled

[dd1dd1]

Hello. I am looking at an impossible bug - on centos7.2 with selinux enabled (the default setting), ZFS cannot delete files. The "rm" command makes files disappear, but free space as reported by "df" does not go down, as if the files become "hidden" (but not hidden from "zdb -dddd", where the deleted files are listed all right, with "links" equal to 0. Rebooting with selinux disabled, makes the deleted files go away, "df" becomes good again. Rebooting with selinux enabled makes the problem return.

I see this on 3 different machines with CentOS-7.2 (and it is good I caught this before they went into production!):
[root@grifstore0 grifstore0]# uname -a
Linux grifstore0.triumf.ca 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@grifstore0 grifstore0]# rpm -q zfs
zfs-0.6.5.7-1.el7.centos.x86_64
[root@grifstore0 grifstore0]#

To reproduce this problem on vanilla CentOS 7.2:

[root@grifstore0 ~]# cd /pool0/grifstore0/
[root@grifstore0 grifstore0]# dd if=/dev/zero of=test4 bs=1024k
35116810240 bytes (35 GB) copied, 26.9898 s, 1.3 GB/s
[root@grifstore0 grifstore0]# df -kl
Filesystem         1K-blocks     Used   Available Use% Mounted on
pool0            13074746880        0 13074746880   0% /pool0
pool0/grifstore0 13108988544 34241664 13074746880   1% /pool0/grifstore0
[root@grifstore0 grifstore0]# 
[root@grifstore0 grifstore0]# rm test4 
rm: remove regular file ‘test4’? y
[root@grifstore0 grifstore0]# ls -la
total 1
drwxr-xr-x. 2 root root 2 Jul 12 08:30 .
drwxr-xr-x. 3 root root 3 Jul 12 08:15 ..
[root@grifstore0 grifstore0]# df -kl
Filesystem         1K-blocks     Used   Available Use% Mounted on
pool0            13074746880        0 13074746880   0% /pool0
pool0/grifstore0 13108988544 34241664 13074746880   1% /pool0/grifstore0
[root@grifstore0 grifstore0]# zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
pool0             32.7G  12.2T  40.4K  /pool0
pool0/grifstore0  32.7G  12.2T  32.7G  /pool0/grifstore0
[root@grifstore0 grifstore0]# 

At this point, observe how /pool0/grifstore0 has no files, as reported by "ls -la", but "df" is not zero and "zfs list" shows "REFER 32.7G".

Change /etc/selinux/config SELINUX=enforcing to SELINUX=disabled, reboot and see everything goes back to normal: both "df" and "zfs list" show free space is now correct ("used" is 0).

[root@grifstore0 ~]# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
pool0 13108988416 0 13108988416 0% /pool0
pool0/grifstore0 13108988416 0 13108988416 0% /pool0/grifstore0
[root@grifstore0 ~]#
[root@grifstore0 ~]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
pool0 13.4M 12.2T 40.4K /pool0
pool0/grifstore0 40.4K 12.2T 40.4K /pool0/grifstore0
[root@grifstore0 ~]#

Reboot with selinux enabled, and the problem returns.
K.O.

[dweeezil]

@dd1dd1 I presume you don't have xattr=sa? Sounds like you're running into #1548 most likely.

[tuxoko]

This will be fixed when #4827 #4838 are merged.

[dd1dd1]

@dweeezil I know nothing about "xattr=sa". I do not use extended attributes. I create a 7TB file on an 8TB zfs volume using "dd", delete it using "rm", observe 7TB of space is used, 1TB free; hence "rm" is broken as I reported.

ZFS instructions for RHEL/CentOS say nothing about "must use xattr=sa", i.e. https://github.com/zfsonlinux/pkg-zfs/wiki/HOWTO-install-EL7-(CentOS-RHEL)-to-a-Native-ZFS-Root-Filesystem or https://github.com/zfsonlinux/zfs/wiki/RHEL-%26-CentOS.

This bug is a showstopper for ZFS on RHEL/CentOS 7 (where selinux is enabled by default) and until it is fixed, the appropriate instructions should say "must disable selinux" or "must do xattr=sa" or whatever is the best workaround is deemed by zfs developers.

For my needs, "disable selinux" is good enough.
K.O.

[svennd]

I think I have seen similar issue; an now empty subvolume (zfs create /volume/part) was empty (ls -la) but was still taking ~40G, after an unmount (umount /volume/part) that took some time, the volume is now only taking 42Mb. (selinux is still on) which arguably is still allot, but not like 40G's. I was not using any extended file attributes either.

uname -r
3.10.0-327.18.2.el7.x86_64
rpm -q zfs
zfs-0.6.5.7-1.el7.centos.x86_64

[dd1dd1]

@svennd I confirm that in my case, unmount and reboot (reboot does same thing as unmount, yes?) do not free up the lost space. (unless selinux is disabled). K.O.

[dd1dd1]

FWIW, if the lost space is inside a zfs subvolume (zfs create pool/vol; cat /dev/zero > /pool/vol/file; rm /pool/vol/file), deleting the subvolume does recover the lost space (zfs destroy pool/vol). (with selinux enabled). A rather drastic way to delete 1 file...

I suspect that is why some use cases do not see this problem: my main zfs test machine uses snapshots and deleting of snapshots probably works the same as deleting a subvolume and does not leak free space.

But of course after you create and then try to delete a 7TB file, you cannot miss the problem.
K.O.

[behlendorf]

@dd1dd1 I've verified that this issue is now addressed in master by commits ddae16a and 31b6111. We'll get these cherry picked back for 0.6.5.8.

[virtuallynathan]

I hit this same bug on Fedora23, same version of ZFS. After unmount (which took ages) I went from ZFS showing 15.6T used to 4.77T used, which is exactly what 'du -sh' shows. I have unmounted and mounted, and disabled SELinux. All seems OK now.

[behlendorf]

These fixes have been applied to the 0.6.5.8 release.

[virtuallynathan]

Thanks for the update/fix!
When do you think a Fedora RPM will be pushed to the Repo? Looking forward to upgrading.

[behlendorf]

Should be this week, maybe today.

[virtuallynathan]

Awesome, thanks!