samba intermittend hangs when /var/cache/samba is on zfs dataset

[GregorKopka]

After i put /var/cache/samba on a zfs dataset the samba server processes (both smbd and nmbd) intermittendly DEFUNC.

Accoding to strace smbd stalls on syscalls to:
fdatasync(16)
fdatasync(17)
fcntl(17, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=364, len=1})
fcntl(16, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=8, len=1})

The moment i put /var/cache/samba back on the ext2 rootfs or a tmpfs the effect vanishes.

Issue is with spl/zfs @ current master (but experienced also with version as of 2 weekends ago), pool version 28.
Linux version 3.0.6-gentoo (root@backend) (gcc version 4.5.3 (Gentoo 4.5.3-r1 p1.0, pie-0.4.5) ) #3 SMP Sat Apr 14 18:02:30 CEST 2012

Dataset used for testing this was freshly created with:

$ zfs create data/system/var/cache/samba
$ zfs get all data/system/var/cache/samba
NAME PROPERTY VALUE SOURCE
data/system/var/cache/samba type filesystem -
data/system/var/cache/samba creation Fri Apr 27 2:26 2012 -
data/system/var/cache/samba used 424K -
data/system/var/cache/samba available 559G -
data/system/var/cache/samba referenced 424K -
data/system/var/cache/samba compressratio 1.00x -
data/system/var/cache/samba mounted yes -
data/system/var/cache/samba quota none default
data/system/var/cache/samba reservation none default
data/system/var/cache/samba recordsize 128K default
data/system/var/cache/samba mountpoint /var/cache/samba inherited from data/system
data/system/var/cache/samba sharenfs off default
data/system/var/cache/samba checksum on default
data/system/var/cache/samba compression off default
data/system/var/cache/samba atime off inherited from data
data/system/var/cache/samba devices on default
data/system/var/cache/samba exec on default
data/system/var/cache/samba setuid on default
data/system/var/cache/samba readonly off default
data/system/var/cache/samba zoned off default
data/system/var/cache/samba snapdir hidden default
data/system/var/cache/samba aclinherit restricted default
data/system/var/cache/samba canmount on default
data/system/var/cache/samba xattr on default
data/system/var/cache/samba copies 1 default
data/system/var/cache/samba version 5 -
data/system/var/cache/samba utf8only off -
data/system/var/cache/samba normalization none -
data/system/var/cache/samba casesensitivity sensitive -
data/system/var/cache/samba vscan off default
data/system/var/cache/samba nbmand off default
data/system/var/cache/samba sharesmb off default
data/system/var/cache/samba refquota none default
data/system/var/cache/samba refreservation none default
data/system/var/cache/samba primarycache all default
data/system/var/cache/samba secondarycache all default
data/system/var/cache/samba usedbysnapshots 0 -
data/system/var/cache/samba usedbydataset 424K -
data/system/var/cache/samba usedbychildren 0 -
data/system/var/cache/samba usedbyrefreservation 0 -
data/system/var/cache/samba logbias latency default
data/system/var/cache/samba dedup off default
data/system/var/cache/samba mlslabel none default
data/system/var/cache/samba sync standard default
data/system/var/cache/samba refcompressratio 1.00x -

Related datasets:

$ zfs get mountpoint,canmount data/system data/system/var data/system/var/cache
NAME PROPERTY VALUE SOURCE
data/system mountpoint / local
data/system canmount off local
data/system/var mountpoint /var inherited from data/system
data/system/var canmount off local
data/system/var/cache mountpoint /var/cache inherited from data/system
data/system/var/cache canmount off local

[GregorKopka]

Same problem surfaces when samba accesses /var/lib/samba/private/schannel_store.tdb
Hangs several seconds on these msync calls with MS_SYNC, i guess they're from the tdb library:

fdatasync(129) = 0
msync(0x7f153e80f000, 4132, MS_SYNC) = 0
fdatasync(129) = 0
msync(0x7f153e80f000, 24, MS_SYNC) = 0
fdatasync(129) = 0
msync(0x7f153e80d000, 16384, MS_SYNC) = 0
utime("/var/lib/samba/private/schannel_store.tdb", NULL) = 0
fdatasync(129) = 0
msync(0x7f153e80f000, 24, MS_SYNC) = 0

Fun parts are:

• only one W7 client is experiencing this
• setting sync=disabled and remounting the zfs filesystem dosn't fix this
• moving the file somewhere to a non-zfs filesystem vanishes the hangs

So it looks like ZFS really syncs to disk and the msync call hangs while waiting for the txg to complete.