zfs list slow

[leelists]

Hi,

the command zfs list -t snapshot -r pool/xxxx is very very slow.
i have approx 1200 snapshot under pool/xxxx the command take's approx 20 minutes.

Any clues ?

Thanks Lee

[behlendorf]

That's a lot of snapshots, but 20 minutes does seem unreasonably long. Can you try running top during the zfs list and see if any of the zfs threads are cpu bound?

[leelists]

It seem's that no zfs process is cpu bound, you will find an extract of iostat -kx 5

avg-cpu: %user %nice %system %iowait %steal %idle
0,22 0,00 5,18 0,00 0,00 94,60

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sde 6,60 1,20 79,80 59,20 5392,00 232,00 80,92 1,06 7,61 2,21 30,72
sdf 3,60 2,40 93,80 48,60 5395,00 217,60 78,83 1,07 7,53 1,94 27,68

can it have anything to do with this bug : http://wesunsolve.net/bugid/id/6755389 ?

[behlendorf]

It sounds like it may be caused by a similar issue. However, the Linux port does already contain the fixes referenced in that issue. We'll need to dig in to it to determine exactly what's causing the slow down.

[behlendorf]

Pawel Jakub Dawidek of the FreeBSD port took a look at this same issue and came up with the following patch. It still needs to be reviewed and ported to Linux but it looks very promising.

http://people.freebsd.org/~pjd/patches/zfs_list_name.2.patch

[leelists]

Effectively it look's promising, keep on your good work

[Rudd-O]

Looks good to me too.

[Rudd-O]

My zfs list processes are IO bound. They do this a few times per second while they appear to be reading data (almost a megabyte every few seconds) from the disk according to atop:

ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0
ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0
ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0
ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0
ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0
ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0
ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0
ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0
ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0
ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0
ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0
ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0
ioctl(6, 0x5a15, 0x7fffd7e20c80) = 0

[behlendorf]

That I/O would likely be ZFS prefetching the additional list data. It should be reasonably straight forward to port Pawel's patch to disable this behavior if you want to try that.

[behlendorf]

@Rudd-O Please try this patch and see if it resolves you issue. It's a port of Pawel's FreeBSD change for ZoL, note it doesn't impact the default behavior of zfs list. To see the speed up you need to request just the names zfs list -t snapshot -o name -s name. If you don't all the properties will be fetched as well and this is the major slowdown.

[behlendorf]

@dajhorn You might want to look at this as well for zfs-auto-snapshot. If simply the list of snapshot names is enough than this should considerably speed things up. The change is safe enough so assuming it tests well for other it could be merged in soon.

[dajhorn]

@behlendorf The zfs-auto-snapshot for ZoL does not call zfs list in a way that satisfies the test at line 2,822 for the new ZFS_ITER_SIMPLE flag. The code needs some finesse before it will benefit from this optimization.

[timbrody]

Some more background ... two identical spec. systems:

$ time sudo zfs list -t snapshot | wc
    142     142    7375

real    0m6.451s
user    0m0.036s
sys 0m0.408s

# time sudo zfs list -t snapshot | wc
    148     740   14217

real    0m0.253s
user    0m0.012s
sys 0m0.188s

The first system has a lot more data in its snapshots. The first system is ~8% CPU to 'zfs' and ~1% in SYS (watching top).

gdb --args zfs list -t snapshot
) run
CTRL + C
) bt

#0  0x00007ffff6f8f747 in ioctl () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff747b4da in ?? () from /lib/libzfs.so.1
#2  0x00007ffff747f222 in zfs_iter_snapshots () from /lib/libzfs.so.1
#3  0x00000000004047d0 in ?? ()
#4  0x00007ffff747f0a0 in zfs_iter_filesystems () from /lib/libzfs.so.1
#5  0x00000000004047e8 in ?? ()
#6  0x00007ffff747f0a0 in zfs_iter_filesystems () from /lib/libzfs.so.1
#7  0x00000000004047e8 in ?? ()
#8  0x00007ffff747b099 in zfs_iter_root () from /lib/libzfs.so.1
...

NB version 31 refers to "support for improved 'zfs list' performance.":
http://hub.opensolaris.org/bin/view/Community+Group+zfs/31

[Rudd-O]

The patch won'twork for me because I need at least one property (namely the snapshot date which I use to sort the sync ops).

BTW github.com/Rudd-O/zfs-tools check the zreplicate command. I am so proud of it.

[leelists]

Thanks a lot, the patch really helps

Now zfs list -t snapshot -o name -r pool/xxxx returns in severals seconds,

Great job

[byteharmony]

Thanks for the above information, it really helped improve my script speeds. Getting volume snapshot information went from 7 minutes (for example on a small system with 2,600 snapshots) down to 30 seconds. I think it's important to note, however, that this is not a ZoL (ZFS on Linux) issue, rather it is a ZFS issue. I see this same snapshot list speed on Solaris ZFS systems. When there is a large number of snapshots (over 1000 on a pool) scripts managing the snaps take longer and average system output is significantly reduced.

BK

[behlendorf]

I'd be curious to hear how your original scripts perform on the latest Solaris. They claim significant speed ups here in zpool version 31. But I've never seen any hard data.

[byteharmony]

Sorry if I mislead you, perhaps I should have been more clear that the Solaris machine was a Nexenta system. So it's also running version 28.

BK

[ryao]

@byteharmony You could ask @mmatuska what the upstream status of that patch is. He tends to do the work to get FreeBSD improvements to ZFS sent back to Illumos, which would be needed before it goes into Nexenta.