Implement a receive / send buffer on zfs send | zfs receive

[RubenKelevra]

I found this article, which shows an issue on zfs send | zfs receive.

http://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/

This is especially an issue when receiving over a slow network, like dsl. First at all, when the disks are able to receive they got to less data and when they stall no data is transferred at all.

Secondly I use a tcp session to transfer the data over ssh, so I think the issue is increased because the tcp is constantly running it's slow start to avoid congestion, with bursty data transfers the bandwidth ist further wasted.

With mbuffer with 1 GB size on both sending systems and 2x 1 GB on receiving side I reduced my backup-time massively because my dsl is now constantly filled up with data.

So it would be nice if it's possible to implement a similar function in zfs send | zfs receive. A progress graph would be also nice, as optional feature.

[kernelOfTruth]

referencing: #1161 Implement zfs recv buffer ,
#1112 ZFS send dedup multi-thread

ryao@3530cf2 Increase Linux pipe buffer size on zfs recv to the maximum system size
(in-tree 0.6.4*)

in addition to behlendorf's comment (#1112 (comment)) Illumos 5694 - traverse_prefetcher does not prefetch enough

there were also changes to prefetch itself: 7f60329 Illumos 5987 - zfs prefetch code needs work

options you have for buffering (mostly on the receiving side appears to be advantageous, mentioned in the issues):

lz4, lz4hc
pv
mbuffer
...

[ryao]

@RubenKelevra As @kernelOfTruth mentioned, an increased buffer size has already been implemented in zfs recv. It just is not configurable via the commandline tool and instead opts to use the value of /proc/sys/fs/pipe-max-size.

5c3f61e

I had only touched zfs recv after observing that zfs send blocked on zfs recv in my tests. We likely could give zfs send the same treatment though.

That being said, I would prefer not to duplicate the functionality of mbuffer. We likely would not have adequate test coverage for that functionality, such that it would be easier for regressions to occur versus the status quo of doing a Linux-specific fcntl() to increase the pipe size to the maximum supported by the system and letting mbuffer be used in network situations were additional buffering is helpful when something more is needed.

[RubenKelevra]

I'm still importing from one machine while importing several hundert snapshots as diff (via send -I a@1 a@800 ). On the second import the import just stall several dozend seconds, than he imports the full buffer and stall again after some seconds ...

The 6 disks are only avg 6% busy... the z_wr_iss processes seem to block here ... what are the purpose of these processes?

[ryao]

They issue write IOs to disk.

[RubenKelevra]

I use dedup and gzip-9 compression, but sometimes the receive stalls completely and use 3 of 4 cores completely without issuing much IOs to disk. How may I debug this for you?

[RubenKelevra]

My issue seem to be the same at this:

https://serverfault.com/questions/507587/zfs-send-i-receive-stalling

[CMCDragonkai]

So is mbuffer still needed at all with all the latest changes to send and receive?

[DeHackEd]

mbuffer can still help dramatically as the default buffer size on Linux is still on the order of kilobytes and the receive process can read intermittently and in bursts as it flushes data and reads needed metadata while it runs.

The question is, is this feature worthwhile enough to make it part of send/recv directly? I gave my opinion in the other issues.

[CMCDragonkai]

In that case, does it help only on the receive side or on both send and receive?

[DeHackEd]

While it helps either side, if you could only have one side implement the buffer or have the larger buffer then the right choice is the receiver.

[tuxoko]

If you use current master. It does use bqueue as send/recv buffer. Though the size seems to be hardcoded, so you need to recompile if you want to adjust it.

[mailinglists35]

I use dedup and gzip-9 compression

switch to lz4. I had an old dataset using gz9 and I went crazy trying to find out why is my cpu burning on receive. switched to lz4 and transfer went back to expected speed.

and I hope you have a very particular case to justify dedup.

[dioni21]

Old open issue. Let me leave my opinion:

I think zfs send/recv should have a minimum buffer. Enough for most critical operations, but not more than that. Maybe something like block size (128k?) times number of simultaneous physical I/O permitted on the pool in question. Maybe, also, this same sized buffer on both sides, send and recv.

Obviously more buffer may mean more performance, but Unix Philosophy is to create small programs to do small things, and pipe them through. So, IFF the user wants, let the user add mbuffer or whatever to his pipe.