[Not a real benchmark] 0.4.17 vs 0.4.18: File transfer over LAN

[schomatis]

Two Ubuntu 16.04 64-bit VMs connected to an isolated internal network with their .ipfs repo mounted on an tmpfs, the "file server" VM does:

ipfs init
ipfs add go-ipfs -r # About 135 MB
ipfs daemon

the "client" does:

ipfs init
ipfs repo gc && time ./ipfs get <go-ipfs-hash> # Repeatedly to get various times.

I'm getting a wall clock time of 8.5±0.2 seconds for both v0.4.17 and v0.4.18. (I've added the files with v0.4.18, didn't bother to recreate the repo with v0.4.17 since both report a version of fs-repo@7.)

This is a very basic test but I wanted to get some kind of quantification on the performance improvements between versions as suggested by @Stebalien, any ideas on variations to the test?

[schomatis]

/cc @Stebalien @magik6k

[Stebalien]

It would be nice to get a sense for many small files versus a single large file. Bandwidth overhead (that is, ask the OS how much traffic each node is actually sending) would also be great.

Note: in these tests, you should probably use:

1. --local (both on add and get). Otherwise, provider records will hide everything.
2. ipfs pin on the server side. It has better parallelism ATM. Otherwise, we just test the fact that we suck at prefetching.

[schomatis]

2. ipfs pin on the server side. It has better parallelism ATM. Otherwise, we just test the fact that we suck at prefetching.

You mean to also run ipfs pin <go-ipfs-hash> after ipfs add? (I thought it pinned by default.)

[Stebalien]

Ah, no. I mean ipfs pin --local instead of ipfs get on the client-side.

[schomatis]

ipfs pin add --local is definitely showing an improvement between versions of around 10-20% less time (I'm having a lot more variability with this command than with ipfs get), I'll try to produce a more systematic test tomorrow to throw some more substantial results.

[Stebalien]

Thanks! (we'll probably see more when we bump the wantlist size but we've been reluctant to do that without fixing sessions first).

[eingenito]

@schomatis where are your running your VMs? And how are you provisioning and driving them? Your timing is pretty stable which is neat. Hannah and I have been trying out different styles of benchmark tests (iptb, driving IpfsNodes in Go, and Jeromy's pure bitswap exchange tests) as we investigate bitswap sessions and they all have their merits but they all test different things.

We were thinking of trying roughly what you did here and maybe using Fargate (or something) to deploy a dozen IPFS containers and point them at each other. I haven't checked on the cost or what we'd use to control them.

I'm curious to try just running a well connected IPFS instance with experimental changes and enough logging output to evaluate them IRL and then just making it do a lot of work over and over. And comparing that to the same workload on a stock IPFS node. Some things like bitswap duplicate blocks would be easy to measure. Transfer speed improvements would be all over the map but maybe over a few days we'd get useful information - especially if they're getting well provided data repetitively.

[schomatis]

where are your running your VMs?

On my local machine, they are just two VMs connected to an internal network (and nothing else) where the nodes find themselves fairly quickly. But your proposed solution is much better, this is just an ad hoc test suggested by Steven to get a coarse sense of some of the performance improvements of the latest release, but I'm doing it manually and is really time consuming so it doesn't scale at all.

and enough logging

I think that is a key point in the analysis process, the kind of black-box testing we're doing at the moment is valuable but has a lot of limitations to really understand what's going on. Personally, just placing a lot of Printfs all over the place debugging ipfs/go-bitswap#8 was very helpful to understand how the code operated (we shouldn't actually use that but it was an indication of the complexity of a code with many threads where a lot is going on).

[schomatis]

I'm going to close this since the tests I'm doing locally are not conclusive enough to assert that any real performance improvement has been achieved, ipfs pin add is too volatile in my tests, the variance intervals between both releases overlap too much to be sure of anything. ipfs get on the other hand is very stable but it takes the same time on both releases. This is talking too much time and (worst) I'm not producing any reproducible systematic test to add to the repo, I just have a bunch of ad-hoc bash scripts, we should invest on a real benchmark framework for any user to be able to reproduce our tests.

[schomatis]

(@Stebalien If you can think of a way to make the ipfs pin add --local command more stable I can try again to at least have some coarse idea of the performance gain of this release.)

[Stebalien]

@schomatis unfortunately, no.

[schomatis]

So, I ended up stabilizing ipfs pin add (fixing my own local problems) to a reasonable 5-10% and while I still can't measure any concrete improvement between versions, that is to a large extent because IPFS is transferring close to the maximum speed of the local virtualized network (to the point that ipfs transferred at over 90% the speed of rsync) for both a large file and a directory with many files, playing with activeWantsLimit and FetchGraphConcurrency yield no significant change. This is good because it means we are not adding much overhead over a basic data transmission but we'll need to tune tests to mimic WAN traffic with delays and drops to have any meaningful idea of how we behave in that scenario (instead of just testing through the internet which may be too random).

I'm curious if other local/virtualized tests like iptb or containers also have this problem of reaching the maximum transfer limit, @eingenito did you experience something like it?

[eingenito]

For our iptb sharness test we actually didn't pay much attention to timing, we were just looking for bitswap duplicate blocks so we didn't get a feel for what was limiting our transfer speeds. I would think that for large blocks bitswap doesn't introduce much overhead (if we ignore duplicates) but for lots of small blocks it might start to get significant.