Improve beats build and test time

[ruflin]

Building a beat currently takes quite a bit of time because of all the dependencies. The reasons is that Golang rebuilds all packages every time. The -i flag can be used to save the packages under $GOPATH/pkg so they can be later reused.

Here the output of a few example commands I used:

metricbeat$ rm -rf $GOPATH/pkg

$ time go build

real	0m51.015s
user	1m33.506s
sys	0m17.141s

$ time go build

real	0m47.963s
user	1m32.396s
sys	0m15.702s

$ time go build -i

real	0m49.321s
user	1m32.670s
sys	0m15.620s

$ time go build

real	0m5.614s
user	0m4.511s
sys	0m1.426s

$ time go build

real	0m5.526s
user	0m4.437s
sys	0m1.385s

First the pkg directory is removed and then go build is run twice as an example, then once with -i and then again twice with the prebuilt packages. The difference is ~45s which is quite a lot. The -i flag also has quite a bit of an effect when running unit tests.

I was thinking we could add this to our Makefile but the part I worry about is how Golang will invalidate existing package files (or not).

[andrewkroh]

According to the last post here there is logic to determine if the packages need to be rebuilt.

One test would be to modify something in the vendor dir and see if the change is picked up.

[tsg]

That's awesome, on my laptop it reduced the time to compile metricbeat from 70s to 5s. In my test it also correctly detected a change in the vendor/ folder.

I'm +1 to adding it to the makefile and see if we hit any weird conditions from it.

Kind of surprised we didn't see this before, but the docs are kind of misleading on it:

The -i flag installs the packages that are dependencies of the target.

[ruflin]

I was also confused by the docs. I stumbled over it because I wanted to know why go test . sometimes takes very long and started to play with the -x flag. Then I discovered the -i for tests and so it also for builds. The problem with tests is that you need to run go test -i . and then go test . as if -i is used, tests are not run. Looking at the combined time of the two commands without having pkg it is almost identical to only run go test .. In case the packages are already build, go test -i is very quick. So I'm thinking if also here we should add the flag.

Will open a PR for further tests / discussions.

[ruflin]

Closing this as the -i flag was added. In Go 1.10 it actually is not needed anymore as new detection mechanisms are in place.