Create a makefile target that checks number of races in e2e tests

[JekaMas]

Problem

From time to time we found new data races in our code or in Ethereum. However, we haven't any process to clean up the code regularly.
We're migrating to Geth 1.8 and we don't know is it safe for us.

E.g we found a race in les.downloader and after it was fixed one more was raised #709 (comment)

Implementation

Make a race-check script that runs tests with -race -v flag and counts the number of discovered races. If that number is higher than expected number of races, it should fail. If less or equal — it should still succeed.

Acceptance Criteria

1. make race-check is added to a Makefile
2. The script runs e2e tests with -race -v flag with a customizable networkID
3. There is a setting somewhere with the number of accepted (or known) races

Future steps

Make a nightly run on Jenkins for the race check.

[mandrigin]

One more idea about how to make it slightly less strict, so we don't have to fix all the data races before making this go green.
Every nightly should fail if the number of races is more than in the previous nightly. Then we'll be able to start dirty, and then fix it one by one.

[JekaMas]

Totally agree! This is the way to have a good CI one day.
We can control new data races and clean up old ones in a few iterations.

[pedropombeiro]

In TeamCity this is easy to implement. You can tell it to watch a build metric and fail the build if that metric increases. I was using that to watch for the number of build warnings (it would fail if a new warning appeared). The best thing about that system is that it allows the metric to go down, but then the new number becomes the new limit, and you don't need to manage it.

[mandrigin]

@pombeirp exactly!
I guess we can do that in Jenkins as well, but that requires access to our internal resources.
If we want to have it as a bounty we need a platform-independent solution.
If anyone from the core team is interested in taking over this issue, we can remove the bounty tag from it.

[pedropombeiro]

I wouldn't mind taking this on. I'm getting sick of hearing myself say "it's easy with TeamCity, not sure about Jenkins", so might as well take advantage of something like this to get my hands dirty with Jenkins.