Transition the coverage-linux64 pipeline to Buildkite

[DilumAluthge]

No description provided.

[DilumAluthge]

I've temporarily disabled the commit statuses. We can re-enable them right before we merge this PR.

In the meantime, go here to view statuses and build logs: https://buildkite.com/julialang/julia-coverage-linux64

[DilumAluthge]

@vtjnash Any idea why the coverage is lower here compared to the Buildbot approach? This PR calculates coverage at 79% (58650/73776), but Codecov (which was uploaded from Buildbot) calculates coverage at 88% (65731/75061).

This PR runs the tests in parallel. Could that be the sole reason for the discrepancy, or is there something else that I'm missing?

[vtjnash]

You should be able to go to codecov or coveralls and compare the two commits directly

[DilumAluthge]

You should be able to go to codecov or coveralls and compare the two commits directly

Ah, we'll have to add a Codecov token to Buildkite first - I'll need @staticfloat's help with that.

[DilumAluthge]

Hmmm, looks like it's failing: https://buildkite.com/julialang/julia-coverage-linux64/builds/68

[staticfloat]

So I have updated this to now correctly build Julia, and I've also updated the buildkite webui settings to not build in reaction to GH events, but to instead build on a nightly schedule.

[staticfloat]

The build is proceeding here: https://buildkite.com/julialang/julia-coverage-linux64/builds/85#8b067d64-5d75-4139-b75a-81289b567f47

[DilumAluthge]

I've also updated the buildkite webui settings to not build in reaction to GH events, but to instead build on a nightly schedule.

Nice! This sounds good.

If I want to e.g. trigger a manual job, based on the latest commit to master, without waiting for the next nightly build, will I still be able to trigger a manual coverage job through the Buildkite web UI?

[DilumAluthge]

┌ Info:
--
  | │   ncores = 1
  | │   Sys.CPU_THREADS = 128
  | └   Threads.nthreads() = 1

We should set ncores to something more useful. 16?

[staticfloat]

will I still be able to trigger a manual coverage job through the Buildkite web UI?

Yep; just click the big green "new build" button in the top right of the pipeline UI.

We should set ncores to something more useful. 16?

Hmmm, yeah. What's the best way to do that? Define JULIA_NUM_THREADS globally, for all jobs? I guess if we really want to run a job single-threaded we can override that, and this allows us to scale our jobs based on the agent itself defining how much parallelism it can sustain.

[DilumAluthge]

Hmmm, yeah. What's the best way to do that? Define JULIA_NUM_THREADS globally, for all jobs? I guess if we really want to run a job single-threaded we can override that, and this allows us to scale our jobs based on the agent itself defining how much parallelism it can sustain.

Sounds good to me. Except I would define both JULIA_NUM_THREADS and JULIA_CPU_THREADS.

[staticfloat]

Looks like it worked!

[DilumAluthge]

Now we just need to uncommon the last line (# Coverage.Codecov.submit_local(fcs)) and make sure the upload works.

[staticfloat]

I rebased onto latest master and the tests broke. I regret everything.

[DilumAluthge]

I restarted the failing Windows Buildbots, and now all the Buildbots are passing.

Also, I ran a new Buildkite coverage-linux64 job, and if you look at the logs (https://buildkite.com/julialang/julia-coverage-linux64/builds/89), everything passed, including both the upload to Coveralls and the upload to Codecov.

It does look like coverage dropped approximately 6%. I'm guessing that is because of the tests being run in parallel.

In a future PR, I'll add a step that runs a subset of the tests in serial (specifically, the subset on which coverage dropped) so we can get coverage back up.

But I don't think that needs to block this PR.

[DilumAluthge]

From my point of view, this PR is good to go now.

[vtjnash]

Why is coverage being miscomputed since this PR was merged? https://codecov.io/gh/JuliaLang/julia/compare/6d2c0a7766142cbeb52c21494484038a558d9b33...9d5f31e9231c1d77e24ee820908e32f559e23057

It seems like the wrong flags must be getting passed to the test runners?

[DilumAluthge]

We may be passing the wrong flags. Here are the relevant scripts:

1. https://github.com/JuliaLang/julia/blob/master/.buildkite/pipelines/scheduled/coverage/coverage_linux64.yml
2. https://github.com/JuliaLang/julia/blob/master/.buildkite/pipelines/scheduled/coverage/run_tests_parallel.jl
3. https://github.com/JuliaLang/julia/blob/master/.buildkite/pipelines/scheduled/coverage/upload_coverage.jl

Also, we're running the tests in parallel. Buildbot used to run them in serial for coverage. So that may be missing some stuff.

@staticfloat Any ideas?

[DilumAluthge]

Hmmm. Here is one possibility.

So, we need to make sure we pass certain command-line flags, right? Including the following:

1. --code-coverage=all
2. --sysimage-native-code=no

Now, when we run the tests in parallel, we are using Base.runtests, which uses Distributed to launch workers. What if the above flags are not getting correctly forwarded to the Distributed workers?

[DilumAluthge]

For example, I don't think Distributed is correctly forwarding the --sysimage-native-code=no flag.

julia> Base.JLOptions().use_sysimage_native_code
0

julia> Distributed.@distributed vcat for i = 1:20
       Base.JLOptions().use_sysimage_native_code
       end
20-element Vector{Int8}:
 1
 1
 1
 1
 1
 1
 1
 1
 1
 1
 1
 1
 1
 1
 1
 1
 1
 1
 1
 1

[vtjnash]

I think test/testenv.jl forwards different flags? Also recommended is --code-coverage=lcov/tracefile-%p.info

c.f. https://github.com/JuliaCI/CoverageBase.jl/blob/master/examples/run_coverage_codecov.sh#L16