compiletest: Default to one CGU when compiling tests.

[michaelwoerister]

@aidanhs, maybe you want to give this a try. It reduced the execution time for run-pass from 8 min 30s to 7 min 5s on my machine (though that was just one test run, so ymmv).

r? @aidanhs

[kennytm]

So I've compare the timing of this PR against the master branch and two other unrelated PRs. It doesn't seem that there's any advantage on Travis CI's docker, in fact it becomes slower. Not sure about the performance on Windows and macOS.

Raw data

Table obtained by running grep -o 'travis_time:end.*,duration=\d*' log.txt | sed -E 's/travis_time:end:([^:]+):.*,duration=/\1,/g' on the raw log.

	This PR	Base	Latest Master	PR47676	PR47795
stage0-std	78592639117	56126925920	61407951207	55773335390	88508007006
stage0-tidy	8069706267	5978188129	6523924392	6080932434	9231249361
tidy	88214124708	63616532470	69524967093	63264049484	99605234896
014a3822	89530948137	64875897879	70771498343	64589584858	101343017616
stage0-std	506807704	512148301	484436586	531335308	729745946
stage0-test	14178504457	10060915165	11284057357	10100718083	15421561879
stage0-rustc	1223336058424	895037350470	986496881436	894181745112	1362614163094
stage1-std	78394790050	51527984773	54549372093	49972678850	86937304360
stage1-test	9100705345	6650431669	7534905153	6592962854	9916295171
stage1-rustc	850325923532	651194815969	705098789463	639660365412	926112982859
stage2-rustdoc	242769346157	167273952221	183338663283	164314134840	260825835448
stage0-std	602083657	531937000	490588091	544782368	612866768
stage0-test	550248986	519285731	479952038	525511956	588994832
stage0-rustc	588016315	582316279	506971373	542125061	635143822
stage1-std	400062965	369476021	344442212	367040959	439905089
stage1-test	338370964	283825458	269536938	279705159	351172083
stage1-rustc	362387135	328761079	304789214	319063830	391863102
stage0-unstable-book-gen	9389909121	7350250926	7509127580	7457715700	9711786239
stage0-rustbook	194348209994	149244685365	156009618060	138917964783	206108478026
stage2-rustdoc	333367335	310393875	312372563	318839760	396061512
stage2-error_index_generator	6171241870	5455658844	4975500525	5390227134	6946201285
15a60314	2698130648165	1991547690308	2167733796627	1962625482740	2961879669122
stage0-std	538180870	553098090	494893398	540110862	559174375
stage0-tidy	1224998795	1152724207	1070250843	1153611751	1430043736
tidy	3289675978	3238393612	3172910771	3197255004	3641671513
stage0-test	495509436	538088999	495193128	558052594	583935524
stage0-rustc	534236155	562696420	525732146	580408644	589073422
stage1-std	379416863	386725344	354509989	410555581	412717342
stage1-test	300324550	388024562	268296320	305316634	349206260
stage1-rustc	350877033	321157530	313335068	345958421	388472157
build_test_helpers	39789564	36991921	38014134	44542199	45504121
stage0-compiletest	29232683960	21877335571	24386912949	21368235292	32966693304
test_ui	107067559851	85383610259	88801604115	81974738153	113915238131
test_run-pass	1256824876344	913756750439	1000276153687	909643638182	1307968806619
test_compile-fail	260608831800	219306271300	229404059545	219981497561	269970510702
test_parse-fail	4707254193	4582241847	4689482025	4822771735	5068159889
test_run-fail	33324577975	23023404433	26479573516	24310247000	35278977061
test_run-pass-valgrind	5107106619	3749214697	4188096633	3965229917	5488670634
test_mir-opt	28188145650	21191323568	23672992514	22318049377	28848061280
test_codegen	11646297495	7841187407	9127954896	8207767344	12081372061
test_codegen-units	10615863799	6962416967	8076827466	7207675521	10810972321
test_incremental	80116563626	51663715428	57878955176	54236546509	80414518509
test_debuginfo	28426516268	20814330120	23550002490	21860711067	30076909585
test_ui-fulldeps	18780453311	12810786528	14990392128	13301460742	19812676949
test_run-pass-fulldeps	326221215724	213864880964	257864551246	222446473076	369155410000
test_run-fail-fulldeps	3599633420	2311215595	2478712927	2355109682	3459625347
test_compile-fail-fulldeps	54603012770	37106465415	43278212469	38511424617	56036911355
test_incremental-fulldeps	6481307540	4598562186	4891061340	4713257616	6992113421
stage2-rustdoc	346880093	294330006	314503412	300202641	431475874
test_run-make	131226856318	88866500950	102102698301	92779566091	133315798061
test_rustdoc	200450251096	149035748700	177891240791	152349722302	220247137823
test_stage1-libstd	1029686395987	654716486806	775625164407	695288331439	1039390710264
test_stage1-libtest	7598260360	5261907379	6251759053	5915745567	12324680750
test_stage1-librustc	220049426123	153715524105	178912343929	160360240497	241623770761
test_stage2-rustdoc	42965605060	29301018269	33073642519	30593204691	46241877515
stage0-unstable-book-gen	540293225	466071453	501797031	524692622	604592056
stage0-rustbook	547002605	482647023	511328950	564230039	600771757
stage2-error_index_generator	327629660	286363961	293505793	310137953	384443462
stage0-linkchecker	4111680727	3027848970	3329422820	3245560630	4444792413
test_docs	311036415452	213191547593	248141707395	223573139716	334901033360
test_error_index	87458066705	57046231936	68310213963	59190431368	88860845146

Normalized against total time taken for `make all`

	This PR	Base	Latest Master	PR47676	PR47795
stage0-std	5.826	5.637	5.666	5.684	5.976
stage0-tidy	0.598	0.600	0.602	0.620	0.623
tidy	6.539	6.389	6.415	6.447	6.726
014a3822	6.637	6.515	6.530	6.582	6.843
stage0-std	0.038	0.051	0.045	0.054	0.049
stage0-test	1.051	1.010	1.041	1.029	1.041
stage0-rustc	90.680	89.884	91.016	91.121	92.010
stage1-std	5.811	5.175	5.033	5.092	5.870
stage1-test	0.675	0.668	0.695	0.672	0.670
stage1-rustc	63.031	65.396	65.054	65.184	62.535
stage2-rustdoc	17.995	16.798	16.915	16.744	17.612
stage0-std	0.045	0.053	0.045	0.056	0.041
stage0-test	0.041	0.052	0.044	0.054	0.040
stage0-rustc	0.044	0.058	0.047	0.055	0.043
stage1-std	0.030	0.037	0.032	0.037	0.030
stage1-test	0.025	0.029	0.025	0.029	0.024
stage1-rustc	0.027	0.033	0.028	0.033	0.026
stage0-unstable-book-gen	0.696	0.738	0.693	0.760	0.656
stage0-rustbook	14.406	14.988	14.394	14.156	13.917
stage2-rustdoc	0.025	0.031	0.029	0.032	0.027
stage2-error_index_generator	0.457	0.548	0.459	0.549	0.469
15a60314	200.000	200.000	200.000	200.000	200.000
stage0-std	0.040	0.056	0.046	0.055	0.038
stage0-tidy	0.091	0.116	0.099	0.118	0.097
tidy	0.244	0.325	0.293	0.326	0.246
stage0-test	0.037	0.054	0.046	0.057	0.039
stage0-rustc	0.040	0.057	0.049	0.059	0.040
stage1-std	0.028	0.039	0.033	0.042	0.028
stage1-test	0.022	0.039	0.025	0.031	0.024
stage1-rustc	0.026	0.032	0.029	0.035	0.026
build_test_helpers	0.003	0.004	0.004	0.005	0.003
stage0-compiletest	2.167	2.197	2.250	2.178	2.226
test_ui	7.936	8.575	8.193	8.354	7.692
test_run-pass	93.163	91.763	92.288	92.697	88.320
test_compile-fail	19.318	22.024	21.165	22.417	18.230
test_parse-fail	0.349	0.460	0.433	0.491	0.342
test_run-fail	2.470	2.312	2.443	2.477	2.382
test_run-pass-valgrind	0.379	0.377	0.386	0.404	0.371
test_mir-opt	2.089	2.128	2.184	2.274	1.948
test_codegen	0.863	0.787	0.842	0.836	0.816
test_codegen-units	0.787	0.699	0.745	0.734	0.730
test_incremental	5.939	5.188	5.340	5.527	5.430
test_debuginfo	2.107	2.090	2.173	2.228	2.031
test_ui-fulldeps	1.392	1.287	1.383	1.355	1.338
test_run-pass-fulldeps	24.181	21.477	23.791	22.668	24.927
test_run-fail-fulldeps	0.267	0.232	0.229	0.240	0.234
test_compile-fail-fulldeps	4.047	3.726	3.993	3.924	3.784
test_incremental-fulldeps	0.480	0.462	0.451	0.480	0.472
stage2-rustdoc	0.026	0.030	0.029	0.031	0.029
test_run-make	9.727	8.924	9.420	9.455	9.002
test_rustdoc	14.858	14.967	16.413	15.525	14.872
test_stage1-libstd	76.326	65.750	71.561	70.853	70.185
test_stage1-libtest	0.563	0.528	0.577	0.603	0.832
test_stage1-librustc	16.311	15.437	16.507	16.341	16.316
test_stage2-rustdoc	3.185	2.943	3.051	3.118	3.122
stage0-unstable-book-gen	0.040	0.047	0.046	0.053	0.041
stage0-rustbook	0.041	0.048	0.047	0.057	0.041
stage2-error_index_generator	0.024	0.029	0.027	0.032	0.026
stage0-linkchecker	0.305	0.304	0.307	0.331	0.300
test_docs	23.056	21.410	22.894	22.783	22.614
test_error_index	6.483	5.729	6.302	6.032	6.000

[kennytm]

Ping from triage @aidanhs! Do we want to try this?

[aidanhs]

Yeah I think so.

But - isn't codegen-units=1 the default for --release builds? I.e. we should combine this PR with #44675 to use codegen units for non-dist bots.

[aidanhs]

I had lost track of what codegen-units is currently doing. After reading #45444 and #47834, I now see that codegen units are enabled for everything except beta/stable dist builds.

I'll try this locally in the next couple of weeks to see if I observe the similar speedups, in which case we can take a more careful look at why it slows travis down.

[michaelwoerister]

This PR just leads to test being compiled with one codegen unit. The compiler is not affected.

If this doesn't make much of a difference, it's probably better to close this PR, since it will run tests in a non-default compiler mode.

[pnkfelix]

@michaelwoerister what was your test platform where you observed the time go from 8m30s to 7m05s?

(the data that says "this makes no difference" was from linux, but that's not the primary build platform that we need to speed up anyway, is it?)

[michaelwoerister]

I tested on Ubuntu 16.04 with a 4C/8T CPU.

[arazabishov]

Ping from triage @aidanhs :) Did you have a chance to take a look why builds on travis are slowed down?

[aidanhs]

I'm sorry, I didn't - give me another couple of days to give it a try :)

[aidanhs]

I tried this locally via ./stamp ./x.py test --stage 1 src/test/run-pass on 16.04, 4C/8T CPU

• with --disable-optimize (the default), the tests went from 221.195s to 192.175s
• with --enable-optimize, the tests went from 271.008s to 198.000s

Definitely seems to make a difference. I had a thought though - codegen-units allows rustc itself to parallelise, and rust tests are run in parallel. Doesn't this mean that we're parallelising squared? Watching the load avg on my machine seems to confirm this - it hovers around 8 with this PR enabled (i.e. with codegen-units=1), but becomes at least double without it. Will do some more digging.

[michaelwoerister]

Doesn't this mean that we're parallelising squared?

Yes, that's one of the two main opportunities for saving time here. Splitting a program into multiple CGUs introduces some overhead but because we are already running so many compiler instances in parallel that additional overhead won't amortize itself.

The second reason one CGU might be faster for small programs (at least with --enable-optimize) is that we default to using ThinLTO, which again introduces overhead (and quite a bit of it) that can only be paid off by tapping into unused CPU cores. Since we already use all CPU cores we won't see any speedups.

However, I'm not sure how much we should change settings here from a test coverage perspective. Multiple CGUs and ThinLTO are the current defaults so I'm wary of running the test suite in a different mode.

[ishitatsuyuki]

Doesn't this mean that we're parallelising squared?

Sounds like this is a valid usecase for #48532, but well disabling multiple CGUs works either.

[shepmaster]

Ping from triage, @aidanhs !

[shepmaster]

Gonna reassign to...

r? @kennytm

[kennytm]

@bors r+

Let's try this.

[bors]

📌 Commit 3ffa6da has been approved by kennytm

[bors]

⌛ Testing commit 3ffa6da with merge 1c1e3fa0c13d7b542bafb3b6401bbd57403d2cbc...

[bors]

💔 Test failed - status-travis

[kennytm]

Failed with wasm32-unknown-unknown 🤔

[01:05:58] failures:
[01:05:58] 
[01:05:58] ---- [run-pass] run-pass/issue-33992.rs stdout ----
[01:05:58] 	
[01:05:58] error: compilation failed!
[01:05:58] status: signal: 6
[01:05:58] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/run-pass/issue-33992.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass" "--target=wasm32-unknown-unknown" "-Ccodegen-units=1" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass/issue-33992.stage2-wasm32-unknown-unknown.wasm" "-Crpath" "-O" "-Zmiri" "-Zunstable-options" "-Lnative=/checkout/obj/build/wasm32-unknown-unknown/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass/issue-33992.stage2-wasm32-unknown-unknown.aux"
[01:05:58] stdout:
[01:05:58] ------------------------------------------
[01:05:58] 
[01:05:58] ------------------------------------------
[01:05:58] stderr:
[01:05:58] ------------------------------------------
[01:05:58] Common symbols are not yet implemented for Wasm
[01:05:58] UNREACHABLE executed at /checkout/src/llvm/lib/MC/MCWasmStreamer.cpp:134!
[01:05:58] 
[01:05:58] ------------------------------------------
[01:05:58] 
[01:05:58] thread '[run-pass] run-pass/issue-33992.rs' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:2918:9
[01:05:58] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:05:58] 
[01:05:58] 
[01:05:58] failures:
[01:05:58]     [run-pass] run-pass/issue-33992.rs

[shepmaster]

Ping from triage, @michaelwoerister ! Will you have time to address the test failures?

[michaelwoerister]

I've actually come around to the opinion that we should not do this as it would cause our tests to be run in a non-standard compilation mode.