This repository has been archived by the owner on Aug 2, 2020. It is now read-only.
GHC testsuite rules status #669
Assignees
Labels
Comments
|
Many thanks @alpmestan! I will check if there are any outdated testsuite related issues and will close them. There is also this PR #639, which now seems abandoned. Feel free to close it. |
This was referenced Sep 1, 2018
Closed
|
With this patch, f I only run the failing tests from the first comment, I get: Test resultsSo, 500 unexpected failures are gone with the PR. Now, as a note for future me, I've noticed that |
|
https://phabricator.haskell.org/D5127 should fix all the |
|
With the aforementionned patch, I'm down to 184 unexpected failures Test resultsNote: We're still skipping many many tests for some reason. For reference, a |
|
My differential has been merged: ghc/ghc@2254912 Meanwhile, I looked at #15570 and #15286 and it turns out removing those New test resultsMany, many of the remaining failures are due to ghci printing things differently (e.g |
|
Great work, @alpmestan! I'll try to figure out why |
|
OK; make sure you use |
|
#677 makes all the ghc-compact tests pass. |
|
With #684, I get: This is looking better and better. |
|
@alpmestan Awesome! What about |
|
See my comment above:
You can see that the normal validate script skips many tests too. Many tests are restricted to be executed only on some arch, some os, some way, some [something] that'll cause it to be skipped in many situations. We're still missing some tests and probably skipping too many of them, but we're somewhat close to what the make build system does, aren't we? |
|
Hi @alpmestan, this might not be the right place but I wanted to ask about workflows for the testsuite. Very often I do something like to run a single test with some custom options. This is really convenient (and fast). Will we have something equally convenient with Hadrian, and will it be just as fast? |
|
@simonmar We don't do anything (for reference: Usage: hadrian [options] [target] ...
Options:
-a FULL=SHORT, --abbrev=FULL=SHORT
Use abbreviation in status messages.
--no-build Don't build anything.
-C DIRECTORY, --directory=DIRECTORY
Change to DIRECTORY before doing anything.
--color, --colour Colorize the output.
--no-color, --no-colour Don't colorize the output.
-d[FILE], --debug[=FILE] Print lots of debugging information.
--demo Run in demo mode.
--digest Files change when digest changes.
--digest-and Files change when modtime and digest change.
--digest-and-input Files change on modtime (and digest for inputs).
--digest-or Files change when modtime or digest change.
--digest-not Files change when modtime changes.
--exception Throw exceptions directly.
--flush=N Flush metadata every N seconds.
--never-flush Never explicitly flush metadata.
-h, --help Print this message and exit.
-j[N], --jobs[=N] Allow N jobs/threads at once [default CPUs].
-k, --keep-going Keep going when some targets can't be made.
-l, --lint Perform limited validation after the run.
--lint-fsatrace Use fsatrace to do validation.
--no-lint Turn off --lint.
--live[=FILE] List the files that are live [to live.txt].
-m PREFIX, --metadata=PREFIX
Prefix for storing metadata files.
--numeric-version Print just the version number and exit.
--skip-commands Try and avoid running external programs.
--rebuild[=PATTERN] Rebuild matching files.
--no-rebuild[=PATTERN] Rebuild matching files if necessary.
--skip[=PATTERN] Don't rebuild matching files this run.
-r[FILE], --report[=FILE], --profile[=FILE]
Write out profiling information [to report.html].
--no-reports Turn off --report.
--rule-version=VERSION Version of the build rules.
--no-rule-version Ignore the build rules version.
-s, --silent Don't print anything.
--sleep Sleep for a second before building.
-S, --no-keep-going, --stop
Turns off -k.
--storage Write a storage log.
-p[N], --progress[=N] Show progress messages [every N secs, default 5].
--no-progress Don't show progress messages.
-q, --quiet Print less (pass repeatedly for even less).
--no-time Don't print build time.
--timings Print phase timings.
-V, --verbose, --trace Print more (pass repeatedly for even more).
-v, --version Print the version number and exit.
-w, --print-directory Print the current directory.
--no-print-directory Turn off -w, even if it was turned on implicitly.
-c, --configure Run the boot and configure scripts (if you do not want to run them manually).
-o[BUILD_ROOT], --build-root[=BUILD_ROOT]
Where to store build artifacts. (Default _build).
--flavour[=FLAVOUR] Build flavour (Default, Devel1, Devel2, Perf, Prof, Quick or Quickest).
--freeze1 Freeze Stage1 GHC.
--integer-simple Build GHC with integer-simple library.
--progress-colour[=MODE] Use colours in progress info (Never, Auto or Always).
--progress-info[=STYLE] Progress info style (None, Brief, Normal or Unicorn).
--split-objects Generate split objects (requires a full clean rebuild).
--test-compiler[=TEST_COMPILER]
Use given compiler [Default=stage2].
--test-config-file[=CONFIG_FILE]
congiguration file for testsuite. Default=testsuite/config/ghc
--config[=EXTRA_TEST_CONFIG]
Configurations to run test, in key=value format.
--summary-junit[=TEST_SUMMARY_JUNIT]
Output testsuite summary in JUnit format.
--only[=TESTS] Test cases to run.
--only-perf Only run performance tests.
--skip-perf Skip performance tests.
--test-speed[=SPEED] fast, slow or normal. Normal by default
--summary[=TEST_SUMMARY] Where to output the test summary file.
--test-verbose[=TEST_VERBOSE]
A verbosity value between 0 and 5. 0 is silent, 4 and higher activates extra output.
--test-way[=TEST_WAY] only run these ways) |
|
In that case can I request that support for this be added? It's really important when developing to be able to work on a single test, or a single directory of tests, and to be able to customize options, compilers and cleanup behaviour. I think we'll need this before we can drop the old build system. |
This was referenced Sep 11, 2018
|
Thanks @alpmestan. To add a bit more to this:
|
|
@simonmar As I pointed out in earlier comments, we can already specify a particular
Thanks for the input! It's helpful to have very precise TODOs like this :-) |
|
The PR at #687 fixes another batch of failures, in addition to solving a rather important problem. With this patch I get: Remaining failures: |
|
Aaaaand we're down to 46 unexpected failures with #688: |
|
#689 and https://phabricator.haskell.org/D5146 solve some more. Down to 42 unexpected failures. I can already smell the wonderful perfume of a completely green |
|
Earlier today, I ran the I looked at all the failures and managed to classify them in a few categories. Require a GHC built with -dynamicThis will be fixed once the joint work with David lands (Cabal + ghc + hadrian patch). See #691. Cannot locate libffi or libHSrtsWe don't build ghc-iserv-profThe make build system builds a -prof version of ghc-iserv which it calls ghc-iserv-prof, and this is required by a few tests. I'm going to look into how we can produce two executables from just one package... Less output than expectedThose in generally look benign, but I have no idea why the output differs. Bad handling of -osuf + -prof options ?haddock perf failuresWe quite likely don't fit with the expected inplace layout, something like that? |
|
Thanks @alpmestan -- looks like we're almost there! |
|
With https://phabricator.haskell.org/D5255 I get: |
|
Test run against ghc/ghc@1a3b9bd: |
|
I believe this issue is out of date, so closing. |
ghc-mirror-bot
pushed a commit
to ghc/ghc
that referenced
this issue
Jul 29, 2019
snowleopard/hadrian#187 was superseded by snowleopard/hadrian#669, which has also been closed. So, optimistically, dropping this as a limitation issue.
ghc-mirror-bot
pushed a commit
to ghc/ghc
that referenced
this issue
Jul 29, 2019
snowleopard/hadrian#187 was superseded by snowleopard/hadrian#669, which has also been closed. So, optimistically, dropping this as a limitation issue.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I'm creating this issue to track the work on the testsuite rules that I just resumed. Right now, when I run:
hadrian/build.sh -c -j4 --flavour=perf --build-root=_tmp testI see a lot of failures.
Test results
Let's get this unexpected failures number down. If it's alright, I will use this issue to report the problems that cause those failures and track PRs that address them, along with the new test results that result from my patches. It just didn't feel appropriate to steal existing issues for this.
The text was updated successfully, but these errors were encountered: