Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test harness for running all of Julia's tests #12

Merged
merged 1 commit into from
Feb 10, 2019
Merged

Conversation

timholy
Copy link
Member

@timholy timholy commented Feb 10, 2019

This adds an enormous test suite for JuliaInterpreter: all of Julia's own tests. It is presumably impractical to run these on Travis, but local devs can do so. The idea here is to use this to see how robust (or not) JuliaInterpreter really is. (If anyone has ideas about how to get them to run on PRs without bogging things down too much, I'm all ears.)

The tests do not pass yet, but this allows people to work towards that goal. Meanwhile I'll get started on an issue that flags failures and some of the reasons why. Help (even just triaging files that pass and those that do not) would be very welcome!

@codecov-io
Copy link

codecov-io commented Feb 10, 2019

Codecov Report

Merging #12 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master      #12   +/-   ##
=======================================
  Coverage   70.72%   70.72%           
=======================================
  Files           6        6           
  Lines        1117     1117           
=======================================
  Hits          790      790           
  Misses        327      327

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3015557...137e6a7. Read the comment docs.

Tests do not pass yet, but this allows people to work towards that goal.
@timholy timholy merged commit 62bb82a into master Feb 10, 2019
@timholy timholy deleted the teh/juliatests branch February 10, 2019 18:04
@vchuravy
Copy link
Member

We might be able to run them through Gitlab on our own Hardware and in stages

@timholy
Copy link
Member Author

timholy commented Feb 13, 2019

OK. My mission for the morning is to rewrite this to:

  • support resource-limiting: e.g., each lowered block gets at most n statements that can be executed before bailing (thus preventing, e.g., infinite loops or really slow tests from blocking everything)
  • aggregate test results
  • write out results in Markdown table format for pasting into Julia tests don't pass #13 as status updates
  • parallelize

Would be nice to have an overall summary but then click on individual test files to see which lines pass and which do not. Hopefully once that's done, maybe it will be easier to "parallelize development" and have more folks digging into specific failures.

EDIT: done!

@timholy timholy mentioned this pull request Feb 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants