-
-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rationalize tests for CI vs. releases #1117
Comments
Since most time is spent on compilation, it significantly increases to total CPU time when the tests are run as many separate jobs. However, the user time might take a significant hit if run sequentially. Might also be interesting to see the effect to running tests with a low optimization level. |
Here is sorted test time.
The |
They also have unitful tests. I had already thought about hoisting them out in an extra test file, but haven't got to it. |
An extra data point #1118 (comment) |
Do we really need to test Triangular this much? Perhaps we can just test fewer types as well. |
Might be possible to reduce the number of combinations but looping over types did reveal several issues when it was introduced. |
Yes, that's what I was thinking. It was greatly helpful early on, but presumably for catching regressions, we may only need fewer combinations. |
It should be possible to ensure that code coverage isn't reduced by trimming the combinations. |
Code coverage is just an incomplete measure when it comes to some of the possible issues that can show up when mixing types. I'm not saying that we can't reduce the tests. I'm just saying that it is possible to make them less useful while maintaining code coverage. |
There should still be some thought behind it instead of "spray and pray" so it is established what is unique about every combination. There are no comments now which makes it hard to know what the thought was. |
Just browsed |
One should then test things "orthogonally":
There is absolutely no way you should be required to test a semirandom 5x5x5x5 type combination. You are still missing 99% of custom types from the ecosystem. Again, I completely reject the notion that you have to spend one hour compiling to test the tridiagonal support in Julia's linear algebra. 😆 |
I thought back in the day, doing this in deeply nested loops etc. helped discover issues in the compiler. |
I actually don't think that many compiler issues surfaced as part of this testing but there were many issues related to promotion. There are just a lot of type combinations related to the triangular types. It's cool that Julia allow you to maintain structures such UnitLowerTriangular*UnitLowerTriangular = UnitLowerTriangular
UnitLowerTriangular*LowerTriangular = LowerTriangular
LowerTriangular*UpperTriangular = (Some)Matrix and that backsolve with
The full cartesian product here is probably not adding much value, though. It's just an easy. Let's see how much of a difference #1123 makes as a first step. |
Is it possible to rationalize the way we test in CI to use much lesser time than right now?
Maybe have a set of CI tests and a set of complete tests. Perhaps the complete tests can be run during releases of LinearAlgebra.jl into julia master.
I suppose the harder part of all of this is the workflow.
@DilumAluthge Any ideas?
The text was updated successfully, but these errors were encountered: