WIP: Some benchmarks for the documentation #123
Conversation
|
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #123 +/- ##
==========================================
- Coverage 98.26% 98.05% -0.21%
==========================================
Files 7 7
Lines 1556 1647 +91
==========================================
+ Hits 1529 1615 +86
- Misses 27 32 +5 ☔ View full report in Codecov by Sentry. |
SKopecz
changed the title
docs/src/convergence.md
Outdated
| #```@example eoc | ||
| using PrettyTables | ||
|
|
||
| # collect data and create headers |
There was a problem hiding this comment.
Coding this almost drove me crazy!
Originally I used something like data = dts outside the loop and data = [data err[i] [NaN; eoc[i]]] within the loop. This was perfectly fine on my local machine, but caused an error like "data not defined" in the build process.
@ranocha: Any idea what the problem was/is?
There was a problem hiding this comment.
Did you built the docs locally or copy and paste the code to the REPL?
There was a problem hiding this comment.
Oh. I didn't build the docs locally, I copied & pasted everything from my vscode repl to the markdown files. But I ensured to use the docs environment.
There was a problem hiding this comment.
Then it might be an issue of soft global scope, see https://docs.julialang.org/en/v1/manual/variables-and-scoping/#on-soft-scope
There was a problem hiding this comment.
You may need to add a keyword global
|
@ranocha: I played around with the I'm also going to add WP diagrams for constant time step sizes. |
docs/src/convergence.md
Outdated
| formatters = (ft_printf("%5.4e", [1, 2, 4, 6, 8]), | ||
| ft_printf("%5.4f", [3, 5, 7, 9]))) |
There was a problem hiding this comment.
I think we don't need all of the digits - this would also keep the tables smaller so that we don't have to scroll as much. And maybe we can just show "EOC" right next to the value so that the headers in the first row are not duplicated.
docs/src/npzd_model_benchmark.md
Outdated
| # select problem | ||
| prob = prob_pds_npzd | ||
|
|
||
| # compute reference solution (standard tolerances are too low) |
There was a problem hiding this comment.
Do you mean not strict enough? The question is if you consider 1e-5 to be lower (smaller) than 1e-13
docs/src/npzd_model_benchmark.md
Outdated
| labels = ["MPRK22(0.5)" | ||
| "MPPRK22(2/3)" | ||
| "MPRK22(1.0)" | ||
| "SSPMPRK22(0.5,1.0)" | ||
| "MPRK43I(1.0,0.5)" | ||
| "MPRK43I(0.5,0.75)" | ||
| "MPRK43II(0.5)" | ||
| "MPRK43II(2.0/3.0)"] |
There was a problem hiding this comment.
This pattern seems to be present several times. Maybe we shall overload show to automate this? But we would likely lose some information so I am not fully sure...
docs/src/npzd_model_benchmark.md
Outdated
| Although the SSPMPRK22 solution seems to be more accurate at first glance, the `l∞`-error of the SSPMPRK22 scheme is 0.506359, whereas the `l∞`-error of the MPRK43 scheme is 0.413915. Both errors occurs at approximately $t=2$, where there is a sharp kink in the first component. | ||
|
|
||
|
|
||
| Next we compare `SSPMPRK22(0.5, 1.0)` and `MPRK43I(1.0, 0.5)` with some second and third order methods from [OrdinaryDiffEq.jl](https://docs.sciml.ai/OrdinaryDiffEq/stable/). To guarantee positive solutions with these methods, we must select the solver option `isoutofdomain = isnegative`. |
There was a problem hiding this comment.
Is isoutofdomain = isnegative to get good work-precision diagrams? How would they look like without it?
docs/src/npzd_model_benchmark.md
Outdated
| # set tolerances and error | ||
| abstols = 1.0 ./ 10.0 .^ (2:0.5:8) | ||
| reltols = 1.0 ./ 10.0 .^ (1:0.5:7) | ||
| err_est = :l∞ |
There was a problem hiding this comment.
Do other error choices lead to significantly different results?
docs/src/robertson_benchmark.md
Outdated
| plot!(p1, sol1; tspan = (1e-6, 1e11), xaxis = :log, denseplot = false, markers = :circle, ylims = (-0.2, 1.2), idxs = [(0, 1), ((x, y) -> (x, 1e4 .* y), 0, 2), (0, 3)], title = "SSPMPRK22(1.0, 0.5)", xticks =10.0 .^ (-6:4:10)) | ||
| ``` | ||
|
|
||
| With `abstol=1e-6` and `reltol = 1e-5` the `MPRK22(0.5)` schemes needs over 800.000 steps to integrate the Robertson problem. In comparison, `MPRK22(0.5)` needs less than 1.000. |
There was a problem hiding this comment.
| With `abstol=1e-6` and `reltol = 1e-5` the `MPRK22(0.5)` schemes needs over 800.000 steps to integrate the Robertson problem. In comparison, `MPRK22(0.5)` needs less than 1.000. | |
| With `abstol=1e-6` and `reltol = 1e-5` the `MPRK22(0.5)` schemes needs over 800.000 steps to integrate the Robertson problem. | |
| In comparison, `MPRK22(1.0)` needs less than 1.000. |
Is this related to solutions close to zero and the issues we discovered with Davide and Philipp? Or due to maybe a bad error estimator?
There was a problem hiding this comment.
Relative errors would make sense here as well
There was a problem hiding this comment.
Yes, especially for the stratospheric reaction problem.
|
Maybe it would also make sense to check the controllers Thomas optimized in https://arxiv.org/abs/2312.01796 |
Co-authored-by: Hendrik Ranocha <[email protected]>
|
In the Robertson benchmark tutorial we see
But on my local machine the situation is different. With
When I originally created the benchmark tutorial I excluded In addition, the question arises if schemes that deliver good results in the benchmark tutorials perform significantly worse on other machines? I suggest to keep @ranocha: Any thoughts on this? |
|
@ranocha The CI tests fail because there are issues with |
Sounds reasonable to me 👍 |
test/runtests.jl
Outdated
There was a problem hiding this comment.
Since you're using the median below, you should change the line
using Statistics: meanin line 4 to
using Statistics: mean, medianI can't comment on the line directly
This will add some benchmarks for #7.