You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The code for regular simulation and the code for :raw and :compact boosted options are essentially three different implementations of the same model. This is necessary for performance reasons, but it's rather fragile as we must be very careful that all implementations remain consistent with each other, and yield the same results no matter the model parametrization used.
In this respect, I have made it mandatory that every call to simulate[s] in the tests is actually turned into one call for each version and the results compared to each other before actually returning. This is a good smoke test, but it's not enough to rigorously cover the variety of possible parametrizations. Also, the tests in ./test/ are short tests not either useful for benchmarking.
So long as this situation remains, I have chosen to emit this warning every time a boosted simulation is used.
To remove this warning, we need a proper set of controlled simulation scenarii that we would use to verify that regular, :raw and :compact code yield the same results and measure their relative performances. This set would ideally be:
complete: in the sense that running it would cover all possible functional forms of the model (e.g.Bioenergetic, ClassicalResponse, LinearResponse, with mortality, with stochasticity, with NTIs etc.)
non-trivial: in the sense that resulting trajectories would not be stationary like "all species extinct" or "all biomasses constant", because comparing precision and performances of models only crunching zeroes is not super meaningful. Ideally, the simulation scenarii would result in or cycling or chaotic coexistence trajectories, that we can extend to any duration we want for the benchmarks.
diverse in complexity: in the sense that the set would span parametrizations with only a few species (e.g.2 <= S <= 5) (best boosted with :raw) and parametrizations with numerous species (e.g.20 <= S <= 100) (best boosted with :compact).
I am aware that constructing such a set is non-trivial, especially when it comes to crafting a model with a large number of coexisting species, but since it's about testing the code and its performance, it does not exactly matter whether it would be biologically meaningful.. and maybe this would help?
Here is an example list of the expected scenarii, we can refine it so it makes better sense:
featuring
:raw (2<= S<=10)
:compact (20<=S<=100)
boost status yet
LinearResponse
TODO
TODO
untested
ClassicalResponse
TODO
TODO
untested
BioenergeticResponse
TODO
TODO
untested
mortality × LinearResponse
TODO
TODO
untested
mortality × ClassicalResponse
TODO
TODO
untested
mortality × BioenergeticResponse
TODO
TODO
untested
facilitation
TODO
TODO
un-boosted
competition
TODO
TODO
un-boosted
interference
TODO
TODO
un-boosted
refuge
TODO
TODO
un-boosted
producers competition
TODO
TODO
un-boosted
stochasticity
TODO
TODO
un-boosted
...
With such a list of controlled parametrizations, our test suite / benchmark would effectively become strong and we would pretend to high-quality software ;)
The text was updated successfully, but these errors were encountered:
The code for regular simulation and the code for
:raw
and:compact
boosted options are essentially three different implementations of the same model. This is necessary for performance reasons, but it's rather fragile as we must be very careful that all implementations remain consistent with each other, and yield the same results no matter the model parametrization used.In this respect, I have made it mandatory that every call to
simulate[s]
in the tests is actually turned into one call for each version and the results compared to each other before actually returning. This is a good smoke test, but it's not enough to rigorously cover the variety of possible parametrizations. Also, the tests in./test/
are short tests not either useful for benchmarking.So long as this situation remains, I have chosen to emit this warning every time a boosted simulation is used.
To remove this warning, we need a proper set of controlled simulation scenarii that we would use to verify that regular,
:raw
and:compact
code yield the same results and measure their relative performances. This set would ideally be:complete: in the sense that running it would cover all possible functional forms of the model (e.g.
Bioenergetic
,ClassicalResponse
,LinearResponse
,with mortality
,with stochasticity
,with NTI
s etc.)non-trivial: in the sense that resulting trajectories would not be stationary like "all species extinct" or "all biomasses constant", because comparing precision and performances of models only crunching zeroes is not super meaningful. Ideally, the simulation scenarii would result in or cycling or chaotic coexistence trajectories, that we can extend to any duration we want for the benchmarks.
diverse in complexity: in the sense that the set would span parametrizations with only a few species (e.g.
2 <= S <= 5
) (best boosted with:raw
) and parametrizations with numerous species (e.g.20 <= S <= 100
) (best boosted with:compact
).I am aware that constructing such a set is non-trivial, especially when it comes to crafting a model with a large number of coexisting species, but since it's about testing the code and its performance, it does not exactly matter whether it would be biologically meaningful.. and maybe this would help?
Here is an example list of the expected scenarii, we can refine it so it makes better sense:
:raw
(2<= S<=10):compact
(20<=S<=100)LinearResponse
ClassicalResponse
BioenergeticResponse
mortality × LinearResponse
mortality × ClassicalResponse
mortality × BioenergeticResponse
facilitation
competition
interference
refuge
producers competition
stochasticity
With such a list of controlled parametrizations, our test suite / benchmark would effectively become strong and we would pretend to high-quality software ;)
The text was updated successfully, but these errors were encountered: