Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Construct interesting simulation scenarii for boost testing. #92

Open
iago-lito opened this issue Jan 16, 2023 · 0 comments
Open

Construct interesting simulation scenarii for boost testing. #92

iago-lito opened this issue Jan 16, 2023 · 0 comments
Assignees

Comments

@iago-lito
Copy link
Collaborator

iago-lito commented Jan 16, 2023

The code for regular simulation and the code for :raw and :compact boosted options are essentially three different implementations of the same model. This is necessary for performance reasons, but it's rather fragile as we must be very careful that all implementations remain consistent with each other, and yield the same results no matter the model parametrization used.

In this respect, I have made it mandatory that every call to simulate[s] in the tests is actually turned into one call for each version and the results compared to each other before actually returning. This is a good smoke test, but it's not enough to rigorously cover the variety of possible parametrizations. Also, the tests in ./test/ are short tests not either useful for benchmarking.

So long as this situation remains, I have chosen to emit this warning every time a boosted simulation is used.

To remove this warning, we need a proper set of controlled simulation scenarii that we would use to verify that regular, :raw and :compact code yield the same results and measure their relative performances. This set would ideally be:

  • complete: in the sense that running it would cover all possible functional forms of the model (e.g. Bioenergetic, ClassicalResponse, LinearResponse, with mortality, with stochasticity, with NTIs etc.)

  • non-trivial: in the sense that resulting trajectories would not be stationary like "all species extinct" or "all biomasses constant", because comparing precision and performances of models only crunching zeroes is not super meaningful. Ideally, the simulation scenarii would result in or cycling or chaotic coexistence trajectories, that we can extend to any duration we want for the benchmarks.

  • diverse in complexity: in the sense that the set would span parametrizations with only a few species (e.g. 2 <= S <= 5) (best boosted with :raw) and parametrizations with numerous species (e.g. 20 <= S <= 100) (best boosted with :compact).

I am aware that constructing such a set is non-trivial, especially when it comes to crafting a model with a large number of coexisting species, but since it's about testing the code and its performance, it does not exactly matter whether it would be biologically meaningful.. and maybe this would help?

Here is an example list of the expected scenarii, we can refine it so it makes better sense:

featuring :raw (2<= S<=10) :compact (20<=S<=100) boost status yet
LinearResponse TODO TODO untested
ClassicalResponse TODO TODO untested
BioenergeticResponse TODO TODO untested
mortality × LinearResponse TODO TODO untested
mortality × ClassicalResponse TODO TODO untested
mortality × BioenergeticResponse TODO TODO untested
facilitation TODO TODO un-boosted
competition TODO TODO un-boosted
interference TODO TODO un-boosted
refuge TODO TODO un-boosted
producers competition TODO TODO un-boosted
stochasticity TODO TODO un-boosted
...

With such a list of controlled parametrizations, our test suite / benchmark would effectively become strong and we would pretend to high-quality software ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

No branches or pull requests

2 participants