You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently model-check is designed in such a way that comparison tests can be defined independent of the models/bundles being compared. However, there are cases where it would be better for the bundle to declare a scenario that it can handle in a certain way.
For example, for En-ROADS we have a comparison scenario defined for "all main policy sliders at best case". We want to change a couple of sliders that are included in the main interface (remove a couple, add a couple), and we also want to change the meaning of better/worse for a couple sliders (before, the user would slide to the left for "better", now the user slides to the right for "better"). This is difficult to express in a comparison scenario defined in the YAML/JSON format because the input IDs and min/max depend on the model version being tested.
For this use case, it would be better to have the bundle define a scenario with a certain ID (e.g., extreme_all_main_sliders_at_best_case). That way, the bundle knows the best way to configure the scenario, and no conditional logic needs to be added in the test package.
The text was updated successfully, but these errors were encountered:
chrispcampbell
changed the title
Add support for comparison scenarios defined in the model-check bundle
Add support for comparison scenarios that have different input settings in the two models
Mar 26, 2024
I decided that this can be done more simply for now (without touching as much code) if we don't make any changes at the bundle interface level, and instead add a new variant of ComparisonScenarioSpec that takes separate lists of input settings for the left and right sides:
This approach doesn't require changing the UI, and for now we don't need to include support for this in JSON/YAML parsing side of things, which simplifies the implementation.
If there is an error in resolving an input on one or both sides, the scenario will be marked with an error, same as how it works for resolving the existing scenarios.
Currently model-check is designed in such a way that comparison tests can be defined independent of the models/bundles being compared. However, there are cases where it would be better for the bundle to declare a scenario that it can handle in a certain way.
For example, for En-ROADS we have a comparison scenario defined for "all main policy sliders at best case". We want to change a couple of sliders that are included in the main interface (remove a couple, add a couple), and we also want to change the meaning of better/worse for a couple sliders (before, the user would slide to the left for "better", now the user slides to the right for "better"). This is difficult to express in a comparison scenario defined in the YAML/JSON format because the input IDs and min/max depend on the model version being tested.
For this use case, it would be better to have the bundle define a scenario with a certain ID (e.g.,
extreme_all_main_sliders_at_best_case
). That way, the bundle knows the best way to configure the scenario, and no conditional logic needs to be added in the test package.The text was updated successfully, but these errors were encountered: