Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for comparison scenarios that have different input settings in the two models #453

Closed
chrispcampbell opened this issue Mar 25, 2024 · 2 comments · Fixed by #456 or #452

Comments

@chrispcampbell
Copy link
Contributor

Currently model-check is designed in such a way that comparison tests can be defined independent of the models/bundles being compared. However, there are cases where it would be better for the bundle to declare a scenario that it can handle in a certain way.

For example, for En-ROADS we have a comparison scenario defined for "all main policy sliders at best case". We want to change a couple of sliders that are included in the main interface (remove a couple, add a couple), and we also want to change the meaning of better/worse for a couple sliders (before, the user would slide to the left for "better", now the user slides to the right for "better"). This is difficult to express in a comparison scenario defined in the YAML/JSON format because the input IDs and min/max depend on the model version being tested.

For this use case, it would be better to have the bundle define a scenario with a certain ID (e.g., extreme_all_main_sliders_at_best_case). That way, the bundle knows the best way to configure the scenario, and no conditional logic needs to be added in the test package.

@chrispcampbell chrispcampbell self-assigned this Mar 25, 2024
@chrispcampbell chrispcampbell changed the title Add support for comparison scenarios defined in the model-check bundle Add support for comparison scenarios that have different input settings in the two models Mar 26, 2024
@chrispcampbell
Copy link
Contributor Author

I decided that this can be done more simply for now (without touching as much code) if we don't make any changes at the bundle interface level, and instead add a new variant of ComparisonScenarioSpec that takes separate lists of input settings for the left and right sides:

image

This approach doesn't require changing the UI, and for now we don't need to include support for this in JSON/YAML parsing side of things, which simplifies the implementation.

If there is an error in resolving an input on one or both sides, the scenario will be marked with an error, same as how it works for resolving the existing scenarios.

@chrispcampbell
Copy link
Contributor Author

I've updated the model-check sample tests to demonstrate this:

image image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment