Skip to content

Implement tests on inference scenarios for models #1076

Closed
Mr-Geekman opened this issue Jan 20, 2023 · 0 comments · Fixed by #1082
Closed

Implement tests on inference scenarios for models #1076

Mr-Geekman opened this issue Jan 20, 2023 · 0 comments · Fixed by #1082
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@Mr-Geekman
Copy link
Contributor

Mr-Geekman commented Jan 20, 2023

🚀 Feature Request

We want to have tests that check different inference scenarios for models.

Proposal

Add tests in test_inference on:

  1. Forecast on subset of segments:
  • Check that forecast on subset of segments is equal to subset of forecast on all segments.
  1. Predict on subset of segments.
  • Check that predict on subset of segments is equal to subset of predict on all segments.
  1. Forecast on new segments.
  • Check that forecast fills the target without fail.
  1. Predict on new segments.
  • Check that predict fills the target without fail (target will be already filled, but we want to check that it doesn't fail).
  1. Predict on future data with gap (we can try to do it like with forecast: TestForecastOutSamplePrefix,
    TestForecastOutSampleSuffix)
  • Check that prediction on prefix of future dataset is the same as prefix of prediction on bigger future dataset.
  • Check that prediction on suffix of future dataset is the same as suffix of prediction on bigger future dataset.

If some tests are failed for some models mark this tests as to_be_fixed if we want to fix them. If we understand that it isn't possible to fix this (model's properties) make a separate test case for such models.

We have already tested

  1. How forecast works on data in future with gap: TestForecastOutSampleSuffix.
  2. How predict works on data right after train: TestPredictOutSample.

Test cases

No response

Additional context

No response

@Mr-Geekman Mr-Geekman added the enhancement New feature or request label Jan 20, 2023
@Mr-Geekman Mr-Geekman added this to the Inference 2.1 milestone Jan 20, 2023
@github-project-automation github-project-automation bot moved this to Specification in etna board Jan 20, 2023
@Mr-Geekman Mr-Geekman moved this from Specification to Todo in etna board Jan 23, 2023
@Mr-Geekman Mr-Geekman self-assigned this Jan 23, 2023
@Mr-Geekman Mr-Geekman moved this from Todo to In Progress in etna board Jan 23, 2023
@Mr-Geekman Mr-Geekman moved this from In Progress to In Review in etna board Jan 26, 2023
@github-project-automation github-project-automation bot moved this from In Review to Done in etna board Jan 31, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants