Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic manual e2e regression test suite #285

Merged
merged 1 commit into from
Jan 19, 2023
Merged

Conversation

mitjat
Copy link
Contributor

@mitjat mitjat commented Jan 17, 2023

This PR adds a script that lets us check the HTTP output of various endpoints against old versions of the indexer.

Alluded to in #273, but this script is NOT integrated into CI. Its major caveat is that it's most informative when you run the indexer against prod data for a good number of rounds, and then check what it outputs. I don't think the script is useful enough to do it on every CI run -- it takes time, and it increases brittleness (because the test needs an external, synced oasis-node).

Changes in the PR:

  • tests/e2e_regression/run.sh - the main script
  • tests/e2e_regression/expected/* - files with the output as produced currently. I'm not checking these in because of the one-off nature of the script's intended usage.
  • analyzer/metadata_registry.go - a patch that lets one set up an analyzer with deterministic outputs: Use a config that runs the consensus+emerald analyzer only up to a certain height (this was already possible before this PR), and use the metadata_registry analyzer against a static intended-for-test branch of the registry.
    • EDIT: I'm excluding this too, now. The static, intended-for-test branch cannot be parsed by the current analyzer. Nobody is sure why (even Jernej, who wrote the tool a good while ago), and I don't think it's important enough to research or fix further. Since expected outputs are not checked in, they don't need to be long-term stable. I added a note about all of this to the main script.

@mitjat mitjat force-pushed the mitjat/e2e_regression branch from ac5432a to 1f60e9f Compare January 17, 2023 21:22
@mitjat mitjat changed the title analyzer/metadata_registry: Support for test backend Basic e2e regression test suite Jan 17, 2023
Copy link
Collaborator

@pro-wh pro-wh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alright, but this might rot

tests/e2e_regression/run.sh Outdated Show resolved Hide resolved
tests/e2e_regression/run.sh Outdated Show resolved Hide resolved
@mitjat
Copy link
Contributor Author

mitjat commented Jan 19, 2023

alright, but this might rot

Agreed. I'd say it's mostly valuable for refactoring PRs. I so happened to be working on these more, and found this test useful (and it found a bug!) several times already, and we're likely to do some more refactoring to accommodate multiple runtimes. So I think it has short-term potential :)

@mitjat mitjat force-pushed the mitjat/e2e_regression branch from 1f60e9f to 089bb4a Compare January 19, 2023 18:26
@mitjat mitjat enabled auto-merge January 19, 2023 18:27
@mitjat mitjat changed the title Basic e2e regression test suite Basic manual e2e regression test suite Jan 19, 2023
@mitjat mitjat force-pushed the mitjat/e2e_regression branch from 089bb4a to 7db584f Compare January 19, 2023 18:29
@mitjat mitjat merged commit c600084 into main Jan 19, 2023
@mitjat mitjat deleted the mitjat/e2e_regression branch January 19, 2023 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants