This repository has been archived by the owner on Sep 30, 2024. It is now read-only.
Add automated regression tests #4309
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Please use draft PRs to avoid sending notifications 🙇 |
beyang
force-pushed
the
bl/regression-test
branch
from
048c35c
to
2df9ea8
Compare
beyang
force-pushed
the
bl/regression-test
branch
from
2df9ea8
to
5692679
Compare
chrismwendt
approved these changes
Jun 3, 2019
There was a problem hiding this comment.
Nice, the Phabricator e2e tests could also use Driver. https://github.com/sourcegraph/sourcegraph/pull/4226
felixfbecker
suggested changes
Jun 4, 2019
beyang
force-pushed
the
bl/regression-test
branch
from
cf98054
to
31f5851
Compare
felixfbecker
approved these changes
Jun 5, 2019
| } | ||
| }) | ||
| }, | ||
| 5 * 1000 |
There was a problem hiding this comment.
Why do we pass the same timeout to every test if we also set the timeout at the top?
There was a problem hiding this comment.
That timeout applies generally, but the search test behavior is explicitly to timeout after 5s (as taking longer than 5s to serve the query indicates a broken state).
There was a problem hiding this comment.
This would have been good to mention in a comment - this timeout seems to have been copy-pasted for e.g. external service tests too
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a regression test file
regression.test.tsxthat defines regression tests. This is separate from theindex.e2e.test.tsxfile, which defines e2e tests that run in CI. The reason that the two are separate is that we expect there to be a lot of regression tests and it would be prohibitively expensive to run them each time in CI and they might be flakier than our tolerance for the in-CI e2e tests.Run it with
GITHUB_TOKEN=$TOKEN yarn run test-regressionin thewebdirectory.regression.test.tsxcurrently includes test for one of the search use cases, but will eventually grow to cover most of the Release test grid.cc @lguychard because you made a suggestion to use Cucumber.io. This PR does not use Cucumber. It may be the right solution for us down the road, but first I wanted to implement the tests directly in JS to see if this adequately addresses our needs.