Disable flakiest tests #14706
Closed
Disable flakiest tests #14706
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ebyhr
requested changes
Oct 21, 2022
| @@ -23,7 +23,8 @@ | |||
|
|
|||
| public class TestS3WrongRegionPicked | |||
| { | |||
| @Test | |||
| // Disabled due to flakiness per https://nineinchnick.github.io/trino-cicd/reports/flaky/ | |||
| @@ -125,7 +125,8 @@ public void testDatabricksCanReadTrinoCheckpoint() | |||
| } | |||
| } | |||
|
|
|||
| @Test(groups = {DELTA_LAKE_DATABRICKS, PROFILE_SPECIFIC_TESTS}) | |||
| // Disabled due to flakiness per https://nineinchnick.github.io/trino-cicd/reports/flaky/ | |||
| @Test(groups = {DELTA_LAKE_DATABRICKS, PROFILE_SPECIFIC_TESTS}, enabled = false) | |||
There was a problem hiding this comment.
I don't think disabling this test helps to fix Retry-After failure. The failed place isn't deterministic.
| @@ -79,7 +79,8 @@ private void testDatabricksCanReadFromCtasTableCreatedByTrinoWithSpecialCharacte | |||
| } | |||
| } | |||
|
|
|||
| @Test(groups = {DELTA_LAKE_DATABRICKS, PROFILE_SPECIFIC_TESTS}) | |||
| // Disabled due to flakiness per https://nineinchnick.github.io/trino-cicd/reports/flaky/ | |||
| @Test(groups = {DELTA_LAKE_DATABRICKS, PROFILE_SPECIFIC_TESTS}, enabled = false) | |||
There was a problem hiding this comment.
I don't think disabling this test helps to fix Retry-After failure. The failed place isn't deterministic.
ebyhr
reviewed
Oct 21, 2022
| @@ -716,6 +716,11 @@ public void testLimitPushdown() | |||
| assertThat(query("SELECT name FROM nation LIMIT 30")).isNotFullyPushedDown(LimitNode.class); // Use high limit for result determinism | |||
| } | |||
|
|
|||
| // Empty override is to disable test due to flakiness per https://nineinchnick.github.io/trino-cicd/reports/flaky/ | |||
There was a problem hiding this comment.
Please leave GitHub issue link instead of https://nineinchnick.github.io/trino-cicd/reports/flaky/. The link isn't helpful when we access the page in the future.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Using data from @nineinchnick's reports on flaky tests @ https://nineinchnick.github.io/trino-cicd/reports/flaky/, I've taken a look at the flakiest tests that have failed within the last week and disabled them to hopefully address some of our ongoing CI failure issues.
Four tests being disabled, and their error messages when failing:
TestS3WrongRegionPicked.testS3WrongRegionSelection
[Expecting throwable message: ]TestKuduConnectorTest.testAggregation
[Execution of 'actual' query failed: SELECT count(regionkey) FROM nation, Execution of 'actual' query failed: SELECT min(regionkey), max(regionkey) FROM nation, Execution of 'actual' query failed: SELECT regionkey, count(*) FROM nation GROUP BY regionkey]TestDeltaLakeDatabricksPartitioningCompatibility > testTrinoCanReadFromCtasTableCreatedByDatabricksWithSpecialCharactersInPartitioningColumn
[java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: HTTP retry after response received with no Retry-After header, error: HTTP Response code: 503, Error message: Unknown.]TestDeltaLakeDatabricksCheckpointsCompatibility > testTrinoUsesCheckpointInterval
[java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: HTTP retry after response received with no Retry-After header, error: HTTP Response code: 503, Error message: Unknown.]It looks like other tests in the DeltaLakeDatabricks compatibility tests have errored out in the past with a connection error, I'm not sure if these particular tests are more prone to this error or if they're just the ones getting unlucky as of late. It may not make much sense to disable them, would appreciate feedback on that.
Non-technical explanation
Disabling flaky tests.
Release notes
(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: