Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] RepositoryHdfsClientYamlTestSuiteIT test {yaml=hdfs_repository_2/30_snapshot/take snapshot} failing #84036

Closed
cbuescher opened this issue Feb 16, 2022 · 6 comments · Fixed by #84038, #84089 or #84210
Assignees
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI

Comments

@cbuescher
Copy link
Member

I tried reproduction a few times locally but without success.

Build scan:
https://gradle-enterprise.elastic.co/s/fngwbzlxxdsfa/tests/:plugins:repository-hdfs:integTest2/org.elasticsearch.repositories.hdfs.RepositoryHdfsClientYamlTestSuiteIT/test%20%7Byaml=hdfs_repository_2%2F30_snapshot%2Ftake%20snapshot%7D

Reproduction line:
./gradlew ':plugins:repository-hdfs:integTest2' --tests "org.elasticsearch.repositories.hdfs.RepositoryHdfsClientYamlTestSuiteIT.test {yaml=hdfs_repository_2/30_snapshot/take snapshot}" -Dtests.seed=FE97822C475ED786 -Dtests.locale=et-EE -Dtests.timezone=Asia/Riyadh -Druntime.java=17

Applicable branches:
master

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.repositories.hdfs.RepositoryHdfsClientYamlTestSuiteIT&tests.test=test%20%7Byaml%3Dhdfs_repository_2/30_snapshot/take%20snapshot%7D

Failure excerpt:

java.lang.AssertionError: Failure at [hdfs_repository_2/30_snapshot:39]: Response is missing required X-Elastic-Product response header

  at __randomizedtesting.SeedInfo.seed([FE97822C475ED786:76C3BDF6E9A2BA7E]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:493)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:462)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:833)

  Caused by: java.lang.AssertionError: Response is missing required X-Elastic-Product response header

    at org.junit.Assert.fail(Assert.java:88)
    at org.elasticsearch.test.rest.yaml.section.DoSection.checkElasticProductHeader(DoSection.java:398)
    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:370)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:482)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:462)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:568)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
    at java.lang.Thread.run(Thread.java:833)

@cbuescher cbuescher added :Core/Infra/REST API REST infrastructure and utilities >test-failure Triaged test failures from CI labels Feb 16, 2022
@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label Feb 16, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@DaveCTurner DaveCTurner self-assigned this Feb 16, 2022
@DaveCTurner DaveCTurner added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs and removed :Core/Infra/REST API REST infrastructure and utilities labels Feb 16, 2022
@elasticmachine elasticmachine added Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. and removed Team:Core/Infra Meta label for core/infra team labels Feb 16, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@mark-vieira
Copy link
Contributor

@DaveCTurner
Copy link
Contributor

DaveCTurner commented Feb 17, 2022

The failing assertion was added in #83290 but it's failing because there is a genuine bug that affects most of the snapshot APIs. It's intermittent because it only bites if the API call happens concurrently with another internal snapshot-related action. #84038 should fix it - just merged, backports are en route.

@DaveCTurner
Copy link
Contributor

Confusingly https://gradle-enterprise.elastic.co/s/cawtyt3bvhgfg seems to have failed for similar reasons on the very commit that was supposed to fix this. Reopening...

@jtibshirani
Copy link
Contributor

Reopening since this failure popped up several more times today:

@jtibshirani jtibshirani reopened this Feb 17, 2022
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Feb 22, 2022
In elastic#83290 we added an assertion that Elasticsearch returns the product
header in every REST response. Unfortunately this isn't always the case,
we found bugs in a couple of released versions and fixed them in elastic#84038
and elastic#84089. With this commit we skip the new assertion in the
known-buggy versions.

Closes elastic#84036 again.
elasticsearchmachine pushed a commit that referenced this issue Feb 22, 2022
In #83290 we added an assertion that Elasticsearch returns the product
header in every REST response. Unfortunately this isn't always the case,
we found bugs in a couple of released versions and fixed them in #84038
and #84089. With this commit we skip the new assertion in the
known-buggy versions.

Closes #84036 again.
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Feb 22, 2022
In elastic#83290 we added an assertion that Elasticsearch returns the product
header in every REST response. Unfortunately this isn't always the case,
we found bugs in a couple of released versions and fixed them in elastic#84038
and elastic#84089. With this commit we skip the new assertion in the
known-buggy versions.

Closes elastic#84036 again.
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Feb 22, 2022
In elastic#83290 we added an assertion that Elasticsearch returns the product
header in every REST response. Unfortunately this isn't always the case,
we found bugs in a couple of released versions and fixed them in elastic#84038
and elastic#84089. With this commit we skip the new assertion in the
known-buggy versions.

Closes elastic#84036 again.
elasticsearchmachine pushed a commit that referenced this issue Feb 22, 2022
In #83290 we added an assertion that Elasticsearch returns the product
header in every REST response. Unfortunately this isn't always the case,
we found bugs in a couple of released versions and fixed them in #84038
and #84089. With this commit we skip the new assertion in the
known-buggy versions.

Closes #84036 again.
elasticsearchmachine pushed a commit that referenced this issue Feb 22, 2022
In #83290 we added an assertion that Elasticsearch returns the product
header in every REST response. Unfortunately this isn't always the case,
we found bugs in a couple of released versions and fixed them in #84038
and #84089. With this commit we skip the new assertion in the
known-buggy versions.

Closes #84036 again.
probakowski pushed a commit to probakowski/elasticsearch that referenced this issue Feb 23, 2022
probakowski pushed a commit to probakowski/elasticsearch that referenced this issue Feb 23, 2022
In elastic#83290 we added an assertion that Elasticsearch returns the product
header in every REST response. Unfortunately this isn't always the case,
we found bugs in a couple of released versions and fixed them in elastic#84038
and elastic#84089. With this commit we skip the new assertion in the
known-buggy versions.

Closes elastic#84036 again.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI
Projects
None yet
5 participants