Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] MixedClusterClientYamlTestSuiteIT test {p0=field_caps/10_basic/Get date_nanos field caps} failing #104383

Closed
maxhniebergall opened this issue Jan 15, 2024 · 7 comments
Closed

[CI] MixedClusterClientYamlTestSuiteIT test {p0=field_caps/10_basic/Get date_nanos field caps} failing #104383

maxhniebergall opened this issue Jan 15, 2024 · 7 comments
Assignees
@ldematte
Labels
:Core/Infra/Core Core issues without another label low-risk An open issue or test failure that is a low risk to future releases Team:Core/Infra Meta label for core/infra team >test-failure Triaged test failures from CI

Comments

@maxhniebergall
Copy link
Member

maxhniebergall commented Jan 15, 2024 •
edited
Loading

Build scan:
https://gradle-enterprise.elastic.co/s/gsj6swvfsz6aq/tests/:qa:mixed-cluster:v7.7.0%23mixedClusterTest/org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT/test%20%7Bp0=field_caps%2F10_basic%2FGet%20date_nanos%20field%20caps%7D

Reproduction line:

./gradlew ':qa:mixed-cluster:v7.7.0#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=field_caps/10_basic/Get date_nanos field caps}" -Dtests.seed=6E8A3C9370A6C697 -Dtests.bwc=true -Dtests.locale=es-EC -Dtests.timezone=SystemV/EST5EDT -Druntime.java=21

Applicable branches:
7.17

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT#test {p0=field_caps/10_basic/Get date_nanos field caps}

Failure excerpt:

java.lang.AssertionError: Failure at [field_caps/10_basic:177]: field [fields.date_nanos.date_nanos.searchable] is null

  at __randomizedtesting.SeedInfo.seed([6E8A3C9370A6C697:E6DE0349DE5AAB6F]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:507)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:480)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)

  Caused by: java.lang.AssertionError: field [fields.date_nanos.date_nanos.searchable] is null

    at org.junit.Assert.fail(Assert.java:89)
    at org.junit.Assert.assertTrue(Assert.java:42)
    at org.junit.Assert.assertNotNull(Assert.java:713)
    at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:78)
    at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:65)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:496)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:480)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke(Method.java:580)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:1583)
@maxhniebergall maxhniebergall added :Core/Infra/Core Core issues without another label >test-failure Triaged test failures from CI labels Jan 15, 2024
@elasticsearchmachine elasticsearchmachine added blocker Team:Core/Infra Meta label for core/infra team labels Jan 15, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@thecoop
Copy link
Member

thecoop commented Jan 16, 2024

Just to clarify, this is on 7.17, not main?

@maxhniebergall
Copy link
Member Author

@thecoop, I believe you're right, thanks!

@thecoop thecoop added medium-risk An open issue or test failure that is a medium risk to future releases and removed blocker labels Jan 16, 2024
@stu-elastic stu-elastic self-assigned this Jan 16, 2024
@stu-elastic
Copy link
Contributor

There are three other failures of this test, 7.1.0, 7.3.1 and 7.17.7. Those seem unrelated to this time around, resource already exists or using the old test framework
https://gradle-enterprise.elastic.co/s/wxete7xfh7he4
https://gradle-enterprise.elastic.co/s/dfte34zfdx7vq
https://gradle-enterprise.elastic.co/s/t2haqpkzl7r36

@stu-elastic
Copy link
Contributor

Does not reproduce locally, looks like the field is not created

[2024-01-15T08:39:05,555][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{	
  "stash" : {	
    "body" : {	
      "indices" : [	
        "test_nanos"	
      ],	
      "fields" : { }	
    }	
  }	
}]

Test:

"Get date_nanos field caps":
  - skip:
      version: " - 6.99.99"
      reason: date_nanos field mapping type has been introcued in 7.0

  - do:
        indices.create:
          include_type_name: false
          index: test_nanos
          body:
              mappings:
                properties:
                  date_nanos:
                    type:     date_nanos

  - do:
      field_caps:
        index: 'test_nanos'
        fields: [date_nanos]

  - match: {fields.date_nanos.date_nanos.searchable:                true}
  - match: {fields.date_nanos.date_nanos.aggregatable:              true}
  - is_false: fields.date_nanos.date_nanos.indices
  - is_false: fields.date_nanos.date_nanos.non_searchable_indices
  - is_false: fields.date_nanos.date_nanos.non_aggregatable_indices

@stu-elastic stu-elastic removed their assignment Jan 16, 2024
@ldematte ldematte self-assigned this Mar 19, 2024
@ldematte
Copy link
Contributor

Looking at past issues, I found this to be pretty close: #80335

That was solved by adding in the setup a check to wait on cluster state and be sure shards are "stable":

  - do:
      cluster.health:
        wait_for_no_relocating_shards: true
        wait_for_no_initializing_shards: true
        wait_for_events: languid

See #82716

Here in this test we are creating another index test_nanos, so we might want to do the same after (wait_for_no_relocating_shards: true + wait_for_no_initializing_shards: true).

I'm not seeing failures in 8.x, and it seems this test if failing very seldomly, but if we get another failure we may want to try that.

@ldematte ldematte added low-risk An open issue or test failure that is a low risk to future releases and removed medium-risk An open issue or test failure that is a medium risk to future releases labels Mar 19, 2024
@ldematte
Copy link
Contributor

There have been no failures for the past 3 months, I'm closing this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ldematte ldematte

Labels
:Core/Infra/Core Core issues without another label low-risk An open issue or test failure that is a low risk to future releases Team:Core/Infra Meta label for core/infra team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ldematte @thecoop @stu-elastic @elasticsearchmachine @maxhniebergall