Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

transforms_stats#"Test Get Transform Stats" fails because pages_processed/checkpoint don't match #42309

Closed
polyfractal opened this issue May 21, 2019 · 5 comments · Fixed by #42351
Closed

transforms_stats#"Test Get Transform Stats" fails because pages_processed/checkpoint don't match #42309

polyfractal opened this issue May 21, 2019 · 5 comments · Fixed by #42351
Labels
:ml/Transform Transform >test-failure Triaged test failures from CI

Comments

@polyfractal
Copy link
Contributor

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/3614/

Reproduction:

./gradlew :x-pack:plugin:integTestRunner --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=data_frame/transforms_stats/Test get transform stats}" -Dtests.seed=5B30818AA97DC12A -Dtests.security.manager=true -Dtests.locale=en-KE -Dtests.timezone=Europe/Sarajevo -Dcompiler.java=12 -Druntime.java=11 -Dtests.rest.blacklist=getting_started/10_monitor_cluster_health/*

This failed on CI with:

java.lang.AssertionError: Failure at [data_frame/transforms_stats:53]: transforms.0.stats.pages_processed didn't match expected value:
transforms.0.stats.pages_processed: expected [0] but was [1]

It also fails for me locally, but with a different part of the test failing (checkpoint instead of the pages_processed). Fails reliably for me

Caused by:
java.lang.AssertionError: transforms.0.state.checkpoint didn't match expected value:
transforms.0.state.checkpoint: expected [0] but was [1]

Stash dump on test failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "count" : 1,
  1>       "transforms" : [
  1>         {
  1>           "id" : "airline-transform-stats",
  1>           "state" : {
  1>             "task_state" : "started",
  1>             "indexer_state" : "started",
  1>             "checkpoint" : 1,
  1>             "progress" : {
  1>               "total_docs" : 0,
  1>               "docs_remaining" : 0,
  1>               "percent_complete" : 100.0
  1>             }
  1>           },
  1>           "stats" : {
  1>             "pages_processed" : 1,
  1>             "documents_processed" : 0,
  1>             "documents_indexed" : 0,
  1>             "trigger_count" : 1,
  1>             "index_time_in_ms" : 0,
  1>             "index_total" : 0,
  1>             "index_failures" : 0,
  1>             "search_time_in_ms" : 8,
  1>             "search_total" : 1,
  1>             "search_failures" : 0
  1>           },
  1>           "checkpointing" : {
  1>             "in_progress" : {
  1>               "timestamp_millis" : 1558460091734
  1>             },
  1>             "operations_behind" : -1
  1>           }
  1>         }
  1>       ]
  1>     }
  1>   }
  1> }]

Looks like it started failing yesterday, so I'm guessing it's somehow related to #42067 /cc @davidkyle

image

Since it fails reliably I'm going to mute this.
@polyfractal polyfractal added >test-failure Triaged test failures from CI :ml/Transform Transform labels May 21, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core

@davidkyle
Copy link
Member

This should be fixed by #41942 which I've just merged. Please rebase and try again before muting

@polyfractal
Copy link
Contributor Author

Reset to current master, cleaned, re-ran test. Looks like it still fails :(

Happy to hand this over if you want to handle it, muting or not, etc :)

$ git rev-parse HEAD
be412ca83f0d4644712a7baa8a6330b1e72b2858

org.elasticsearch.xpack.test.rest.XPackRestIT > test {p0=data_frame/transforms_stats/Test get transform stats} FAILED
    java.lang.AssertionError: Failure at [data_frame/transforms_stats:52]: transforms.0.state.checkpoint didn't match expected value:
     transforms.0.state.checkpoint: expected [0] but was [1]

        Caused by:
        java.lang.AssertionError: transforms.0.state.checkpoint didn't match expected value:
         transforms.0.state.checkpoint: expected [0] but was [1]
REPRODUCE WITH: ./gradlew :x-pack:plugin:integTestRunner --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=data_frame/transforms_stats/Test get transform stats}" -Dtests.seed=5B30818AA97DC12A -Dtests.security.manager=true -Dtests.locale=en-KE -Dtests.timezone=Europe/Sarajevo -Dcompiler.java=12 -Druntime.java=11 -Dtests.rest.blacklist=getting_started/10_monitor_cluster_health/*

Suite: Test class org.elasticsearch.xpack.test.rest.XPackRestIT
  1> [2019-05-21T19:54:25,615][INFO ][o.e.x.t.r.XPackRestIT    ] [test] [p0=data_frame/transforms_stats/Test get transform stats] before test
  1> [2019-05-21T19:54:25,775][INFO ][o.e.x.t.r.XPackRestIT    ] [test] initializing REST clients against [http://[::1]:47477]
  1> [2019-05-21T19:54:26,443][INFO ][o.e.x.t.r.XPackRestIT    ] [test] initializing client, minimum es version [8.0.0], master version, [8.0.0], hosts [http://[::1]:47477]
  1> [2019-05-21T19:54:33,568][INFO ][o.e.x.t.r.XPackRestIT    ] [test] Stash dump on test failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "count" : 1,
  1>       "transforms" : [
  1>         {
  1>           "id" : "airline-transform-stats",
  1>           "state" : {
  1>             "task_state" : "started",
  1>             "indexer_state" : "started",
  1>             "checkpoint" : 1,
  1>             "progress" : {
  1>               "total_docs" : 0,
  1>               "docs_remaining" : 0,
  1>               "percent_complete" : 100.0
  1>             }
  1>           },
  1>           "stats" : {
  1>             "pages_processed" : 1,
  1>             "documents_processed" : 0,
  1>             "documents_indexed" : 0,
  1>             "trigger_count" : 1,
  1>             "index_time_in_ms" : 0,
  1>             "index_total" : 0,
  1>             "index_failures" : 0,
  1>             "search_time_in_ms" : 8,
  1>             "search_total" : 1,
  1>             "search_failures" : 0
  1>           },
  1>           "checkpointing" : {
  1>             "current" : {
  1>               "timestamp_millis" : 1558461273424
  1>             },
  1>             "operations_behind" : 0
  1>           }
  1>         }
  1>       ]
  1>     }
  1>   }
  1> }]

@jkakavas jkakavas mentioned this issue May 21, 2019
Merged
polyfractal added a commit that referenced this issue May 21, 2019
@polyfractal
Mute transforms_stats yaml test
07ab45a 
AwaitsFix #42309
@polyfractal
Copy link
Contributor Author

Went ahead and muted this.

@polyfractal
Copy link
Contributor Author

Also muting "Test get multiple transform stats where one does not have a task" which is failing with a related set of errors

java.lang.AssertionError: Failure at [data_frame/transforms_stats:166]: transforms.0.state.indexer_state didn't match expected value:
transforms.0.state.indexer_state: expected [started] but was [indexing]

 Stash dump on test failure [{
14:52:46   1>   "stash" : {
14:52:46   1>     "body" : {
14:52:46   1>       "count" : 2,
14:52:46   1>       "transforms" : [
14:52:46   1>         {
14:52:46   1>           "id" : "airline-transform-stats",
14:52:46   1>           "state" : {
14:52:46   1>             "task_state" : "started",
14:52:46   1>             "indexer_state" : "indexing",
14:52:46   1>             "checkpoint" : 1,
14:52:46   1>             "progress" : {
14:52:46   1>               "total_docs" : 0,
14:52:46   1>               "docs_remaining" : 0,
14:52:46   1>               "percent_complete" : 100.0
14:52:46   1>             }
14:52:46   1>           },
14:52:46   1>           "stats" : {
14:52:46   1>             "pages_processed" : 1,
14:52:46   1>             "documents_processed" : 0,
14:52:46   1>             "documents_indexed" : 0,
14:52:46   1>             "trigger_count" : 1,
14:52:46   1>             "index_time_in_ms" : 0,
14:52:46   1>             "index_total" : 0,
14:52:46   1>             "index_failures" : 0,
14:52:46   1>             "search_time_in_ms" : 2,
14:52:46   1>             "search_total" : 1,
14:52:46   1>             "search_failures" : 0
14:52:46   1>           },
14:52:46   1>           "checkpointing" : {
14:52:46   1>             "in_progress" : {
14:52:46   1>               "timestamp_millis" : 1558464256316
14:52:46   1>             },
14:52:46   1>             "operations_behind" : -1
14:52:46   1>           }
14:52:46   1>         },
14:52:46   1>         {
14:52:46   1>           "id" : "airline-transform-stats-dos",
14:52:46   1>           "state" : {
14:52:46   1>             "task_state" : "stopped",
14:52:46   1>             "indexer_state" : "stopped",
14:52:46   1>             "checkpoint" : 0
14:52:46   1>           },
14:52:46   1>           "stats" : {
14:52:46   1>             "pages_processed" : 0,
14:52:46   1>             "documents_processed" : 0,
14:52:46   1>             "documents_indexed" : 0,
14:52:46   1>             "trigger_count" : 0,
14:52:46   1>             "index_time_in_ms" : 0,
14:52:46   1>             "index_total" : 0,
14:52:46   1>             "index_failures" : 0,
14:52:46   1>             "search_time_in_ms" : 0,
14:52:46   1>             "search_total" : 0,
14:52:46   1>             "search_failures" : 0
14:52:46   1>           },
14:52:46   1>           "checkpointing" : {
14:52:46   1>             "operations_behind" : 0
14:52:46   1>           }
14:52:46   1>         }
14:52:46   1>       ]
14:52:46   1>     }
14:52:46   1>   }
14:52:46   1> }]

./gradlew :x-pack:plugin:integTestRunner --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=data_frame/transforms_stats/Test get multiple transform stats where one does not have a task}" -Dtests.seed=7856C8AD2A4F2538 -Dtests.security.manager=true -Dtests.locale=hsb -Dtests.timezone=Africa/Malabo -Dcompiler.java=12 -Druntime.java=12 -Dtests.rest.blacklist=getting_started/10_monitor_cluster_health/*

polyfractal added a commit that referenced this issue May 21, 2019
@polyfractal
Mute another transforms_stats yaml test
1dcaf4f 
AwaitsFix #42309
This was referenced May 22, 2019
Merged
Merged
davidkyle added a commit that referenced this issue May 23, 2019
@davidkyle
[ML Data Frame] Account for completed data frames in test (#42351)
da77b97 
When asserting on the checkpoint value if the DF has completed the checkpoint will be 1 else 0.
Similarly state may be started or indexing. Closes #42309
davidkyle added a commit that referenced this issue May 23, 2019
@davidkyle
[ML Data Frame] Account for completed data frames in test (#42351)
a23257c 
When asserting on the checkpoint value if the DF has completed the checkpoint will be 1 else 0.
Similarly state may be started or indexing. Closes #42309
davidkyle added a commit that referenced this issue May 23, 2019
@davidkyle
[ML Data Frame] Account for completed data frames in test (#42351)
28e0f98 
When asserting on the checkpoint value if the DF has completed the checkpoint will be 1 else 0.
Similarly state may be started or indexing. Closes #42309
gurkankaymak pushed a commit to gurkankaymak/elasticsearch that referenced this issue May 27, 2019
@polyfractal
Mute transforms_stats yaml test
9dc386b 
AwaitsFix elastic#42309
gurkankaymak pushed a commit to gurkankaymak/elasticsearch that referenced this issue May 27, 2019
@polyfractal
Mute another transforms_stats yaml test
e90d68d 
AwaitsFix elastic#42309
gurkankaymak pushed a commit to gurkankaymak/elasticsearch that referenced this issue May 27, 2019
@davidkyle
[ML Data Frame] Account for completed data frames in test (elastic#42351
4166fd2 
)

When asserting on the checkpoint value if the DF has completed the checkpoint will be 1 else 0.
Similarly state may be started or indexing. Closes elastic#42309
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:ml/Transform Transform >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML Data Frame] Account for completed data frames in test davidkyle/elasticsearch
3 participants
@polyfractal @davidkyle @elasticmachine