Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - when migrating to a new version and the mappings remain the same the migrator skips reindexing #152472

Closed
kibanamachine opened this issue Mar 1, 2023 · 6 comments · Fixed by #154446
Closed

Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - when migrating to a new version and the mappings remain the same the migrator skips reindexing #152472

kibanamachine opened this issue Mar 1, 2023 · 6 comments · Fixed by #154446
Labels
Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects failed-test A test failure on a tracked branch, potentially flaky-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Mar 1, 2023 •
edited
Loading

A test failed on a tracked branch

Error: expect(received).toMatch(expected)

Expected substring: "CHECK_VERSION_INDEX_READY_ACTIONS -> DONE."
Received string:    "{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T11:02:55.205-05:00\",\"message\":\"[.kibana_migrator_tests] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 5ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2608}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T11:02:55.208-05:00\",\"message\":\"[.kibana_migrator_tests] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 3ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2608}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T11:02:55.214-05:00\",\"message\":\"[.kibana_migrator_tests] Starting to process 8 documents.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2608}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T11:02:55.214-05:00\",\"message\":\"[.kibana_migrator_tests] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_TRANSFORM. took: 6ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2608}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T11:02:55.215-05:00\",\"message\":\"[.kibana_migrator_tests] OUTDATED_DOCUMENTS_TRANSFORM -> TRANSFORMED_DOCUMENTS_BULK_INDEX. took: 1ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2608}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T11:02:55.229-05:00\",\"message\":\"[.kibana_migrator_tests] TRANSFORMED_DOCUMENTS_BULK_INDEX -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 14ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2608}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T11:02:55.234-05:00\",\"message\":\"[.kibana_migrator_tests] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 5ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2608}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T11:02:55.238-05:00\",\"message\":\"[.kibana_migrator_tests] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> OUTDATED_DOCUMENTS_REFRESH. took: 4ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2608}}
{\"ecs\":{\"version\":\"8.6.1\"},\"@timestamp\":\"2023-03-01T11:02:55.261-05:00\",\"message\":\"[.kibana_migrator_tests] OUTDATED_DOCUMENTS_REFRESH -> CHECK_TARGET_MAPPINGS. took: 23ms.\",\"log\":{\"level\":\"INFO\",\"logger\":\"savedobjects-service\"},\"process\":{\"pid\":2608}}
"
    at Object.toMatch (/var/lib/buildkite-agent/builds/kb-n2-4-spot-1afb405c1edc6547/elastic/kibana-on-merge/kibana/src/core/server/integration_tests/saved_objects/migrations/group3/skip_reindex.test.ts:125:18)

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Mar 1, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Mar 1, 2023
@kibanamachine kibanamachine added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Mar 1, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Mar 1, 2023
@gsoldevila gsoldevila mentioned this issue Mar 7, 2023
Merged
gsoldevila added a commit that referenced this issue Mar 7, 2023
@gsoldevila @kibanamachine
[Kibana migrator test kit] Flush logger after running migrations (#15…
163dae7 
…2818)

This PR addresses flakiness of:

- #152472
- #152448

These tests fail because we have a race condition. We are checking
certain conditions in the log files, and sometimes we do it before the
logs are actually written.

With the Kibana migrator test kit, we can actually flush + wait for all
the logging appenders. This way, we are sure that the logs are complete
when we inspect them in the tests.

---------

Co-authored-by: kibanamachine <[email protected]>
gsoldevila added a commit to gsoldevila/kibana that referenced this issue Mar 8, 2023
@gsoldevila
[Kibana migrator test kit] Flush logger after running migrations (ela…
bd17a5b 
…stic#152818)

This PR addresses flakiness of:

- elastic#152472
- elastic#152448

These tests fail because we have a race condition. We are checking
certain conditions in the log files, and sometimes we do it before the
logs are actually written.

With the Kibana migrator test kit, we can actually flush + wait for all
the logging appenders. This way, we are sure that the logs are complete
when we inspect them in the tests.

---------

Co-authored-by: kibanamachine <[email protected]>
(cherry picked from commit 163dae7)
gsoldevila referenced this issue Mar 8, 2023
@gsoldevila
[8.7] [Kibana migrator test kit] Flush logger after running migrations (
e70452f 
#152818) (#152890)

# Backport

This will backport the following commits from `main` to `8.7`:
- [[Kibana migrator test kit] Flush logger after running migrations
(#152818)](#152818)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gerard
Soldevila","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-03-07T17:57:17Z","message":"[Kibana
migrator test kit] Flush logger after running migrations
(#152818)\n\nThis PR addresses flakiness of:\r\n\r\n-
https://github.com/elastic/kibana/issues/152472\r\n-
https://github.com/elastic/kibana/issues/152448\r\n\r\nThese tests fail
because we have a race condition. We are checking\r\ncertain conditions
in the log files, and sometimes we do it before the\r\nlogs are actually
written.\r\n\r\nWith the Kibana migrator test kit, we can actually flush
+ wait for all\r\nthe logging appenders. This way, we are sure that the
logs are complete\r\nwhen we inspect them in the
tests.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"163dae7154c11d92ff148bc6ac45cbea54680769","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","Team:Core","release_note:skip","test-failure-flaky","Feature:Migrations","backport:prev-minor","v8.7.0","v8.8.0"],"number":152818,"url":"https://github.com/elastic/kibana/pull/152818","mergeCommit":{"message":"[Kibana
migrator test kit] Flush logger after running migrations
(#152818)\n\nThis PR addresses flakiness of:\r\n\r\n-
https://github.com/elastic/kibana/issues/152472\r\n-
https://github.com/elastic/kibana/issues/152448\r\n\r\nThese tests fail
because we have a race condition. We are checking\r\ncertain conditions
in the log files, and sometimes we do it before the\r\nlogs are actually
written.\r\n\r\nWith the Kibana migrator test kit, we can actually flush
+ wait for all\r\nthe logging appenders. This way, we are sure that the
logs are complete\r\nwhen we inspect them in the
tests.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"163dae7154c11d92ff148bc6ac45cbea54680769"}},"sourceBranch":"main","suggestedTargetBranches":["8.7"],"targetPullRequestStates":[{"branch":"8.7","label":"v8.7.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/152818","number":152818,"mergeCommit":{"message":"[Kibana
migrator test kit] Flush logger after running migrations
(#152818)\n\nThis PR addresses flakiness of:\r\n\r\n-
https://github.com/elastic/kibana/issues/152472\r\n-
https://github.com/elastic/kibana/issues/152448\r\n\r\nThese tests fail
because we have a race condition. We are checking\r\ncertain conditions
in the log files, and sometimes we do it before the\r\nlogs are actually
written.\r\n\r\nWith the Kibana migrator test kit, we can actually flush
+ wait for all\r\nthe logging appenders. This way, we are sure that the
logs are complete\r\nwhen we inspect them in the
tests.\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"163dae7154c11d92ff148bc6ac45cbea54680769"}}]}]
BACKPORT-->
@gsoldevila gsoldevila added the Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects label Mar 8, 2023
bmorelli25 pushed a commit to bmorelli25/kibana that referenced this issue Mar 10, 2023
@gsoldevila @kibanamachine @bmorelli25
[Kibana migrator test kit] Flush logger after running migrations (ela…
17a4ede 
…stic#152818)

This PR addresses flakiness of:

- elastic#152472
- elastic#152448

These tests fail because we have a race condition. We are checking
certain conditions in the log files, and sometimes we do it before the
logs are actually written.

With the Kibana migrator test kit, we can actually flush + wait for all
the logging appenders. This way, we are sure that the logs are complete
when we inspect them in the tests.

---------

Co-authored-by: kibanamachine <[email protected]>
@kibanamachine kibanamachine reopened this Mar 22, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@gsoldevila gsoldevila mentioned this issue Mar 22, 2023
Merged
@mistic mistic closed this as completed Mar 22, 2023
mistic pushed a commit that referenced this issue Mar 22, 2023
@gsoldevila
Flush logger after running migrations, fix remaining test (#153450)
3f5961e 
Fixes flakiness on:
* #152448
* #152472

And also related PR builds failures, e.g.
#153332
([failure](https://s3.amazonaws.com/buildkiteartifacts.com/e0f3970e-3a75-4621-919f-e6c773e2bb12/0fda5127-f57f-42fb-8e5a-146b3d535916/018705a0-a7cc-422a-89a3-026a2637b9e4/018705a4-3025-4e3b-884c-ee805b529428/target/test_failures/018705a4-3025-4e3b-884c-ee805b529428_d47f90a974496415048dfbef2bc5134d.html?response-content-type=text%2Fhtml&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7LYVRIJ2HP%2F20230322%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230322T131425Z&X-Amz-Expires=600&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEN3%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJIMEYCIQDA9S4KjEvUes%2BBf46MNFrR91uVacUL4Fb6hFz2c%2BDUNAIhALC%2FK1HFIS%2FaFsiZeyn99na1OnoHESROe%2FPbjcYK%2F2RmKvoDCKX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQABoMMDMyMzc5NzA1MzAzIgyIyDrn3p%2Bf98NXvT4qzgM4eyTaC4EwuV5t7xCmqtUrfCy2lm9ftD3NmJplYEdeuQcOaKtd5Acz4hTEuReaDZYde1eoa9WgR%2BabHy5N3cWO1ndI%2FqfanO5CeUspza7kfaGg4jcDh1e68QAXRy2ciuK2hWWWw0OF8FErTYG3q5dmjOaHaLeUipG%2FdeWF4iIs4BLPVvJA3ZTNGTFPxwc6%2BtD1eJSobey%2Bi45tcTsrCkXm9UtuOSBtW1vYpFL2zkxLoDWV7m%2F%2F5MSNU53L4%2BWGp8bM4aFCo1fXa9IDYv5MQTn7z1CBi2qUn%2FJzAN0LlBwLaAYIDd7TdLN5c3FPzyoTGoknhQQh%2BB6G1qVmZtXwfSYHGYL7rOjv%2FmPHYHpwniyTKitGBHKGAvToHr%2B37wpFpBhpGzZoa8q4e6esRN%2F94wItt%2BoYTh%2FraQTxtqsG8PEZnUJ%2F2H%2FyeNck4rbePnirxVBTt1FAwABpM5TfJEFHd9EYESsSsJ6WIpv43Gc1XbOeiC6MUf502NfAlxhqayhJ1Rux6L9y%2FCI5wbVC9VwI9LrbsavgTt21re6TZgazqt6x9jTZYAqfAZXhIcoUXOreoEBsfqhSTf6RlDY%2FzJ31rDz0dAhXl0QiEZnp%2By%2F6S7cwyeHroAY6pAFByPZ8bWAwDobR8BwKUCZShocd7SssJQqhMyo5E%2FK5aYYPXo3uCijbUcKvsHnyqllCFpA1t5LtyPjoyxSRewnrEueEop%2FAAukBF7%2FrIVZEdwhgWzne5QqK2bV9SMP92RYpDQ8Pff9NrRsgk9qVTKfMdE%2BTKSDt0zBl5gguqn7DLKsJdjBfs4nnXrJlxAKh%2F%2BXpSvKSX2vNh%2F7KZaOEB5R1uuZCFw%3D%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=0a5787d518338968b49ac9bc085ad1b7d019b47be1ad2cb860ecc8b92e11e4a0))


[#152818](#152818) was a first
attempt at fixing this, but I forgot to update one of the tests.
@kibanamachine kibanamachine reopened this Mar 28, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@gsoldevila gsoldevila mentioned this issue Apr 5, 2023
Merged
gsoldevila added a commit that referenced this issue Apr 6, 2023
@gsoldevila
Make sure test kit migrators are run only once (#154446)
1f7aaff 
Fixes #152472 (hopefully)

In the failing test, we were reusing the _kibana migrator test kit_
migrator more than once.
This can lead to unexpected results, as the `runMigrations()` method
closes and flushes logging appenders when it is run.
This PR ensures that each migrator in the test kit is run only once.
@kibanamachine kibanamachine reopened this May 23, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@dokmic
Copy link
Contributor

dokmic commented Jul 28, 2023

This was reopened by #157802, which is unrelated to the failed test. The issue can be safely closed.

@dokmic dokmic closed this as completed Jul 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects failed-test A test failure on a tracked branch, potentially flaky-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make sure test kit migrators are run only once gsoldevila/kibana
5 participants
@mistic @dokmic @elasticmachine @gsoldevila @kibanamachine