-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update archives to 8.17.0 binary data format, unskip tests #196641
Merged
gsoldevila
merged 3 commits into
elastic:main
from
gsoldevila:kbn-167676-update-migration-archives-to-8.17.0
Oct 17, 2024
Merged
Update archives to 8.17.0 binary data format, unskip tests #196641
gsoldevila
merged 3 commits into
elastic:main
from
gsoldevila:kbn-167676-update-migration-archives-to-8.17.0
Oct 17, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gsoldevila
added
Team:Core
Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
release_note:skip
Skip the PR/issue when compiling release notes
Feature:Migrations
backport:prev-minor
Backport to (8.x) the previous minor version (i.e. one version back from main)
labels
Oct 17, 2024
Pinging @elastic/kibana-core (Team:Core) |
afharo
approved these changes
Oct 17, 2024
gsoldevila
changed the title
Update archives to 8.17.0 binary data format, unskip test
Update archives to 8.17.0 binary data format, unskip tests
Oct 17, 2024
Starting backport for target branches: 8.x https://github.com/elastic/kibana/actions/runs/11383438936 |
💚 Build Succeeded
Metrics [docs]
|
💔 All backports failed
Manual backportTo create the backport manually run:
Questions ?Please refer to the Backport tool documentation |
This was referenced Oct 17, 2024
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
gsoldevila
added a commit
to gsoldevila/kibana
that referenced
this pull request
Oct 17, 2024
…6641) ## Summary Addresses elastic#167676 Addresses elastic#158318 Addresses elastic#163254 Addresses elastic#163255 #### Fix for `multiple_es_nodes.test.ts` Inspired on elastic#193899 1. Start both nodes of ES 8.17.0 with the affected data-archive on separate terminals: 1. Node 01: `yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01` 2. Node 02: `yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02` 2. After ES is ready (without starting Kibana), reindex the index `.kibana_7.13.0_002` 1. Retrieve the settings from the original index via `curl -L 'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''` 2. Create the target index with those settings: ```shell curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "mappings": { "properties": { "bar": { "properties": { "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "coreMigrationVersion": { "type": "keyword" }, "foo": { "properties": { "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "migrationVersion": { "dynamic": "true", "properties": { "bar": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "foo": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "namespace": { "type": "keyword" }, "namespaces": { "type": "keyword" }, "originId": { "type": "keyword" }, "references": { "type": "nested", "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "type": { "type": "keyword" } } }, "type": { "type": "keyword" }, "updated_at": { "type": "date" } } }, "settings": { "index": { "hidden": "true", "number_of_shards": "1", "number_of_replicas": "0" } } }' ``` 3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": { "index": ".kibana_7.13.0_002" }, "dest": { "index": ".kibana_7.13.0_003" } }'` 4. Remove the old index and recreate the aliases ```shell curl -L 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "actions": [ { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0_001" } }, { "remove_index": {"index": ".kibana_7.13.0_002" } }, { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0" } }, { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana" } } ] }' ``` 3. Stop both ES nodes. 4. Compress both archives ```shell cd .es/node01/8.17.0 rm -rf data/nodes # we need to remove this dir or it fails to start again zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data -x "*/\.*" cd ../../../ cd .es/node02/8.17.0 rm -rf data/nodes # we need to remove this dir or it fails to start again zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data -x "*/\.*" cd ../../../ ``` 5. Run the tests to confirm that the issue is fixed: `yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts` #### Fix for `incompatible_cluster_routing_allocation.test.ts` Inspired on elastic#193741 ```shell # 1. Start ES 8.17.0 with the affected data-archive yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip # ... after ES has completely started up, stop it. # 2. Compress the archive cd .es/8.17.0 zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data -x "*/\.*" cd ../../ # 3. Run the tests to confirm that the issue is fixed. yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts ``` #### Fix for `read_batch_size.test.ts` Inspired on elastic#193899 ```shell # 1. Start ES 8.17.0 with the affected data-archive yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip # ... after ES has completely started up, stop it. # 2. Compress the archive cd .es/8.17.0 zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data -x "*/\.*" cd ../../ # 3. Run the tests to confirm that the issue is fixed. yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts ``` (cherry picked from commit 3d254c2) # Conflicts: # src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts
gsoldevila
added a commit
that referenced
this pull request
Oct 17, 2024
) (#196706) # Backport This will backport the following commits from `main` to `8.x`: - [Update archives to 8.17.0 binary data format, unskip test (#196641)](#196641) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gerard Soldevila","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-17T10:49:11Z","message":"Update archives to 8.17.0 binary data format, unskip test (#196641)\n\n## Summary\r\n\r\nAddresses https://github.com/elastic/kibana/issues/167676\r\nAddresses https://github.com/elastic/kibana/issues/158318\r\nAddresses https://github.com/elastic/kibana/issues/163254\r\nAddresses https://github.com/elastic/kibana/issues/163255\r\n\r\n#### Fix for `multiple_es_nodes.test.ts`\r\nInspired on https://github.com/elastic/kibana/pull/193899\r\n\r\n1. Start both nodes of ES 8.17.0 with the affected data-archive on\r\nseparate terminals:\r\n1. Node 01: `yarn es snapshot --version=8.17.0 --data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\n--base-path .es/node01`\r\n2. Node 02: `yarn es snapshot --version=8.17.0 --data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\n--base-path .es/node02`\r\n\r\n2. After ES is ready (without starting Kibana), reindex the index\r\n`.kibana_7.13.0_002`\r\n1. Retrieve the settings from the original index via `curl -L\r\n'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type:\r\napplication/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic\r\nc3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''`\r\n\t2. Create the target index with those settings: \r\n\t```shell\r\ncurl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H\r\n'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization:\r\nBasic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{\r\n \"mappings\": {\r\n \"properties\": {\r\n \"bar\": {\r\n \"properties\": {\r\n \"status\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"coreMigrationVersion\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"foo\": {\r\n \"properties\": {\r\n \"status\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"migrationVersion\": {\r\n \"dynamic\": \"true\",\r\n \"properties\": {\r\n \"bar\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n },\r\n \"foo\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"namespace\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"namespaces\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"originId\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"references\": {\r\n \"type\": \"nested\",\r\n \"properties\": {\r\n \"id\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"name\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"type\": {\r\n \"type\": \"keyword\"\r\n }\r\n }\r\n },\r\n \"type\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"updated_at\": {\r\n \"type\": \"date\"\r\n }\r\n }\r\n },\r\n \"settings\": {\r\n \"index\": {\r\n \"hidden\": \"true\",\r\n \"number_of_shards\": \"1\",\r\n \"number_of_replicas\": \"0\"\r\n }\r\n }\r\n }'\r\n ```\r\n3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H\r\n'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization:\r\nBasic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ \"source\": {\r\n\"index\": \".kibana_7.13.0_002\" }, \"dest\": { \"index\": \".kibana_7.13.0_003\"\r\n} }'`\r\n 4. Remove the old index and recreate the aliases\r\n ```shell\r\ncurl -L 'http://localhost:9200/_aliases' -H 'Content-Type:\r\napplication/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic\r\nc3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{\r\n \"actions\": [\r\n{ \"add\": { \"index\": \".kibana_7.13.0_003\", \"alias\": \".kibana_7.13.0_001\"\r\n} },\r\n { \"remove_index\": {\"index\": \".kibana_7.13.0_002\" } },\r\n{ \"add\": { \"index\": \".kibana_7.13.0_003\", \"alias\": \".kibana_7.13.0\" } },\r\n { \"add\": { \"index\": \".kibana_7.13.0_003\", \"alias\": \".kibana\" } }\r\n ]\r\n}'\r\n\t```\r\n3. Stop both ES nodes.\r\n4. Compress both archives\r\n\t```shell\r\n\tcd .es/node01/8.17.0\r\nrm -rf data/nodes # we need to remove this dir or it fails to start\r\nagain\r\nzip -r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\ndata -x \"*/\\.*\"\r\n\tcd ../../../\r\n\tcd .es/node02/8.17.0\r\nrm -rf data/nodes # we need to remove this dir or it fails to start\r\nagain\r\nzip -r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\ndata -x \"*/\\.*\"\r\n\tcd ../../../\r\n\t```\r\n5. Run the tests to confirm that the issue is fixed: `yarn\r\ntest:jest_integration\r\nsrc/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts`\r\n\r\n#### Fix for `incompatible_cluster_routing_allocation.test.ts`\r\nInspired on https://github.com/elastic/kibana/pull/193741\r\n\r\n```shell\r\n# 1. Start ES 8.17.0 with the affected data-archive\r\nyarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip\r\n# ... after ES has completely started up, stop it.\r\n\r\n# 2. Compress the archive\r\ncd .es/8.17.0\r\nzip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data -x \"*/\\.*\"\r\ncd ../../\r\n\r\n# 3. Run the tests to confirm that the issue is fixed.\r\nyarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts\r\n```\r\n\r\n#### Fix for `read_batch_size.test.ts`\r\nInspired on https://github.com/elastic/kibana/pull/193899\r\n\r\n```shell\r\n# 1. Start ES 8.17.0 with the affected data-archive\r\nyarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip\r\n# ... after ES has completely started up, stop it.\r\n\r\n# 2. Compress the archive\r\ncd .es/8.17.0\r\nzip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data -x \"*/\\.*\"\r\ncd ../../\r\n\r\n# 3. Run the tests to confirm that the issue is fixed.\r\nyarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts\r\n```","sha":"3d254c223f101a6ee326b57d45b9997a87eb12da","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","v9.0.0","Feature:Migrations","backport:prev-minor"],"number":196641,"url":"https://github.com/elastic/kibana/pull/196641","mergeCommit":{"message":"Update archives to 8.17.0 binary data format, unskip test (#196641)\n\n## Summary\r\n\r\nAddresses https://github.com/elastic/kibana/issues/167676\r\nAddresses https://github.com/elastic/kibana/issues/158318\r\nAddresses https://github.com/elastic/kibana/issues/163254\r\nAddresses https://github.com/elastic/kibana/issues/163255\r\n\r\n#### Fix for `multiple_es_nodes.test.ts`\r\nInspired on https://github.com/elastic/kibana/pull/193899\r\n\r\n1. Start both nodes of ES 8.17.0 with the affected data-archive on\r\nseparate terminals:\r\n1. Node 01: `yarn es snapshot --version=8.17.0 --data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\n--base-path .es/node01`\r\n2. Node 02: `yarn es snapshot --version=8.17.0 --data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\n--base-path .es/node02`\r\n\r\n2. After ES is ready (without starting Kibana), reindex the index\r\n`.kibana_7.13.0_002`\r\n1. Retrieve the settings from the original index via `curl -L\r\n'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type:\r\napplication/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic\r\nc3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''`\r\n\t2. Create the target index with those settings: \r\n\t```shell\r\ncurl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H\r\n'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization:\r\nBasic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{\r\n \"mappings\": {\r\n \"properties\": {\r\n \"bar\": {\r\n \"properties\": {\r\n \"status\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"coreMigrationVersion\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"foo\": {\r\n \"properties\": {\r\n \"status\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"migrationVersion\": {\r\n \"dynamic\": \"true\",\r\n \"properties\": {\r\n \"bar\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n },\r\n \"foo\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"namespace\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"namespaces\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"originId\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"references\": {\r\n \"type\": \"nested\",\r\n \"properties\": {\r\n \"id\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"name\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"type\": {\r\n \"type\": \"keyword\"\r\n }\r\n }\r\n },\r\n \"type\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"updated_at\": {\r\n \"type\": \"date\"\r\n }\r\n }\r\n },\r\n \"settings\": {\r\n \"index\": {\r\n \"hidden\": \"true\",\r\n \"number_of_shards\": \"1\",\r\n \"number_of_replicas\": \"0\"\r\n }\r\n }\r\n }'\r\n ```\r\n3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H\r\n'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization:\r\nBasic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ \"source\": {\r\n\"index\": \".kibana_7.13.0_002\" }, \"dest\": { \"index\": \".kibana_7.13.0_003\"\r\n} }'`\r\n 4. Remove the old index and recreate the aliases\r\n ```shell\r\ncurl -L 'http://localhost:9200/_aliases' -H 'Content-Type:\r\napplication/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic\r\nc3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{\r\n \"actions\": [\r\n{ \"add\": { \"index\": \".kibana_7.13.0_003\", \"alias\": \".kibana_7.13.0_001\"\r\n} },\r\n { \"remove_index\": {\"index\": \".kibana_7.13.0_002\" } },\r\n{ \"add\": { \"index\": \".kibana_7.13.0_003\", \"alias\": \".kibana_7.13.0\" } },\r\n { \"add\": { \"index\": \".kibana_7.13.0_003\", \"alias\": \".kibana\" } }\r\n ]\r\n}'\r\n\t```\r\n3. Stop both ES nodes.\r\n4. Compress both archives\r\n\t```shell\r\n\tcd .es/node01/8.17.0\r\nrm -rf data/nodes # we need to remove this dir or it fails to start\r\nagain\r\nzip -r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\ndata -x \"*/\\.*\"\r\n\tcd ../../../\r\n\tcd .es/node02/8.17.0\r\nrm -rf data/nodes # we need to remove this dir or it fails to start\r\nagain\r\nzip -r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\ndata -x \"*/\\.*\"\r\n\tcd ../../../\r\n\t```\r\n5. Run the tests to confirm that the issue is fixed: `yarn\r\ntest:jest_integration\r\nsrc/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts`\r\n\r\n#### Fix for `incompatible_cluster_routing_allocation.test.ts`\r\nInspired on https://github.com/elastic/kibana/pull/193741\r\n\r\n```shell\r\n# 1. Start ES 8.17.0 with the affected data-archive\r\nyarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip\r\n# ... after ES has completely started up, stop it.\r\n\r\n# 2. Compress the archive\r\ncd .es/8.17.0\r\nzip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data -x \"*/\\.*\"\r\ncd ../../\r\n\r\n# 3. Run the tests to confirm that the issue is fixed.\r\nyarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts\r\n```\r\n\r\n#### Fix for `read_batch_size.test.ts`\r\nInspired on https://github.com/elastic/kibana/pull/193899\r\n\r\n```shell\r\n# 1. Start ES 8.17.0 with the affected data-archive\r\nyarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip\r\n# ... after ES has completely started up, stop it.\r\n\r\n# 2. Compress the archive\r\ncd .es/8.17.0\r\nzip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data -x \"*/\\.*\"\r\ncd ../../\r\n\r\n# 3. Run the tests to confirm that the issue is fixed.\r\nyarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts\r\n```","sha":"3d254c223f101a6ee326b57d45b9997a87eb12da"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/196641","number":196641,"mergeCommit":{"message":"Update archives to 8.17.0 binary data format, unskip test (#196641)\n\n## Summary\r\n\r\nAddresses https://github.com/elastic/kibana/issues/167676\r\nAddresses https://github.com/elastic/kibana/issues/158318\r\nAddresses https://github.com/elastic/kibana/issues/163254\r\nAddresses https://github.com/elastic/kibana/issues/163255\r\n\r\n#### Fix for `multiple_es_nodes.test.ts`\r\nInspired on https://github.com/elastic/kibana/pull/193899\r\n\r\n1. Start both nodes of ES 8.17.0 with the affected data-archive on\r\nseparate terminals:\r\n1. Node 01: `yarn es snapshot --version=8.17.0 --data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\n--base-path .es/node01`\r\n2. Node 02: `yarn es snapshot --version=8.17.0 --data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\n--base-path .es/node02`\r\n\r\n2. After ES is ready (without starting Kibana), reindex the index\r\n`.kibana_7.13.0_002`\r\n1. Retrieve the settings from the original index via `curl -L\r\n'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type:\r\napplication/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic\r\nc3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''`\r\n\t2. Create the target index with those settings: \r\n\t```shell\r\ncurl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H\r\n'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization:\r\nBasic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{\r\n \"mappings\": {\r\n \"properties\": {\r\n \"bar\": {\r\n \"properties\": {\r\n \"status\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"coreMigrationVersion\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"foo\": {\r\n \"properties\": {\r\n \"status\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"migrationVersion\": {\r\n \"dynamic\": \"true\",\r\n \"properties\": {\r\n \"bar\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n },\r\n \"foo\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"namespace\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"namespaces\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"originId\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"references\": {\r\n \"type\": \"nested\",\r\n \"properties\": {\r\n \"id\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"name\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"type\": {\r\n \"type\": \"keyword\"\r\n }\r\n }\r\n },\r\n \"type\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"updated_at\": {\r\n \"type\": \"date\"\r\n }\r\n }\r\n },\r\n \"settings\": {\r\n \"index\": {\r\n \"hidden\": \"true\",\r\n \"number_of_shards\": \"1\",\r\n \"number_of_replicas\": \"0\"\r\n }\r\n }\r\n }'\r\n ```\r\n3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H\r\n'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization:\r\nBasic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ \"source\": {\r\n\"index\": \".kibana_7.13.0_002\" }, \"dest\": { \"index\": \".kibana_7.13.0_003\"\r\n} }'`\r\n 4. Remove the old index and recreate the aliases\r\n ```shell\r\ncurl -L 'http://localhost:9200/_aliases' -H 'Content-Type:\r\napplication/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic\r\nc3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{\r\n \"actions\": [\r\n{ \"add\": { \"index\": \".kibana_7.13.0_003\", \"alias\": \".kibana_7.13.0_001\"\r\n} },\r\n { \"remove_index\": {\"index\": \".kibana_7.13.0_002\" } },\r\n{ \"add\": { \"index\": \".kibana_7.13.0_003\", \"alias\": \".kibana_7.13.0\" } },\r\n { \"add\": { \"index\": \".kibana_7.13.0_003\", \"alias\": \".kibana\" } }\r\n ]\r\n}'\r\n\t```\r\n3. Stop both ES nodes.\r\n4. Compress both archives\r\n\t```shell\r\n\tcd .es/node01/8.17.0\r\nrm -rf data/nodes # we need to remove this dir or it fails to start\r\nagain\r\nzip -r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\ndata -x \"*/\\.*\"\r\n\tcd ../../../\r\n\tcd .es/node02/8.17.0\r\nrm -rf data/nodes # we need to remove this dir or it fails to start\r\nagain\r\nzip -r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\ndata -x \"*/\\.*\"\r\n\tcd ../../../\r\n\t```\r\n5. Run the tests to confirm that the issue is fixed: `yarn\r\ntest:jest_integration\r\nsrc/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts`\r\n\r\n#### Fix for `incompatible_cluster_routing_allocation.test.ts`\r\nInspired on https://github.com/elastic/kibana/pull/193741\r\n\r\n```shell\r\n# 1. Start ES 8.17.0 with the affected data-archive\r\nyarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip\r\n# ... after ES has completely started up, stop it.\r\n\r\n# 2. Compress the archive\r\ncd .es/8.17.0\r\nzip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data -x \"*/\\.*\"\r\ncd ../../\r\n\r\n# 3. Run the tests to confirm that the issue is fixed.\r\nyarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts\r\n```\r\n\r\n#### Fix for `read_batch_size.test.ts`\r\nInspired on https://github.com/elastic/kibana/pull/193899\r\n\r\n```shell\r\n# 1. Start ES 8.17.0 with the affected data-archive\r\nyarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip\r\n# ... after ES has completely started up, stop it.\r\n\r\n# 2. Compress the archive\r\ncd .es/8.17.0\r\nzip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data -x \"*/\\.*\"\r\ncd ../../\r\n\r\n# 3. Run the tests to confirm that the issue is fixed.\r\nyarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts\r\n```","sha":"3d254c223f101a6ee326b57d45b9997a87eb12da"}}]}] BACKPORT-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
backport:prev-minor
Backport to (8.x) the previous minor version (i.e. one version back from main)
Feature:Migrations
release_note:skip
Skip the PR/issue when compiling release notes
Team:Core
Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
v8.17.0
v9.0.0
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Addresses #167676
Addresses #158318
Addresses #163254
Addresses #163255
Fix for
multiple_es_nodes.test.ts
Inspired on #193899
Start both nodes of ES 8.17.0 with the affected data-archive on separate terminals:
yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01
yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02
After ES is ready (without starting Kibana), reindex the index
.kibana_7.13.0_002
curl -L 'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''
curl -L 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
"actions": [
{ "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0_001" } },
{ "remove_index": {"index": ".kibana_7.13.0_002" } },
{ "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0" } },
{ "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana" } }
]
}'
3. Stop both ES nodes. 4. Compress both archives
shellcd .es/node01/8.17.0
rm -rf data/nodes # we need to remove this dir or it fails to start again
zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data -x "/."
cd ../../../
cd .es/node02/8.17.0
rm -rf data/nodes # we need to remove this dir or it fails to start again
zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data -x "/."
cd ../../../
```
5. Run the tests to confirm that the issue is fixed:
yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts
Fix for
incompatible_cluster_routing_allocation.test.ts
Inspired on #193741
Fix for
read_batch_size.test.ts
Inspired on #193899