Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update archives to 8.17.0 binary data format, unskip tests #196641

Conversation

gsoldevila
Copy link
Contributor

@gsoldevila gsoldevila commented Oct 17, 2024

Summary

Addresses #167676
Addresses #158318
Addresses #163254
Addresses #163255

Fix for multiple_es_nodes.test.ts

Inspired on #193899

  1. Start both nodes of ES 8.17.0 with the affected data-archive on separate terminals:

    1. Node 01: yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01
    2. Node 02: yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02
  2. After ES is ready (without starting Kibana), reindex the index .kibana_7.13.0_002

    1. Retrieve the settings from the original index via curl -L 'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''
    2. Create the target index with those settings:
    curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
      "mappings": {
        "properties": {
          "bar": {
            "properties": {
              "status": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "coreMigrationVersion": {
            "type": "keyword"
          },
          "foo": {
            "properties": {
              "status": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "migrationVersion": {
            "dynamic": "true",
            "properties": {
              "bar": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "foo": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "namespace": {
            "type": "keyword"
          },
          "namespaces": {
            "type": "keyword"
          },
          "originId": {
            "type": "keyword"
          },
          "references": {
            "type": "nested",
            "properties": {
              "id": {
                "type": "keyword"
              },
              "name": {
                "type": "keyword"
              },
              "type": {
                "type": "keyword"
              }
            }
          },
          "type": {
            "type": "keyword"
          },
          "updated_at": {
            "type": "date"
          }
        }
      },
      "settings": {
        "index": {
            "hidden": "true",
            "number_of_shards": "1",
            "number_of_replicas": "0"
        }
      }
    }'
         ```
    3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": { "index": ".kibana_7.13.0_002" }, "dest": { "index": ".kibana_7.13.0_003" } }'`
    4. Remove the old index and recreate the aliases
    ```shell

curl -L 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
"actions": [
{ "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0_001" } },
{ "remove_index": {"index": ".kibana_7.13.0_002" } },
{ "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0" } },
{ "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana" } }
]
}'
3. Stop both ES nodes. 4. Compress both archivesshell
cd .es/node01/8.17.0
rm -rf data/nodes # we need to remove this dir or it fails to start again
zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data -x "/."
cd ../../../
cd .es/node02/8.17.0
rm -rf data/nodes # we need to remove this dir or it fails to start again
zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data -x "/."
cd ../../../
```
5. Run the tests to confirm that the issue is fixed: yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts

Fix for incompatible_cluster_routing_allocation.test.ts

Inspired on #193741

# 1. Start ES 8.17.0 with the affected data-archive
yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip
# ... after ES has completely started up, stop it.

# 2. Compress the archive
cd .es/8.17.0
zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data  -x "*/\.*"
cd ../../

# 3. Run the tests to confirm that the issue is fixed.
yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts

Fix for read_batch_size.test.ts

Inspired on #193899

# 1. Start ES 8.17.0 with the affected data-archive
yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip
# ... after ES has completely started up, stop it.

# 2. Compress the archive
cd .es/8.17.0
zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data  -x "*/\.*"
cd ../../

# 3. Run the tests to confirm that the issue is fixed.
yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts

@gsoldevila gsoldevila added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc release_note:skip Skip the PR/issue when compiling release notes Feature:Migrations backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) labels Oct 17, 2024
@gsoldevila gsoldevila requested a review from a team as a code owner October 17, 2024 08:58
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@gsoldevila gsoldevila enabled auto-merge (squash) October 17, 2024 08:59
@gsoldevila gsoldevila changed the title Update archives to 8.17.0 binary data format, unskip test Update archives to 8.17.0 binary data format, unskip tests Oct 17, 2024
@gsoldevila gsoldevila merged commit 3d254c2 into elastic:main Oct 17, 2024
32 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11383438936

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
8.x Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 196641

Questions ?

Please refer to the Backport tool documentation

@gsoldevila
Copy link
Contributor Author

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

gsoldevila added a commit to gsoldevila/kibana that referenced this pull request Oct 17, 2024
…6641)

## Summary

Addresses elastic#167676
Addresses elastic#158318
Addresses elastic#163254
Addresses elastic#163255

#### Fix for `multiple_es_nodes.test.ts`
Inspired on elastic#193899

1. Start both nodes of ES 8.17.0 with the affected data-archive on
separate terminals:
1. Node 01: `yarn es snapshot --version=8.17.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip
--base-path .es/node01`
2. Node 02: `yarn es snapshot --version=8.17.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip
--base-path .es/node02`

2. After ES is ready (without starting Kibana), reindex the index
`.kibana_7.13.0_002`
1. Retrieve the settings from the original index via `curl -L
'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type:
application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic
c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''`
	2. Create the target index with those settings:
	```shell
curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H
'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization:
Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
      "mappings": {
        "properties": {
          "bar": {
            "properties": {
              "status": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "coreMigrationVersion": {
            "type": "keyword"
          },
          "foo": {
            "properties": {
              "status": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "migrationVersion": {
            "dynamic": "true",
            "properties": {
              "bar": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "foo": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "namespace": {
            "type": "keyword"
          },
          "namespaces": {
            "type": "keyword"
          },
          "originId": {
            "type": "keyword"
          },
          "references": {
            "type": "nested",
            "properties": {
              "id": {
                "type": "keyword"
              },
              "name": {
                "type": "keyword"
              },
              "type": {
                "type": "keyword"
              }
            }
          },
          "type": {
            "type": "keyword"
          },
          "updated_at": {
            "type": "date"
          }
        }
      },
      "settings": {
        "index": {
            "hidden": "true",
            "number_of_shards": "1",
            "number_of_replicas": "0"
        }
      }
    }'
         ```
3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H
'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization:
Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": {
"index": ".kibana_7.13.0_002" }, "dest": { "index": ".kibana_7.13.0_003"
} }'`
    4. Remove the old index and recreate the aliases
    ```shell
curl -L 'http://localhost:9200/_aliases' -H 'Content-Type:
application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic
c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
  "actions": [
{ "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0_001"
} },
    { "remove_index": {"index": ".kibana_7.13.0_002" } },
{ "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0" } },
    { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana" } }
  ]
}'
	```
3. Stop both ES nodes.
4. Compress both archives
	```shell
	cd .es/node01/8.17.0
rm -rf data/nodes # we need to remove this dir or it fails to start
again
zip -r
../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip
data -x "*/\.*"
	cd ../../../
	cd .es/node02/8.17.0
rm -rf data/nodes # we need to remove this dir or it fails to start
again
zip -r
../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip
data -x "*/\.*"
	cd ../../../
	```
5. Run the tests to confirm that the issue is fixed: `yarn
test:jest_integration
src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts`

#### Fix for `incompatible_cluster_routing_allocation.test.ts`
Inspired on elastic#193741

```shell
# 1. Start ES 8.17.0 with the affected data-archive
yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip
# ... after ES has completely started up, stop it.

# 2. Compress the archive
cd .es/8.17.0
zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data  -x "*/\.*"
cd ../../

# 3. Run the tests to confirm that the issue is fixed.
yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts
```

#### Fix for `read_batch_size.test.ts`
Inspired on elastic#193899

```shell
# 1. Start ES 8.17.0 with the affected data-archive
yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip
# ... after ES has completely started up, stop it.

# 2. Compress the archive
cd .es/8.17.0
zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data  -x "*/\.*"
cd ../../

# 3. Run the tests to confirm that the issue is fixed.
yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts
```

(cherry picked from commit 3d254c2)

# Conflicts:
#	src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts
gsoldevila added a commit that referenced this pull request Oct 17, 2024
) (#196706)

# Backport

This will backport the following commits from `main` to `8.x`:
- [Update archives to 8.17.0 binary data format, unskip test
(#196641)](#196641)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gerard
Soldevila","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-17T10:49:11Z","message":"Update
archives to 8.17.0 binary data format, unskip test (#196641)\n\n##
Summary\r\n\r\nAddresses
https://github.com/elastic/kibana/issues/167676\r\nAddresses
https://github.com/elastic/kibana/issues/158318\r\nAddresses
https://github.com/elastic/kibana/issues/163254\r\nAddresses
https://github.com/elastic/kibana/issues/163255\r\n\r\n#### Fix for
`multiple_es_nodes.test.ts`\r\nInspired on
https://github.com/elastic/kibana/pull/193899\r\n\r\n1. Start both nodes
of ES 8.17.0 with the affected data-archive on\r\nseparate
terminals:\r\n1. Node 01: `yarn es snapshot --version=8.17.0
--data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\n--base-path
.es/node01`\r\n2. Node 02: `yarn es snapshot --version=8.17.0
--data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\n--base-path
.es/node02`\r\n\r\n2. After ES is ready (without starting Kibana),
reindex the index\r\n`.kibana_7.13.0_002`\r\n1. Retrieve the settings
from the original index via `curl
-L\r\n'http://localhost:9200/.kibana_7.13.0_002' -H
'Content-Type:\r\napplication/json' -H 'kbn-xsrf: test' -H
'Authorization: Basic\r\nc3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l'
-d ''`\r\n\t2. Create the target index with those settings:
\r\n\t```shell\r\ncurl -L -X PUT
'http://localhost:9200/.kibana_7.13.0_003' -H\r\n'Content-Type:
application/json' -H 'kbn-xsrf: test' -H 'Authorization:\r\nBasic
c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{\r\n \"mappings\":
{\r\n \"properties\": {\r\n \"bar\": {\r\n \"properties\": {\r\n
\"status\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\":
{\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n
}\r\n }\r\n },\r\n \"coreMigrationVersion\": {\r\n \"type\":
\"keyword\"\r\n },\r\n \"foo\": {\r\n \"properties\": {\r\n \"status\":
{\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n
\"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n }\r\n
}\r\n },\r\n \"migrationVersion\": {\r\n \"dynamic\": \"true\",\r\n
\"properties\": {\r\n \"bar\": {\r\n \"type\": \"text\",\r\n \"fields\":
{\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\":
256\r\n }\r\n }\r\n },\r\n \"foo\": {\r\n \"type\": \"text\",\r\n
\"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n
\"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"namespace\":
{\r\n \"type\": \"keyword\"\r\n },\r\n \"namespaces\": {\r\n \"type\":
\"keyword\"\r\n },\r\n \"originId\": {\r\n \"type\": \"keyword\"\r\n
},\r\n \"references\": {\r\n \"type\": \"nested\",\r\n \"properties\":
{\r\n \"id\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"name\": {\r\n
\"type\": \"keyword\"\r\n },\r\n \"type\": {\r\n \"type\":
\"keyword\"\r\n }\r\n }\r\n },\r\n \"type\": {\r\n \"type\":
\"keyword\"\r\n },\r\n \"updated_at\": {\r\n \"type\": \"date\"\r\n
}\r\n }\r\n },\r\n \"settings\": {\r\n \"index\": {\r\n \"hidden\":
\"true\",\r\n \"number_of_shards\": \"1\",\r\n \"number_of_replicas\":
\"0\"\r\n }\r\n }\r\n }'\r\n ```\r\n3. Reindex the content: `curl -L
'http://localhost:9200/_reindex' -H\r\n'Content-Type: application/json'
-H 'kbn-xsrf: test' -H 'Authorization:\r\nBasic
c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ \"source\":
{\r\n\"index\": \".kibana_7.13.0_002\" }, \"dest\": { \"index\":
\".kibana_7.13.0_003\"\r\n} }'`\r\n 4. Remove the old index and recreate
the aliases\r\n ```shell\r\ncurl -L 'http://localhost:9200/_aliases' -H
'Content-Type:\r\napplication/json' -H 'kbn-xsrf: test' -H
'Authorization: Basic\r\nc3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l'
-d '{\r\n \"actions\": [\r\n{ \"add\": { \"index\":
\".kibana_7.13.0_003\", \"alias\": \".kibana_7.13.0_001\"\r\n} },\r\n {
\"remove_index\": {\"index\": \".kibana_7.13.0_002\" } },\r\n{ \"add\":
{ \"index\": \".kibana_7.13.0_003\", \"alias\": \".kibana_7.13.0\" }
},\r\n { \"add\": { \"index\": \".kibana_7.13.0_003\", \"alias\":
\".kibana\" } }\r\n ]\r\n}'\r\n\t```\r\n3. Stop both ES nodes.\r\n4.
Compress both archives\r\n\t```shell\r\n\tcd .es/node01/8.17.0\r\nrm -rf
data/nodes # we need to remove this dir or it fails to
start\r\nagain\r\nzip
-r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\ndata
-x \"*/\\.*\"\r\n\tcd ../../../\r\n\tcd .es/node02/8.17.0\r\nrm -rf
data/nodes # we need to remove this dir or it fails to
start\r\nagain\r\nzip
-r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\ndata
-x \"*/\\.*\"\r\n\tcd ../../../\r\n\t```\r\n5. Run the tests to confirm
that the issue is fixed:
`yarn\r\ntest:jest_integration\r\nsrc/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts`\r\n\r\n####
Fix for `incompatible_cluster_routing_allocation.test.ts`\r\nInspired on
https://github.com/elastic/kibana/pull/193741\r\n\r\n```shell\r\n# 1.
Start ES 8.17.0 with the affected data-archive\r\nyarn es snapshot
--version=8.17.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip\r\n#
... after ES has completely started up, stop it.\r\n\r\n# 2. Compress
the archive\r\ncd .es/8.17.0\r\nzip -r
../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip
data -x \"*/\\.*\"\r\ncd ../../\r\n\r\n# 3. Run the tests to confirm
that the issue is fixed.\r\nyarn test:jest_integration
src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts\r\n```\r\n\r\n####
Fix for `read_batch_size.test.ts`\r\nInspired on
https://github.com/elastic/kibana/pull/193899\r\n\r\n```shell\r\n# 1.
Start ES 8.17.0 with the affected data-archive\r\nyarn es snapshot
--version=8.17.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip\r\n#
... after ES has completely started up, stop it.\r\n\r\n# 2. Compress
the archive\r\ncd .es/8.17.0\r\nzip -r
../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip
data -x \"*/\\.*\"\r\ncd ../../\r\n\r\n# 3. Run the tests to confirm
that the issue is fixed.\r\nyarn test:jest_integration
src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts\r\n```","sha":"3d254c223f101a6ee326b57d45b9997a87eb12da","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","v9.0.0","Feature:Migrations","backport:prev-minor"],"number":196641,"url":"https://github.com/elastic/kibana/pull/196641","mergeCommit":{"message":"Update
archives to 8.17.0 binary data format, unskip test (#196641)\n\n##
Summary\r\n\r\nAddresses
https://github.com/elastic/kibana/issues/167676\r\nAddresses
https://github.com/elastic/kibana/issues/158318\r\nAddresses
https://github.com/elastic/kibana/issues/163254\r\nAddresses
https://github.com/elastic/kibana/issues/163255\r\n\r\n#### Fix for
`multiple_es_nodes.test.ts`\r\nInspired on
https://github.com/elastic/kibana/pull/193899\r\n\r\n1. Start both nodes
of ES 8.17.0 with the affected data-archive on\r\nseparate
terminals:\r\n1. Node 01: `yarn es snapshot --version=8.17.0
--data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\n--base-path
.es/node01`\r\n2. Node 02: `yarn es snapshot --version=8.17.0
--data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\n--base-path
.es/node02`\r\n\r\n2. After ES is ready (without starting Kibana),
reindex the index\r\n`.kibana_7.13.0_002`\r\n1. Retrieve the settings
from the original index via `curl
-L\r\n'http://localhost:9200/.kibana_7.13.0_002' -H
'Content-Type:\r\napplication/json' -H 'kbn-xsrf: test' -H
'Authorization: Basic\r\nc3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l'
-d ''`\r\n\t2. Create the target index with those settings:
\r\n\t```shell\r\ncurl -L -X PUT
'http://localhost:9200/.kibana_7.13.0_003' -H\r\n'Content-Type:
application/json' -H 'kbn-xsrf: test' -H 'Authorization:\r\nBasic
c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{\r\n \"mappings\":
{\r\n \"properties\": {\r\n \"bar\": {\r\n \"properties\": {\r\n
\"status\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\":
{\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n
}\r\n }\r\n },\r\n \"coreMigrationVersion\": {\r\n \"type\":
\"keyword\"\r\n },\r\n \"foo\": {\r\n \"properties\": {\r\n \"status\":
{\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n
\"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n }\r\n
}\r\n },\r\n \"migrationVersion\": {\r\n \"dynamic\": \"true\",\r\n
\"properties\": {\r\n \"bar\": {\r\n \"type\": \"text\",\r\n \"fields\":
{\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\":
256\r\n }\r\n }\r\n },\r\n \"foo\": {\r\n \"type\": \"text\",\r\n
\"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n
\"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"namespace\":
{\r\n \"type\": \"keyword\"\r\n },\r\n \"namespaces\": {\r\n \"type\":
\"keyword\"\r\n },\r\n \"originId\": {\r\n \"type\": \"keyword\"\r\n
},\r\n \"references\": {\r\n \"type\": \"nested\",\r\n \"properties\":
{\r\n \"id\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"name\": {\r\n
\"type\": \"keyword\"\r\n },\r\n \"type\": {\r\n \"type\":
\"keyword\"\r\n }\r\n }\r\n },\r\n \"type\": {\r\n \"type\":
\"keyword\"\r\n },\r\n \"updated_at\": {\r\n \"type\": \"date\"\r\n
}\r\n }\r\n },\r\n \"settings\": {\r\n \"index\": {\r\n \"hidden\":
\"true\",\r\n \"number_of_shards\": \"1\",\r\n \"number_of_replicas\":
\"0\"\r\n }\r\n }\r\n }'\r\n ```\r\n3. Reindex the content: `curl -L
'http://localhost:9200/_reindex' -H\r\n'Content-Type: application/json'
-H 'kbn-xsrf: test' -H 'Authorization:\r\nBasic
c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ \"source\":
{\r\n\"index\": \".kibana_7.13.0_002\" }, \"dest\": { \"index\":
\".kibana_7.13.0_003\"\r\n} }'`\r\n 4. Remove the old index and recreate
the aliases\r\n ```shell\r\ncurl -L 'http://localhost:9200/_aliases' -H
'Content-Type:\r\napplication/json' -H 'kbn-xsrf: test' -H
'Authorization: Basic\r\nc3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l'
-d '{\r\n \"actions\": [\r\n{ \"add\": { \"index\":
\".kibana_7.13.0_003\", \"alias\": \".kibana_7.13.0_001\"\r\n} },\r\n {
\"remove_index\": {\"index\": \".kibana_7.13.0_002\" } },\r\n{ \"add\":
{ \"index\": \".kibana_7.13.0_003\", \"alias\": \".kibana_7.13.0\" }
},\r\n { \"add\": { \"index\": \".kibana_7.13.0_003\", \"alias\":
\".kibana\" } }\r\n ]\r\n}'\r\n\t```\r\n3. Stop both ES nodes.\r\n4.
Compress both archives\r\n\t```shell\r\n\tcd .es/node01/8.17.0\r\nrm -rf
data/nodes # we need to remove this dir or it fails to
start\r\nagain\r\nzip
-r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\ndata
-x \"*/\\.*\"\r\n\tcd ../../../\r\n\tcd .es/node02/8.17.0\r\nrm -rf
data/nodes # we need to remove this dir or it fails to
start\r\nagain\r\nzip
-r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\ndata
-x \"*/\\.*\"\r\n\tcd ../../../\r\n\t```\r\n5. Run the tests to confirm
that the issue is fixed:
`yarn\r\ntest:jest_integration\r\nsrc/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts`\r\n\r\n####
Fix for `incompatible_cluster_routing_allocation.test.ts`\r\nInspired on
https://github.com/elastic/kibana/pull/193741\r\n\r\n```shell\r\n# 1.
Start ES 8.17.0 with the affected data-archive\r\nyarn es snapshot
--version=8.17.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip\r\n#
... after ES has completely started up, stop it.\r\n\r\n# 2. Compress
the archive\r\ncd .es/8.17.0\r\nzip -r
../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip
data -x \"*/\\.*\"\r\ncd ../../\r\n\r\n# 3. Run the tests to confirm
that the issue is fixed.\r\nyarn test:jest_integration
src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts\r\n```\r\n\r\n####
Fix for `read_batch_size.test.ts`\r\nInspired on
https://github.com/elastic/kibana/pull/193899\r\n\r\n```shell\r\n# 1.
Start ES 8.17.0 with the affected data-archive\r\nyarn es snapshot
--version=8.17.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip\r\n#
... after ES has completely started up, stop it.\r\n\r\n# 2. Compress
the archive\r\ncd .es/8.17.0\r\nzip -r
../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip
data -x \"*/\\.*\"\r\ncd ../../\r\n\r\n# 3. Run the tests to confirm
that the issue is fixed.\r\nyarn test:jest_integration
src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts\r\n```","sha":"3d254c223f101a6ee326b57d45b9997a87eb12da"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/196641","number":196641,"mergeCommit":{"message":"Update
archives to 8.17.0 binary data format, unskip test (#196641)\n\n##
Summary\r\n\r\nAddresses
https://github.com/elastic/kibana/issues/167676\r\nAddresses
https://github.com/elastic/kibana/issues/158318\r\nAddresses
https://github.com/elastic/kibana/issues/163254\r\nAddresses
https://github.com/elastic/kibana/issues/163255\r\n\r\n#### Fix for
`multiple_es_nodes.test.ts`\r\nInspired on
https://github.com/elastic/kibana/pull/193899\r\n\r\n1. Start both nodes
of ES 8.17.0 with the affected data-archive on\r\nseparate
terminals:\r\n1. Node 01: `yarn es snapshot --version=8.17.0
--data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\n--base-path
.es/node01`\r\n2. Node 02: `yarn es snapshot --version=8.17.0
--data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\n--base-path
.es/node02`\r\n\r\n2. After ES is ready (without starting Kibana),
reindex the index\r\n`.kibana_7.13.0_002`\r\n1. Retrieve the settings
from the original index via `curl
-L\r\n'http://localhost:9200/.kibana_7.13.0_002' -H
'Content-Type:\r\napplication/json' -H 'kbn-xsrf: test' -H
'Authorization: Basic\r\nc3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l'
-d ''`\r\n\t2. Create the target index with those settings:
\r\n\t```shell\r\ncurl -L -X PUT
'http://localhost:9200/.kibana_7.13.0_003' -H\r\n'Content-Type:
application/json' -H 'kbn-xsrf: test' -H 'Authorization:\r\nBasic
c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{\r\n \"mappings\":
{\r\n \"properties\": {\r\n \"bar\": {\r\n \"properties\": {\r\n
\"status\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\":
{\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n
}\r\n }\r\n },\r\n \"coreMigrationVersion\": {\r\n \"type\":
\"keyword\"\r\n },\r\n \"foo\": {\r\n \"properties\": {\r\n \"status\":
{\r\n \"type\": \"text\",\r\n \"fields\": {\r\n \"keyword\": {\r\n
\"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n }\r\n }\r\n }\r\n
}\r\n },\r\n \"migrationVersion\": {\r\n \"dynamic\": \"true\",\r\n
\"properties\": {\r\n \"bar\": {\r\n \"type\": \"text\",\r\n \"fields\":
{\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\":
256\r\n }\r\n }\r\n },\r\n \"foo\": {\r\n \"type\": \"text\",\r\n
\"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n
\"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"namespace\":
{\r\n \"type\": \"keyword\"\r\n },\r\n \"namespaces\": {\r\n \"type\":
\"keyword\"\r\n },\r\n \"originId\": {\r\n \"type\": \"keyword\"\r\n
},\r\n \"references\": {\r\n \"type\": \"nested\",\r\n \"properties\":
{\r\n \"id\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"name\": {\r\n
\"type\": \"keyword\"\r\n },\r\n \"type\": {\r\n \"type\":
\"keyword\"\r\n }\r\n }\r\n },\r\n \"type\": {\r\n \"type\":
\"keyword\"\r\n },\r\n \"updated_at\": {\r\n \"type\": \"date\"\r\n
}\r\n }\r\n },\r\n \"settings\": {\r\n \"index\": {\r\n \"hidden\":
\"true\",\r\n \"number_of_shards\": \"1\",\r\n \"number_of_replicas\":
\"0\"\r\n }\r\n }\r\n }'\r\n ```\r\n3. Reindex the content: `curl -L
'http://localhost:9200/_reindex' -H\r\n'Content-Type: application/json'
-H 'kbn-xsrf: test' -H 'Authorization:\r\nBasic
c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ \"source\":
{\r\n\"index\": \".kibana_7.13.0_002\" }, \"dest\": { \"index\":
\".kibana_7.13.0_003\"\r\n} }'`\r\n 4. Remove the old index and recreate
the aliases\r\n ```shell\r\ncurl -L 'http://localhost:9200/_aliases' -H
'Content-Type:\r\napplication/json' -H 'kbn-xsrf: test' -H
'Authorization: Basic\r\nc3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l'
-d '{\r\n \"actions\": [\r\n{ \"add\": { \"index\":
\".kibana_7.13.0_003\", \"alias\": \".kibana_7.13.0_001\"\r\n} },\r\n {
\"remove_index\": {\"index\": \".kibana_7.13.0_002\" } },\r\n{ \"add\":
{ \"index\": \".kibana_7.13.0_003\", \"alias\": \".kibana_7.13.0\" }
},\r\n { \"add\": { \"index\": \".kibana_7.13.0_003\", \"alias\":
\".kibana\" } }\r\n ]\r\n}'\r\n\t```\r\n3. Stop both ES nodes.\r\n4.
Compress both archives\r\n\t```shell\r\n\tcd .es/node01/8.17.0\r\nrm -rf
data/nodes # we need to remove this dir or it fails to
start\r\nagain\r\nzip
-r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\ndata
-x \"*/\\.*\"\r\n\tcd ../../../\r\n\tcd .es/node02/8.17.0\r\nrm -rf
data/nodes # we need to remove this dir or it fails to
start\r\nagain\r\nzip
-r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\ndata
-x \"*/\\.*\"\r\n\tcd ../../../\r\n\t```\r\n5. Run the tests to confirm
that the issue is fixed:
`yarn\r\ntest:jest_integration\r\nsrc/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts`\r\n\r\n####
Fix for `incompatible_cluster_routing_allocation.test.ts`\r\nInspired on
https://github.com/elastic/kibana/pull/193741\r\n\r\n```shell\r\n# 1.
Start ES 8.17.0 with the affected data-archive\r\nyarn es snapshot
--version=8.17.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip\r\n#
... after ES has completely started up, stop it.\r\n\r\n# 2. Compress
the archive\r\ncd .es/8.17.0\r\nzip -r
../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip
data -x \"*/\\.*\"\r\ncd ../../\r\n\r\n# 3. Run the tests to confirm
that the issue is fixed.\r\nyarn test:jest_integration
src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts\r\n```\r\n\r\n####
Fix for `read_batch_size.test.ts`\r\nInspired on
https://github.com/elastic/kibana/pull/193899\r\n\r\n```shell\r\n# 1.
Start ES 8.17.0 with the affected data-archive\r\nyarn es snapshot
--version=8.17.0 --data-archive
src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip\r\n#
... after ES has completely started up, stop it.\r\n\r\n# 2. Compress
the archive\r\ncd .es/8.17.0\r\nzip -r
../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip
data -x \"*/\\.*\"\r\ncd ../../\r\n\r\n# 3. Run the tests to confirm
that the issue is fixed.\r\nyarn test:jest_integration
src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts\r\n```","sha":"3d254c223f101a6ee326b57d45b9997a87eb12da"}}]}]
BACKPORT-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Feature:Migrations release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.17.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants