Add MultiTypeMappingTransformation and test #1189
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
After recently reviewing metadata migration end to end test cases I noticed there was 56 different tests, this was a large increase, taking nearly 20 minutes of runtime. Reviewing the parameters of the tests I'm running evaluate and then immediately following migrate, this should reduce the test cases by nearly half and increase of coverage using http and snapshot sources. I have tweaked how we use templates moving back to the logic of legacy templates for ES 6.X and using both index and index compontent templates for future ES versions, which should be another 30% redunction. Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
AndreKurait
requested review from
chelma,
gregschohn,
lewijacn,
mikaylathompson,
peternied and
sumobrian
as code owners
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
peternied
reviewed
Dec 10, 2024
RFS/src/testFixtures/java/org/opensearch/migrations/bulkload/http/ClusterOperations.java
Outdated
Show resolved
Hide resolved
RFS/src/testFixtures/java/org/opensearch/migrations/bulkload/http/ClusterOperations.java
Outdated
Show resolved
Hide resolved
...c/testFixtures/java/org/opensearch/migrations/bulkload/framework/SearchClusterContainer.java
Outdated
Show resolved
Hide resolved
...c/testFixtures/java/org/opensearch/migrations/bulkload/framework/SearchClusterContainer.java
Outdated
Show resolved
Hide resolved
...on/src/main/java/org/opensearch/migrations/transformation/rules/IndexMappingTypeRemoval.java
Show resolved
Hide resolved
...ataMigration/src/test/java/org/opensearch/migrations/MultiTypeMappingTransformationTest.java
Outdated
Show resolved
Hide resolved
...ataMigration/src/test/java/org/opensearch/migrations/MultiTypeMappingTransformationTest.java
Outdated
Show resolved
Hide resolved
MetadataMigration/src/main/java/org/opensearch/migrations/MigrateOrEvaluateArgs.java
Show resolved
Hide resolved
...ataMigration/src/test/java/org/opensearch/migrations/MultiTypeMappingTransformationTest.java
Outdated
Show resolved
Hide resolved
...ataMigration/src/test/java/org/opensearch/migrations/MultiTypeMappingTransformationTest.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
AndreKurait
force-pushed
the
MetadataMultiType
branch
from
9669ce2 to
bfb5704
Compare
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
peternied
reviewed
Dec 10, 2024
MetadataMigration/src/test/java/org/opensearch/migrations/CustomTransformationTest.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
peternied
reviewed
Dec 10, 2024
RFS/src/testFixtures/java/org/opensearch/migrations/bulkload/http/ClusterOperations.java
Outdated
Show resolved
Hide resolved
...rc/test/java/org/opensearch/migrations/transformation/rules/IndexMappingTypeRemovalTest.java
Show resolved
Hide resolved
| @@ -1,50 +1,33 @@ | |||
| package org.opensearch.migrations; | |||
There was a problem hiding this comment.
Since there is a decent amount of refactoring going on here want to pull #1186 into this change, it should help your cycles iterate more quickly
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
peternied
approved these changes
Dec 10, 2024
Signed-off-by: Andre Kurait <[email protected]>
AndreKurait
force-pushed
the
MetadataMultiType
branch
from
a4a57fe to
406ad78
Compare
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
…support Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]> Signed-off-by: Andre Kurait <[email protected]>
AndreKurait
force-pushed
the
MetadataMultiType
branch
from
72bff28 to
e501f09
Compare
Signed-off-by: Andre Kurait <[email protected]>
Signed-off-by: Andre Kurait <[email protected]>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1189 +/- ##
============================================
+ Coverage 80.55% 80.57% +0.02%
- Complexity 3074 3081 +7
============================================
Files 421 421
Lines 15545 15592 +47
Branches 1047 1053 +6
============================================
+ Hits 12522 12564 +42
- Misses 2380 2386 +6
+ Partials 643 642 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
peternied
reviewed
Dec 11, 2024
| @@ -111,6 +111,44 @@ jobs: | |||
| with: | |||
| gradle-version: ${{ env.gradle-version }} | |||
| gradle-home-cache-cleanup: true | |||
| - name: Generate Cache Key from Dockerfiles | |||
There was a problem hiding this comment.
@AndreKurait Can you revert the change to this file or maybe we can sync on the utility of this change?
The additional workflow complexity + extra effort to remember to sync these lists between java source and CI.yml gives me pause about including this change. It looks like this was done in an effort to speed up and issues with the test case that ended up being disabled due to an unrelated root cause.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Support Union resolution of MultiTypeMappingTransformation in metadata migrations.
Adds argument
--multi-type-behaviorallowing customers to specify UNION and migrate multi type indices.Issues Resolved
[List any issues this PR will resolve]
Is this a backport? If so, please add backport PR # and/or commits #
Testing
Added E2E unit test
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.