[ML] Add runtime mappings to data frame analytics source config #69183
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Users can now specify runtime mappings as part of the source config of a data frame analytics job. Those runtime mappings become part of the mapping of the destination index. This ensures the fields are accessible in the destination index even if the relevant data frame analytics job gets deleted. Closes elastic#65056
|
Pinging @elastic/ml-core (Team:ML) |
benwtrent
reviewed
Feb 18, 2021
| import java.util.Map; | ||
|
|
||
| public final class RuntimeMappingsValidator { | ||
|
|
There was a problem hiding this comment.
private ctor?
Suggested change
| private RuntimeMappingsValidator() { } | |
| Map<String, Object> propNode = new HashMap<>(((Map<String, Object>) entry.getValue())); | ||
| Object typeNode = propNode.get("type"); | ||
| if (typeNode == null) { | ||
| throw ExceptionsHelper.badRequestException("No type specified for runtime field [" + fieldName + "]"); |
There was a problem hiding this comment.
Suggested change
| throw ExceptionsHelper.badRequestException("No type specified for runtime field [" + fieldName + "]"); | |
| throw ExceptionsHelper.badRequestException("No type specified for runtime field [{}]", fieldName); |
Comment on lines
31
to
32
| throw ExceptionsHelper.badRequestException("Expected map for runtime field [" + fieldName + "] " + | ||
| "definition but got a " + fieldName.getClass().getSimpleName()); |
There was a problem hiding this comment.
Suggested change
| throw ExceptionsHelper.badRequestException("Expected map for runtime field [" + fieldName + "] " + | |
| "definition but got a " + fieldName.getClass().getSimpleName()); | |
| throw ExceptionsHelper.badRequestException( | |
| "Expected map for runtime field [{}] definition but got a {}", | |
| fieldName, | |
| fieldName.getClass().getSimpleName() | |
| ); |
I think string formatting like this is much nicer. Feel free to ignore :)
|
@benwtrent Thank you for pointing out lack of coverage for regression/classification. Adding tests revealed a couple more places where I had to add runtime mappings. Pushed the fix, should be good for another review now. |
benwtrent
approved these changes
Feb 19, 2021
|
@elasticmachine update branch |
dimitris-athanasiou
deleted the
search-runtime-mappings-for-data-frame-analytics
branch
dimitris-athanasiou
added a commit
that referenced
this pull request
Feb 19, 2021
Until #69183 is backported to 7.x
dimitris-athanasiou
added a commit
that referenced
this pull request
Feb 19, 2021
#69284) Users can now specify runtime mappings as part of the source config of a data frame analytics job. Those runtime mappings become part of the mapping of the destination index. This ensures the fields are accessible in the destination index even if the relevant data frame analytics job gets deleted. Closes #65056 Backport of #69183
dimitris-athanasiou
added a commit
that referenced
this pull request
Feb 20, 2021
... after backporting #69183
dimitris-athanasiou
added a commit
that referenced
this pull request
Feb 22, 2021
…ily (#69329) Since config runtime mappings were added to data frame analytics jobs in #69183, when we create the destination index we add a `runtime` section in its mappings regardless of whether the job config contains any runtime fields. This causes failures in the BWC tests when in a mixed cluster where there are nodes versioned before runtime fields were introduced. This is the first part of the fix were we do not create a `runtime` section in the destination index mappings unless necessary.
dimitris-athanasiou
added a commit
to dimitris-athanasiou/elasticsearch
that referenced
this pull request
Feb 22, 2021
…essarily (elastic#69329) Since config runtime mappings were added to data frame analytics jobs in elastic#69183, when we create the destination index we add a `runtime` section in its mappings regardless of whether the job config contains any runtime fields. This causes failures in the BWC tests when in a mixed cluster where there are nodes versioned before runtime fields were introduced. This is the first part of the fix were we do not create a `runtime` section in the destination index mappings unless necessary. Backport of elastic#69329
dimitris-athanasiou
added a commit
that referenced
this pull request
Feb 22, 2021
…essarily (#69329) (#69332) Since config runtime mappings were added to data frame analytics jobs in #69183, when we create the destination index we add a `runtime` section in its mappings regardless of whether the job config contains any runtime fields. This causes failures in the BWC tests when in a mixed cluster where there are nodes versioned before runtime fields were introduced. This is the first part of the fix were we do not create a `runtime` section in the destination index mappings unless necessary. Backport of #69329
62 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Users can now specify runtime mappings as part of the source config
of a data frame analytics job. Those runtime mappings become part of
the mapping of the destination index. This ensures the fields are
accessible in the destination index even if the relevant data frame
analytics job gets deleted.
Closes #65056