create a correlation field mapper #8712
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…r with extra fields for the relationship correlation Signed-off-by: YANGDB <[email protected]>
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: YANGDB <[email protected]> Signed-off-by: YANGDB <[email protected]>
Signed-off-by: YANGDB <[email protected]>
…correlation_field_mapping
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: YANGDB <[email protected]>
…correlation_field_mapping
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: YANGDB <[email protected]>
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: YANGDB <[email protected]>
…correlation_field_mapping # Conflicts: # server/src/internalClusterTest/java/org/opensearch/search/fieldcaps/FieldCapabilitiesIT.java
…correlation_field_mapping Signed-off-by: YANGDB <[email protected]> # Conflicts: # server/src/internalClusterTest/java/org/opensearch/search/fieldcaps/FieldCapabilitiesIT.java
…correlation_field_mapping
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
YANG-DB
requested review from
reta,
anasalkouz,
andrross,
Bukhtawar,
CEHENKLE,
dblock,
gbbafna,
setiah,
kartg,
kotwanikunal and
mch2
as code owners
…correlation_field_mapping Signed-off-by: YANGDB <[email protected]>
Gradle Check (Jenkins) Run Completed with:
|
…correlation_field_mapping Signed-off-by: YANGDB <[email protected]>
…correlation_field_mapping # Conflicts: # server/src/main/java/org/opensearch/action/fieldcaps/FieldCapabilities.java # server/src/test/java/org/opensearch/index/mapper/FieldCorrelationMapperTests.java # server/src/test/java/org/opensearch/index/mapper/FieldTypeLookupTests.java
Gradle Check (Jenkins) Run Completed with:
|
…correlation_field_mapping Signed-off-by: YANGDB <[email protected]> # Conflicts: # server/src/main/java/org/opensearch/action/fieldcaps/FieldCapabilities.java # server/src/test/java/org/opensearch/index/mapper/FieldCorrelationMapperTests.java # server/src/test/java/org/opensearch/index/mapper/FieldTypeLookupTests.java
…correlation_field_mapping
Gradle Check (Jenkins) Run Completed with:
|
|
The failure is because there aren't version checks around the StreamInput / StreamOutput serialization in this PR. So when the BWC tests run (which spins up a 3.0 / 2.10 mixed cluster environment) a 3.0 Node is trying to send a variable over the wire to a 2.10 Node that doesn't have the unmarshalling logic. |
# Conflicts: # CHANGELOG.md
Gradle Check (Jenkins) Run Completed with:
|
|
Compatibility status: |
Signed-off-by: YANGDB <[email protected]>
Gradle Check (Jenkins) Run Completed with:
|
|
Compatibility status: |
Signed-off-by: YANGDB <[email protected]>
Gradle Check (Jenkins) Run Completed with:
|
|
Compatibility status: |
Signed-off-by: YANGDB <[email protected]>
|
Compatibility status: |
Gradle Check (Jenkins) Run Completed with:
|
|
This PR is stalled because it has been open for 30 days with no activity. Remove stalled label or comment or this will be closed in 7 days. |
|
@YANG-DB Is this being worked upon? If not, lets close this. |
|
Hi @YANG-DB, do we have any updates? |
ticheng-aws
added
the
enhancement
|
This PR is stalled because it has been open for 30 days with no activity. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New Correlation Mapping Type Support
As part of the Integration campaign and Integration RFC , we have introduction the SimpleSchema for Observability Domain that is based on the concept of a well-structured index which is based on a schema
The general context:
Schema
A schema is associated to an index using the mapping configuration .
This mapping structure is also composable using the
composed_of templatecapabilities which is used extensively to allow the different assemblies of various log types.Another concept behind the schema is the capability of reflecting relationships.
Relationships
Relationships are associations between different fields which may exist within the same mapping - we call them aliases, or defined between different mapping and we shall call them correlations.
Goal
Our goal is to formalize and generalize the relationship semantic concept whether its within the same mapping file or between different mapping files.
Correlation Engine
The emergence of the correlation-engine also significantly prove the importance of the schematic awareness of such relationship and explicitly definition them within the document mapping.
Once such mapping type is defined, auto generated correlation rules can be created and allow a strong capability of performing RCA and diagnostics of different domain investigation whether they are Observability or Security related.
Planned Steps
This new schematic concept will be separated into two parts:
Part 1:
aliasfield type within a dedicated GetFieldsAlias APIaliasPart 2:
Based on Part 1 add the similar capabilities for the correlation field type for both
correlationfield type within a dedicated GetFieldsCorrelation APIcorrelationCurrent Correlations representation
This correlation representation is currently defined in a proprietary way of adding this information to
the index mapping template's metadata
What we want to achieve :
FieldCorrelationMapper has the next mapping schema:
The
spanIdFk&traceIdFkfields act as correlation fields which are connecting the local (path) field to the remote (schema_pattern:remote_path) fieldPart 1
This part will focus on the enhancement of the
aliasfield type:Get Field Aliases API (New)
Lets consider the next mapping
The next API was added to get the Field's aliases:
GET /_mapping/field/{fields}/aliaseswherefields&indexaccept both a single, multiple and regular expressionsFor
GET /_mapping/field/['field1', 'obj.subfield']/aliaseswill return with:Get Field capacity API
The next API was enhanced to get the Field's aliases aspect:
The next call
GET old_index/_field_caps?fields=distance,route_length_mileswill return the capacities for these fields
distance,route_length_milesas they are defined within different mappings, it will also reflect these field's aliases associated with each concrete field (if such exist).When an alias field is queries for its capacity the API will explicitly state it is of an alias type and will state the mappings of-which is functions as an alias:
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.