Handle index.mapping.ignore_malformed in downsampling
#119134
Conversation
lkts
added
>bug
auto-backport
lkts
changed the title
index.mapping.ignore_malformed in downsampling
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
|
Hi @lkts, I've created a changelog YAML for you. |
|
|
||
| if (randomBoolean()) { | ||
| settings.put(IndexMetadata.SETTING_INDEX_HIDDEN, randomBoolean()); | ||
| } | ||
|
|
||
| XContentBuilder mapping = jsonBuilder().startObject().startObject("_doc").startObject("properties"); | ||
| mapping.startObject(FIELD_TIMESTAMP).field("type", "date").endObject(); | ||
| mapping.startObject(FIELD_TIMESTAMP).field("type", "date").field("ignore_malformed", false).endObject(); |
There was a problem hiding this comment.
This should be set when randomBoolean( ) above is also set (and left unset otherwise), for coverage.
| FieldMapper.IGNORE_MALFORMED_SETTING.getKey(), | ||
| sourceIndexMetadata.getSettings().get(FieldMapper.IGNORE_MALFORMED_SETTING.getKey()) | ||
| ); | ||
| } |
There was a problem hiding this comment.
I'm somewhat puzzled by this, so how does @timestamp pick up ignore_above:true if this is not set? I assume it's passed somewhere but not everywhere..
There was a problem hiding this comment.
I also don't fully understand this. Maybe the setting came from an index template?
There was a problem hiding this comment.
There are two sets of index settings:
- Settings applied when index is created (what you are looking at).
- Setting applied after the index is created - everything from the original index is copied (except settings in 1).
I have to admit i don't understand the reason for the split but this is the code structure. FieldMapper.IGNORE_MALFORMED_SETTING was in 2 but now is in 1. The reason is that by the time 2 is applied it's too late to "fix" mapping of @timestamp with it in mind.
|
Backport to 8.16.x, too? |
No more 8.16 releases will happen, so back porting this to 8.x and 8.17 branches is sufficient. |
I see a FFreeze 8.16.3 on the cal? |
I don't see a dev issue for it and given that 8.17.0 has been released, it means no 8.16.x releases will happen. |
Currently
ignore_malformedparameter of@timestampfield is not propagated to downsampled index. This is problematic whenindex.mapping.ignore_malformedis set totruebecause it leads to a validation error for data streams (ignoring timestamp does not make logical sense in a data stream). This PR fixes it.Additionally we need to set
index.mapping.ignore_malformedindex setting earlier in the downsampling logic. Currently it is applied after the index is created with initial mapping. This leads to hitting this assertelasticsearch/server/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java
Line 74 in 103d29f
index.mapping.ignore_malformedis not set it defaults tofalsewhich matches the value ofignore_malformedon@timestampand soignore_malformedis not serialized in mapping but it is present in mapping source.Fixes #119075.