Missing intermediate objects when applying dynamic template that targets an object #87513
Assignees
Labels
Comments
romseygeek
added
the
:Search/Search
|
Pinging @elastic/es-search (Team:Search) |
romseygeek
removed
Team:Search
javanna
changed the title
IllegalArgumentException - Dynamic Mapping failing on large nested objects
|
I was able to reproduce this, the missing bit in the initial description of the problem were the mappings. There's a dynamic template that targets an object which triggers the issue once you index a document with an object that matches it: Indexing the following minimal document against an index with the above dynamic template in the mappings causes the "Missing intermediate object" to be thrown: We may want to, besides fixing this bug, change the type of this exception as it is not expected to be thrown, and here it gives the perception that the user made a mistake, while it is not the case. |
|
Here is another one that came up from our production environment today 😇 : "dynamic_templates": [
{
"map_process_fields": {
"match_pattern": "regex",
"mapping": {
"properties": {
"Ext": {
"properties": {
"code_signature": {
"type": "nested"
},
"dll": {
"type": "nested"
}
}
},
"name": {
"type": "text",
"norms": false,
"fields": {
"wildcard": {
"type": "wildcard"
}
}
},
"command_line": {
"type": "text",
"norms": false,
"fields": {
"wildcard": {
"type": "wildcard"
}
}
},
"executable": {
"type": "text",
"norms": false,
"fields": {
"wildcard": {
"type": "wildcard"
}
}
},
"thread": {
"properties": {
"Ext": {
"properties": {
"start_address_details": {
"properties": {
"compressed_bytes": {
"enabled": false
},
"strings": {
"type": "keyword",
"ignore_above": 512
}
}
}
}
}
}
}
}
},
"match_mapping_type": "object",
"match": "^(process|parent)$"
}
}
] |
|
It's the same bug, thanks. It's to do with objects that are being created from within the definition of a dynamic template. I am working on this. If you want to work around it in the meantime, my suggestion would be to have one dynamic template per field instead of a big one per object. Ping me if you need help with this, |
javanna
added a commit
to javanna/elasticsearch
that referenced
this issue
Jun 14, 2022
When we apply dynamic mappings, we iterate over all the dynamic mappers retrieved from the DocumentParserContext, which are registered while parsing the document, and for each field look up their parent (going back multiple levels if necessary), and add it to the dynamic mapping update with the added/modified sub-field. Dynamic mappers that are iterated through consist of a flat set of mappers, which can be both object mappers or field mappers. Most times, the object mappers from such set have no sub-fields as they are mapped as a result of parsing a document where the object appears for the first time, which also has sub-fields that are going to be added to the set of dynamic mappers themselves once they are parsed. There are scenarios though where a dynamic template matches an object, and defines its structure including its subobjects as well as sub-fields. In that case the dynamically mapped object may hold sub-fields as well as define non-default values for dynamic, enabled or subobjects. The described situation was not well covered in tests so far, and is currently affected by a bug introduced with elastic#81449. When an object mapper is dynamically mapped, it is added to the map of dynamic object mappers, which makes it discoverable for sub-fields that will need to be added to it later, as well as to the set of dynamic mappers so that it becomes part of the mappings in case the document being parsed defines no sub-fields for it. What is missing is to recursively add its sub-fields to the dynamic object mappers. As a result of missing this step, intermediate objects that were dynamically mapped are not made discoverable which causes a cryptic "Missing intermediate object" error. This commit fixed the problem by recursively registering inner sub-objects to the dynamic mappers whenever an object mapper is added to the dynamic mappers. It also changes the "missing intermediate object" error to make it more evident that it's an internal error and not a user error: it is now an IllegalStateException instead of an IllegalArgumentException. Closes elastic#87513
javanna
added a commit
that referenced
this issue
Jun 17, 2022
…87622) When we apply dynamic mappings, we iterate over all the dynamic mappers retrieved from the DocumentParserContext, which are registered while parsing the document, and for each field look up their parent (going back multiple levels if necessary), and add it to the dynamic mapping update with the added/modified sub-field. Dynamic mappers that are iterated through consist of a flat set of mappers, which can be both object mappers or field mappers. Most times, the object mappers from such set have no sub-fields as they are mapped as a result of parsing a document where the object appears for the first time, which also has sub-fields that are going to be added to the set of dynamic mappers themselves once they are parsed. There are scenarios though where a dynamic template matches an object, and defines its structure including its subobjects as well as sub-fields. In that case the dynamically mapped object may hold sub-fields as well as define non-default values for dynamic, enabled or subobjects. The described situation was not well covered in tests so far, and is currently affected by a bug introduced with #81449. When an object mapper is dynamically mapped, it is added to the map of dynamic object mappers, which makes it discoverable for sub-fields that will need to be added to it later, as well as to the set of dynamic mappers so that it becomes part of the mappings in case the document being parsed defines no sub-fields for it. What is missing is to recursively add its sub-fields to the dynamic object mappers. As a result of missing this step, intermediate objects that were dynamically mapped are not made discoverable which causes a cryptic "Missing intermediate object" error. This commit fixed the problem by recursively registering inner sub-objects to the dynamic mappers whenever an object mapper is added to the dynamic mappers. It also changes the "missing intermediate object" error to make it more evident that it's an internal error and not a user error: it is now an IllegalStateException instead of an IllegalArgumentException. Closes #87513
javanna
added a commit
to javanna/elasticsearch
that referenced
this issue
Jun 17, 2022
…lastic#87622) When we apply dynamic mappings, we iterate over all the dynamic mappers retrieved from the DocumentParserContext, which are registered while parsing the document, and for each field look up their parent (going back multiple levels if necessary), and add it to the dynamic mapping update with the added/modified sub-field. Dynamic mappers that are iterated through consist of a flat set of mappers, which can be both object mappers or field mappers. Most times, the object mappers from such set have no sub-fields as they are mapped as a result of parsing a document where the object appears for the first time, which also has sub-fields that are going to be added to the set of dynamic mappers themselves once they are parsed. There are scenarios though where a dynamic template matches an object, and defines its structure including its subobjects as well as sub-fields. In that case the dynamically mapped object may hold sub-fields as well as define non-default values for dynamic, enabled or subobjects. The described situation was not well covered in tests so far, and is currently affected by a bug introduced with elastic#81449. When an object mapper is dynamically mapped, it is added to the map of dynamic object mappers, which makes it discoverable for sub-fields that will need to be added to it later, as well as to the set of dynamic mappers so that it becomes part of the mappings in case the document being parsed defines no sub-fields for it. What is missing is to recursively add its sub-fields to the dynamic object mappers. As a result of missing this step, intermediate objects that were dynamically mapped are not made discoverable which causes a cryptic "Missing intermediate object" error. This commit fixed the problem by recursively registering inner sub-objects to the dynamic mappers whenever an object mapper is added to the dynamic mappers. It also changes the "missing intermediate object" error to make it more evident that it's an internal error and not a user error: it is now an IllegalStateException instead of an IllegalArgumentException. Closes elastic#87513
javanna
added a commit
that referenced
this issue
Jun 17, 2022
…87622) When we apply dynamic mappings, we iterate over all the dynamic mappers retrieved from the DocumentParserContext, which are registered while parsing the document, and for each field look up their parent (going back multiple levels if necessary), and add it to the dynamic mapping update with the added/modified sub-field. Dynamic mappers that are iterated through consist of a flat set of mappers, which can be both object mappers or field mappers. Most times, the object mappers from such set have no sub-fields as they are mapped as a result of parsing a document where the object appears for the first time, which also has sub-fields that are going to be added to the set of dynamic mappers themselves once they are parsed. There are scenarios though where a dynamic template matches an object, and defines its structure including its subobjects as well as sub-fields. In that case the dynamically mapped object may hold sub-fields as well as define non-default values for dynamic, enabled or subobjects. The described situation was not well covered in tests so far, and is currently affected by a bug introduced with #81449. When an object mapper is dynamically mapped, it is added to the map of dynamic object mappers, which makes it discoverable for sub-fields that will need to be added to it later, as well as to the set of dynamic mappers so that it becomes part of the mappings in case the document being parsed defines no sub-fields for it. What is missing is to recursively add its sub-fields to the dynamic object mappers. As a result of missing this step, intermediate objects that were dynamically mapped are not made discoverable which causes a cryptic "Missing intermediate object" error. This commit fixed the problem by recursively registering inner sub-objects to the dynamic mappers whenever an object mapper is added to the dynamic mappers. It also changes the "missing intermediate object" error to make it more evident that it's an internal error and not a user error: it is now an IllegalStateException instead of an IllegalArgumentException. Closes #87513
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Elasticsearch Version
8.1.2
Installed Plugins
n/a
Java Version
bundled
OS Version
Cloud / Docker
Problem Description
RE: #81449
Upon upgrading to 8.1.2 we have started to see failures on our telemetry ingest server. I believe it is related to the above PR after a brief chat with @romseygeek. I'd also be up for tackling this myself.
Steps to Reproduce
Here is an example document:
It is a datastream with a not very interesting index template - ingest pipelines are disabled.
Example Document
Seems to be upset about the artifacts section
{ "_index": ".ds-alert_telemetry_elastic-2022.05.30-000035", "_id": "Mdw/NYPcxhdiVpU1+++++t7/-2022-06-01T11:14:48Z", "_version": 1, "_score": 1, "_source": { "cluster_name": "redacted", "process": { "args": [ "xpcproxy", "redacted" ], "parent": { "args": [ "/sbin/launchd" ], "code_signature": { "signing_id": "com.apple.xpc.launchd", "trusted": true, "subject_name": "Software Signing", "exists": true, "team_id": "", "status": "No error." }, "name": "launchd", "pid": 1, "entity_id": "YWY2NGVjZjctYjljYy00Yzc3LWJkYmUtMGJmMGQwZWRhMmIxLTEtMTMyOTkwNjEzMTUuMA==", "hash": { "sha1": "6baa120c22c73667be11a307d052b85fb8cd30e8", "sha256": "7c8f6b99650e1e34fc8a4435c6110e8d29ae4b517aa81e9d11983525144d8379", "md5": "e55968487437e9a3a2b766f85256b502" }, "command_line": "/sbin/launchd", "executable": "/sbin/launchd", "uptime": 107590 }, "name": "CleanMyMac X", "pid": 10178, "entity_id": "YWY2NGVjZjctYjljYy00Yzc3LWJkYmUtMGJmMGQwZWRhMmIxLTEwMTc4LTEzMjk5MTY4OTAzLjk3MDg5NTAwMA==", "hash": { "sha1": "9a1196bfd1373253f0ebdba425bcf7b9f80c643f", "sha256": "83a0a3c542c93557052e79834c24810c2071380eb8f2b8cd08b10c35f1c33712", "md5": "b5c405691df6bceb297a365b96bfb32a" }, "command_line": "redacted", "executable": "/Applications/CleanMyMac X.app/Contents/MacOS/CleanMyMac X", "uptime": 2 }, "agent": { "build": { "original": "version: 8.1.3, compiled: Fri Apr 8 08:00:00 2022, branch: 8.1, commit: 41b69a918785c0e60099ed7564b2a828b4f65e60" }, "id": "af64ecf7-b9cc-4c77-bdbe-0bf0d0eda2b1", "type": "endpoint", "version": "8.1.3" }, "channel": "alerts-endpoint", "rule": { "ruleset": "diagnostic" }, "version": "8.2.0", "cloud": { "deployment_info": { "in_trial": false, "is_elastic_internal": false, "account_uuid": 111222333, "deployment_parent": "redacted", "domain": "redacted.tld", "is_cloud": true, "industry": null }, "cluster_name": "redacted" }, "@timestamp": "2022-06-08T13:35:05.713235Z", "file": { "Ext": { "malware_classification": { "identifier": "endpointmacho-v1-model", "score": 0.05650628730654716, "threshold": 0.05, "version": "1.0.4000" } }, "path": "/Applications/CleanMyMac X.app/Contents/MacOS/CleanMyMac X", "extension": "app/contents/macos/cleanmymac x", "size": 17414784, "created": "1970-01-01T00:00:00.0Z", "name": "CleanMyMac X", "accessed": "2022-06-01T10:57:17.425641088Z", "mtime": "2022-06-01T09:33:06.0Z", "directory": "/Applications/CleanMyMac X.app/Contents/MacOS", "hash": { "sha1": "9a1196bfd1373253f0ebdba425bcf7b9f80c643f", "sha256": "83a0a3c542c93557052e79834c24810c2071380eb8f2b8cd08b10c35f1c33712", "md5": "b5c405691df6bceb297a365b96bfb32a" } }, "cluster_uuid": "redacted", "Endpoint": { "policy": { "applied": { "artifacts": { "global": { "identifiers": [{ "sha256": "8d509bc7713f88211870e3ecaae6cd8a62b53b94a4ce29023977754496ba5ef6", "name": "diagnostic-configuration-v1" }, { "sha256": "78fe4906ed8c3c8984b64f1f39d5eee0673b750349724c46dd354e54e4103abf", "name": "diagnostic-endpointmacho-v1-blocklist" }, { "sha256": "3c859b5cbda994b8ac08e226aaea8177813165c17884bd50bec481a42b99f6a5", "name": "diagnostic-endpointmacho-v1-exceptionlist" }, { "sha256": "bf9966a24bab5c76965b897ec3af5b10261c790b54db018ba8b14940486a5808", "name": "diagnostic-endpointmacho-v1-model" }, { "sha256": "71388bb1d73631d7b961824af2e41d77b80863c4025150d8716bb3324376060f", "name": "diagnostic-malware-signature-v1-macos" }, { "sha256": "035d7bd0586d79f4be4deeff7c83e1ebccaad40bbe9b00295abca67c235c17e8", "name": "diagnostic-rules-macos-v1" }, { "sha256": "98439cff9e3ceadfcddb8a42fef4d555b8ab6ea5a99a2bf811e3073ebbb4c2ee", "name": "endpointmacho-v1-blocklist" }, { "sha256": "e24b08f157dbb4f9f6c90ae63240752428f8409b385e6992b90a891c63c98a8d", "name": "endpointmacho-v1-exceptionlist" }, { "sha256": "b6776bd754e02f61032ad2837e4c6f3eb4f8a34ad57756fbf4c322394220a9d1", "name": "endpointmacho-v1-model" }, { "sha256": "2a747a4548ed22bf57db8c651bb41b0eb96ffe791d8c3a1efa8e13a58f4d8e74", "name": "global-configuration-v1" }, { "sha256": "f5b1bc60e499e0cfdd5c70248a21471f30c9e392e2706981255e74e5d9e6c48e", "name": "global-exceptionlist-macos" }, { "sha256": "138a7a19c2df5a77ee32486650c22311ada3f92131f4234d847f8f4a9814971f", "name": "global-trustlist-macos-v1" }, { "sha256": "a6759b2d125681b447f40bfb5f732b31aeedff39f66c67694de5a10d36ba6793", "name": "production-malware-signature-v1-macos" }, { "sha256": "7b7aa29c63d65256bd5ceae2c61505980be92dd07d0eb006e12b9c63ce237db0", "name": "production-rules-macos-v1" } ], "version": "1.0.306" }, "user": { "identifiers": [{ "sha256": "d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658", "name": "endpoint-eventfilterlist-macos-v1" }, { "sha256": "dc1e63def5da42b67dc47db1714b289e983bdf70f2a9e2877cc0f6100bd70c67", "name": "endpoint-exceptionlist-macos-v1" }, { "sha256": "d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658", "name": "endpoint-hostisolationexceptionlist-macos-v1" }, { "sha256": "d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658", "name": "endpoint-trustlist-macos-v1" } ], "version": "1.0.16" } } } } }, "ecs": { "version": "1.11.0" }, "data_stream": { "namespace": "corporate", "type": ".logs", "dataset": "endpoint.diagnostic.collection" }, "elastic": { "agent": { "id": "redacted" } }, "host": { "os": { "Ext": { "variant": "macOS" }, "kernel": "Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:29 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T8101", "name": "macOS", "family": "macos", "type": "macos", "version": "12.4", "platform": "macos", "full": "macOS 12.4" } }, "location": "gs://redacted", "event": { "severity": 99, "code": "malicious_file", "risk_score": 99, "created": "2022-06-08T13:35:05.713235Z", "kind": "alert", "module": "endpoint", "type": [ "info", "start", "allowed" ], "agent_id_status": "verified", "sequence": 174148, "ingested": "2022-06-01T11:14:48Z", "action": "execution", "id": "Mdw/NYPcxhdiVpU1+++++t7/", "category": [ "malware", "intrusion_detection", "process" ], "dataset": "endpoint.diagnostic.collection", "outcome": "success" }, } }Logs (if relevant)
An example log
The text was updated successfully, but these errors were encountered: