illegal_argument_exception - unexpected metadata [op:[TEST]] in source

[rishabh-mmc]

Elasticsearch Version

8.12.2

Installed Plugins

No response

Java Version

bundled

OS Version

Mac OS Sonomo (14.4.1)

Problem Description

Unable to update any document when there is field named op in mapping.

Steps to Reproduce

// Create Mapping

PUT op_index_test

{
  "mappings": {
    "properties": {
      "firstName": {
        "type": "text"
      },
      "lastName": {
        "type": "text"
      },
      "op": {
        "type": "keyword"
      },
      "orgId": {
        "type": "integer"
      }
    }
  }
}

// insert data
POST op_index_test/_doc/1

{"firstName": "BQCayRMVdN", "lastName": "vVDUmYONQW", "orgId": 1, "op": ["TEST"]}

// update by query
POST op_index_test/_update_by_query

{
  "script": {
    "source": """
    if (ctx._source.orgId == 1) { ctx._source.firstName = ctx._source.firstName}
    """,
    "lang": "painless"
  },
  "query": { 
    "match": {
      "orgId": 1
    }
  }
}

Error:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "unexpected metadata [op:[TEST]] in source"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "unexpected metadata [op:[TEST]] in source"
  },
  "status": 400
}

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[williamrandolph]

I wasn't able to reproduce this issue. Is there a stack trace in the Elasticsearch application logs that you could share?

[rishabh-mmc]

@williamrandolph I am not seeing any logs while hitting this request.

[rishabh-mmc]

@williamrandolph can you share configuration you used while trying to reproduce this?
i have installed Elastic search using docker on ES 8 and server I am using eck-operator helm chart.

also, I have disabled tls using rejectUnauthorized: false in configuration.

[williamrandolph]

I used the .zip archive on MacOS. I'll try the docker image.

[rishabh-mmc]

@williamrandolph any update here?

[jpmat296]

I have faced similar error message with Elasticsearch 8.11.3.

I'm able to reproduce the exact same error as @rishabh-mmc. Please find attached the 3 commands to execute in Dev Tools : dev_tools.txt

[rishabh-mmc]

@williamrandolph @jpmat296 what are next steps here as this is reproduced. we are using op key in our project and this is impacting out functionality thus not able to upgrade to ES 8.

[AlexeySagin]

I have the same issue with Elasticsearch 8.13. But the field is called _type.
I also checked on versions 8.9, 8.13, 8.14. The latest one at this moment is 8.15.
I also checked the original field op issue and it has reproduced.
Bug is still here.

I did a little research. I added a parameter error_trace=true to _update request and got a stacktrace:

...
"type":"illegal_argument_exception","reason":"unexpected metadata [_type:flow] in source","stack_trace":"java.lang.IllegalArgumentException: unexpected metadata [_type:flow] in source
	at [email protected]/org.elasticsearch.script.CtxMap.<init>(CtxMap.java:47)
	at [email protected]/org.elasticsearch.script.UpdateCtxMap.<init>(UpdateCtxMap.java:28)
	at [email protected]/org.elasticsearch.action.update.UpdateHelper.prepareUpdateScriptRequest(UpdateHelper.java:245)
	at [email protected]/org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:79)
	at [email protected]/org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:60)
	at [email protected]/org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:184)
	at [email protected]/org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:173)
	at [email protected]/org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:60)
	at [email protected]/org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction.lambda$handleShardRequest$0(TransportInstanceSingleOperationAction.java:263)
	at [email protected]/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at [email protected]/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)

...

Into the CtxMap.java:
https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/script/CtxMap.java

  public CtxMap(Map<String, Object> source, T metadata) {
        this.source = source;
        this.metadata = metadata;
        Set<String> badKeys = Sets.intersection(this.metadata.keySet(), this.source.keySet());
        if (badKeys.size() > 0) {
            throw new IllegalArgumentException(
                "unexpected metadata ["
                    + badKeys.stream().sorted().map(k -> k + ":" + this.source.get(k)).collect(Collectors.joining(", "))
                    + "] in source"
            );
        }
    }

The issue root cause is that the field has two types at the same time: a metadata and a source .
Look at Metadata.java

https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/script/Metadata.java

public class Metadata {
...
    protected static final String TYPE = "_type"; // type is deprecated, so it's supported in the map but not available as a getter
...
    protected static final String OP = "op";
...

The Metadata fileds causes the issue.

Elasticsearch must to not allow mappings against Metadata names in fields or must handle this fields in a proper way in case names duplicates are used.

[AlexeySagin]

@williamrandolph @gwbrown @elastic/es-core-infra
Any update here?
We are using "_type" key and this is critical for project.