[BUG] Percentiles aggregation through cross-search fails with security_exception

[mhoydis13]

Describe the bug
The percentiles aggregation fails with strange "security_exception" message when used in conjunction with cross-search remote cluster. The same search succeeds against local data. Both clusters are configured with the exact same security configuration. The problem appears to be specific to a search involving a percentiles aggregation - other types of aggregations succeed. This issue manifests in Dashboards as well as the native _search api endpoint (via curl).

To Reproduce
Steps to reproduce the behavior:

• Setup a cluster which we'll be doing the search against, and the cross-search connection configuration is set on. (We'll call this our coordinating cluster.)
  curl --insecure -u admin:redacted -X PUT "https://my-coordinating-cluster-1:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "persistent": { "search": { "remote": { "os3": { "seeds": [ "my-data-cluster-host-1:9300", "my-data-cluster-host-2:9300", "my-data-cluster-host-3:9300" ] } } } } } '
• Setup a cluster which your indices will be located on. (We'll call this our data cluster.).
• Both clusters should be configured with the same security plugin configuration. There should be a user 'admin' which is granted the built-in roll 'all_access'.
• Populate an index on the data cluster. The index should contain multiple documents which contain a field with numeric values and mapped appropriately.
• Attempt to perform a percentiles aggregation against the index directly on the data cluster using a user which is granted the built-in roll 'all_access'.
  -- # curl --insecure -u admin:redacted -X GET "https://my-data-cluster-host-1:9200/my-index-family-000005/_search?pretty" -H 'Content-Type: application/json' -d' { "size": 0, "aggs": { "latency_by_percentile": { "percentiles": { "field": "params.latency_seconds"} } } } ' { "took" : 3317, "timed_out" : false, "_shards" : { "total" : 12, "successful" : 12, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 10000, "relation" : "gte" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "price_by_percentile" : { "values" : { "1.0" : 0.005121995169189605, "5.0" : 0.027560057707115882, "25.0" : 0.3060668743001799, "50.0" : 0.6620858926084826, "75.0" : 2.9884932599837266, "95.0" : 28.044821776170306, "99.0" : 237.3698258252764 } } } }
• Note the above search is successful. The percentiles aggregation works when performed on the index directly on the cluster it natively resides on.
• Next, perform the same search, but targeted at the coordinating cluster. Be sure to prefix the index name with the cluster name from our cross-search configuration. (os3:my-index-family-000005)
  -- curl --insecure -u admin:redacted -X GET "https://my-coordinating-cluster-1:9200/os3:my-index-family-000005/_search?pretty" -H 'Content-Type: application/json' -d' { "size": 0, "aggs": { "latency_by_percentile": { "percentiles": { "field": "params.latency_seconds"} } } } ' { "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "Unexpected exception indices:data/read/search" } ], "type" : "security_exception", "reason" : "Unexpected exception indices:data/read/search" }, "status" : 500 }

The search fails with "security_exception", citing the 'indices:data/read/search' permission.

Expected behavior
I expect the percentiles aggregation search to succeed when run against the coordinating cluster, the same way it succeeds when run directly to the data cluster.

Host/Environment (please complete the following information):

• OS: Ubuntu linux 20.04
• opensearch 1.2.4

Additional context
I also wrote this issue up in the forum: https://discuss.opendistrocommunity.dev/t/percentiles-aggregation-fails-with-security-exception-cross-search-configuration/8772

I have tried to crank up the security_auditlog to the max to find where this rejection is happening, but this is NOT being recorded in security_auditlog in either cluster no matter what I do.

I have tried granting maximum permissions to opendistro_security_anonymous, but that had no effect.

Again, this problem seems to be related uniquely to the percentiles aggregation in conjunction with remote cross-search configuration. Other types of searches I have experimented with all work, with the exception of the percentiles aggregation.

[mhoydis13]

I can confirm this problem is resolved in version 1.3.0.