CoreWithSecurityClientYamlTestSuiteIT yaml=search.aggregation/240_max_buckets/Max bucket fails reliably

[benwtrent]

Reproduces locally
Simplified Reproduce line:

./gradlew :x-pack:qa:core-rest-tests-with-security:integTest -Dtests.class=org.elasticsearch.xpack.security.CoreWithSecurityClientYamlTestSuiteIT -Dtests.method="test {yaml=search.aggregation/240_max_buckets/Max bucket}" -Dtests.seed=AE92013F2F5B6892

Reproduce line from the test (could not get to run locally at all for me):

./gradlew :x-pack:qa:core-rest-tests-with-security:integTestRunner --tests "org.elasticsearch.xpack.security.CoreWithSecurityClientYamlTestSuiteIT.test {yaml=search.aggregation/240_max_buckets/Max bucket}" -Dtests.seed=AE92013F2F5B6892 -Dtests.security.manager=true -Dtests.locale=pt -Dtests.timezone=Asia/Famagusta -Dcompiler.java=12 -Druntime.java=11 -Dtests.rest.blacklist=index/10_with_id/Index with ID,indices.get_alias/10_basic/Get alias against closed indices

This test is expected a HTTP failure code, but it is receiving a 200 with a search failure:

Suite: Test class org.elasticsearch.xpack.security.CoreWithSecurityClientYamlTestSuiteIT
  1> [2019-05-08T16:27:16,224][INFO ][o.e.x.s.CoreWithSecurityClientYamlTestSuiteIT] [test] [yaml=search.aggregation/240_max_buckets/Max bucket] before test
  1> [2019-05-08T16:27:16,336][INFO ][o.e.x.s.CoreWithSecurityClientYamlTestSuiteIT] [test] initializing REST clients against [http://[::1]:58233]
  1> [2019-05-08T16:27:16,806][INFO ][o.e.x.s.CoreWithSecurityClientYamlTestSuiteIT] [test] initializing client, minimum es version [8.0.0], master version, [8.0.0], hosts [http://[::1]:58233]
  1> [2019-05-08T16:27:18,015][INFO ][o.e.x.s.CoreWithSecurityClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "took" : 115,
  1>       "timed_out" : false,
  1>       "_shards" : {
  1>         "total" : 2,
  1>         "successful" : 1,
  1>         "skipped" : 0,
  1>         "failed" : 1,
  1>         "failures" : [
  1>           {
  1>             "shard" : 0,
  1>             "index" : "test",
  1>             "node" : "2TpSOQxvSIevZA4ChKbCpA",
  1>             "reason" : {
  1>               "type" : "too_many_buckets_exception",
  1>               "reason" : "Trying to create too many buckets. Must be less than or equal to: [3] but was [4]. This limit can be set by changing the [search.max_buckets] cluster level setting.",
  1>               "max_buckets" : 3
  1>             }
  1>           }
  1>         ]
  1>       },
  1>       "hits" : {
  1>         "total" : 2,
  1>         "max_score" : 1.0,
  1>         "hits" : [
  1>           {
  1>             "_index" : "test",
  1>             "_type" : "_doc",
  1>             "_id" : "4",
  1>             "_score" : 1.0,
  1>             "_source" : {
  1>               "date" : "2017-03-03T00:00:00",
  1>               "keyword" : "foo"
  1>             }
  1>           },
  1>           {
  1>             "_index" : "test",
  1>             "_type" : "_doc",
  1>             "_id" : "8",
  1>             "_score" : 1.0,
  1>             "_source" : {
  1>               "date" : "2021-03-03T00:00:00",
  1>               "keyword" : "quux"
  1>             }
  1>           }
  1>         ]
  1>       },
  1>       "aggregations" : {
  1>         "test" : {
  1>           "doc_count_error_upper_bound" : 0,
  1>           "sum_other_doc_count" : 0,
  1>           "buckets" : [
  1>             {
  1>               "key" : "foo",
  1>               "doc_count" : 1
  1>             },
  1>             {
  1>               "key" : "quux",
  1>               "doc_count" : 1
  1>             }
  1>           ]
  1>         }
  1>       }
  1>     }
  1>   }
  1> }]
  1> [2019-05-08T16:27:18,563][INFO ][o.e.x.s.CoreWithSecurityClientYamlTestSuiteIT] [test] [yaml=search.aggregation/240_max_buckets/Max bucket] after test
  2> java.lang.AssertionError: Failure at [search.aggregation/240_max_buckets:84]: expected [4xx|5xx] status code but api [search] returned [200 OK] [{"took":115,"timed_out":false,"_shards":{"total":2,"successful":1,"skipped":0,"failed":1,"failures":[{"shard":0,"index":"test","node":"2TpSOQxvSIevZA4ChKbCpA","reason":{"type":"too_many_buckets_exception","reason":"Trying to create too many buckets. Must be less than or equal to: [3] but was [4]. This limit can be set by changing the [search.max_buckets] cluster level setting.","max_buckets":3}}]},"hits":{"total":2,"max_score":1.0,"hits":[{"_index":"test","_type":"_doc","_id":"4","_score":1.0,"_source":{"date":"2017-03-03T00:00:00","keyword":"foo"}},{"_index":"test","_type":"_doc","_id":"8","_score":1.0,"_source":{"date":"2021-03-03T00:00:00","keyword":"quux"}}]},"aggregations":{"test":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"foo","doc_count":1},{"key":"quux","doc_count":1}]}}}]

[elasticmachine]

Pinging @elastic/es-analytics-geo

[polyfractal]

This seems to be caused by changes in #33727, unsure why it didn't fail in CI but is now. Will investigate.