Bulk request failures using the elastic/security track for nightly benchmarks

[dliappis]

When running the elastic/security track against a recent version of ES from main (3 nodes) using 8g heap -- similar to how it's run on the nightly environment we are seeing lots of bulk request failures in rally.log:

2023-04-13 06:41:54,247 ActorAddr-(T|:35053)/PID:32059 esrally.driver.runner WARNING Bulk request failed: [HTTP status: 400, message: [1:2259] failed to parse field [user.id] of type [long] in document with id 'PZBZeYcBDeuj-D9TYxz2'. Prev
iew of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2261] failed to parse field [user.id] of type [long] in document with id 'QJBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2261] f
ailed to parse field [user.id] of type [long] in document with id 'UpBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2263] failed to parse field [user.id] of type [long] in document with id 'NZBZe
YcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2263] failed to parse field [user.id] of type [long] in document with id 'OpBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 40
0, message: [1:2263] failed to parse field [user.id] of type [long] in document with id 'SJBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2263] failed to parse field [user.id] of type [long] in d
ocument with id 'X5BZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2264] failed to parse field [user.id] of type [long] in document with id 'R5BZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5
-18' | HTTP status: 400, message: [1:2264] failed to parse field [user.id] of type [long] in document with id 'RJBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2265] failed to parse field [user.i
d] of type [long] in document with id 'NpBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2265] failed to parse field [user.id] of type [long] in document with id 'UZBZeYcBDeuj-D9TYxz2'. Preview of
 field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2265] failed to parse field [user.id] of type [long] in document with id 'VJBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2265] failed 
to parse field [user.id] of type [long] in document with id 'XJBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2266] failed to parse field [user.id] of type [long] in document with id 'PpBZeYcBDeu
j-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2266] failed to parse field [user.id] of type [long] in document with id 'S5BZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, mes
sage: [1:2267] failed to parse field [user.id] of type [long] in document with id 'OZBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2267] failed to parse field [user.id] of type [long] in documen
t with id 'SpBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2267] failed to parse field [user.id] of type [long] in document with id 'VpBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' |
 HTTP status: 400, message: [1:2267] failed to parse field [user.id] of type [long] in document with id 'WZBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2267] failed to parse field [user.id] of 
type [long] in document with id 'XZBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2268] failed to parse field [user.id] of type [long] in document with id 'O5BZeYcBDeuj-D9TYxz2'. Preview of field
's value: 'S-1-5-18' | HTTP status: 400, message: [1:2268] failed to parse field [user.id] of type [long] in document with id 'U5BZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2268] failed to par
se field [user.id] of type [long] in document with id 'WJBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2268] failed to parse field [user.id] of type [long] in document with id 'WpBZeYcBDeuj-D9TY
xz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2270] failed to parse field [user.id] of type [long] in document with id 'RZBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: 
[1:2270] failed to parse field [user.id] of type [long] in document with id 'TpBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2270] failed to parse field [user.id] of type [long] in document with
 id 'V5BZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2271] failed to parse field [user.id] of type [long] in document with id 'XpBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP 
status: 400, message: [1:2271] failed to parse field [user.id] of type [long] in document with id 'YJBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2271] failed to parse field [user.id] of type [
long] in document with id 'ZZBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2272] failed to parse field [user.id] of type [long] in document with id 'QZBZeYcBDeuj-D9TYxz2'. Preview of field's val
ue: 'S-1-5-18' | HTTP status: 400, message: [1:2272] failed to parse field [user.id] of type [long] in document with id 'YZBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2272] failed to parse fie
ld [user.id] of type [long] in document with id 'ZJBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2272] failed to parse field [user.id] of type [long] in document with id 'ZpBZeYcBDeuj-D9TYxz2'. 
Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2273] failed to parse field [user.id] of type [long] in document with id 'N5BZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:227
3] failed to parse field [user.id] of type [long] in document with id 'QpBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2273] failed to parse field [user.id] of type [long] in document with id 'T
JBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2275] failed to parse field [user.id] of type [long] in document with id 'P5BZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status
: 400, message: [1:2275] failed to parse field [user.id] of type [long] in document with id 'TZBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2275] failed to parse field [user.id] of type [long] 
in document with id 'W5BZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2277] failed to parse field [user.id] of type [long] in document with id 'RpBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S
-1-5-18' | HTTP status: 400, message: [1:2278] failed to parse field [user.id] of type [long] in document with id 'OJBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2278] failed to parse field [us
er.id] of type [long] in document with id 'T5BZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2278] failed to parse field [user.id] of type [long] in document with id 'VZBZeYcBDeuj-D9TYxz2'. Previe
w of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2278] failed to parse field [user.id] of type [long] in document with id 'YpBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2280] fai
led to parse field [user.id] of type [long] in document with id 'PJBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2281] failed to parse field [user.id] of type [long] in document with id 'SZBZeYc
BDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2281] failed to parse field [user.id] of type [long] in document with id 'UJBZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400,
 message: [1:2284] failed to parse field [user.id] of type [long] in document with id 'Q5BZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18' | HTTP status: 400, message: [1:2284] failed to parse field [user.id] of type [long] in doc
ument with id 'Y5BZeYcBDeuj-D9TYxz2'. Preview of field's value: 'S-1-5-18']

Example reproduction commands:

Assuming 3 ES nodes running using a recent commit from main using 8g heap:

$ esrally race --pipeline=benchmark-only --track-repository="default" --track="elastic/security" --challenge="security-indexing-querying" --exclude-tasks='["insert-pipelines", "insert-ilm", "delete-all-datastreams", "delete-all-composable-templates", "delete-all-component-templates", "create-all-component-templates", "create-all-composable-templates", "create-required-data-streams"]' --telemetry='["disk-usage-stats"]' --on-error="abort" --target-hosts=<list of es ip addresses -- use 3 nodes> --client-options=./client-options.json --track-params=./track-params.json

$ cat ./client-options.json

{
  "default": {
    "basic_auth_password": "REDACTED",
    "timeout": 60,
    "use_ssl": true,
    "verify_certs": false,
    "basic_auth_user": "elastic"
  }
}

$ cat ./track-params.json

{
  "start_date": "2020-01-01",
  "end_date": "2020-01-02",
  "bulk_indexing_clients": 24,
  "data_generation_clients": 16,
  "raw_data_volume_per_day": "100GB",
  "max_generated_corpus_size": "40GB",
  "number_of_replicas": 1,
  "number_of_shards": 3,
  "wait_for_status": "green",
  "integration_ratios": {
    "logs-endpoint": {
      "corpora": {
        "endpoint-events-file": 0.2,
        "endpoint-events-library": 0.1,
        "endpoint-events-network": 0.2,
        "endpoint-events-process": 0.3,
        "endpoint-events-registry": 0.1,
        "endpoint-events-security": 0.1
      }
    }
  }
}