Failing ES Promotion: security (basic license) API Keys GET /internal/security/api_key/_enabled should indicate that API Keys are enabled

[mistic]

This failure is preventing the promotion of the current Elasticsearch nightly snapshot.

https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/2017/testReport/junit/X-Pack%20API%20Integration%20Tests%20(Security%20Basic)/x-pack_test_api_integration_apis_security_api_keys%C2%B7ts/Kibana_Pipeline___kibana_xpack_agent___security__basic_license__API_Keys_GET__internal_security_api_key__enabled_should_indicate_that_API_Keys_are_enabled_2/

For more information on the Elasticsearch snapshot promotion process: https://www.elastic.co/guide/en/kibana/master/development-es-snapshots.html

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: security (basic license)
[00:00:00]           └-> "before all" hook
[00:00:00]           └-: API Keys
[00:00:00]             └-> "before all" hook
[00:00:00]             └-: GET /internal/security/api_key/_enabled
[00:00:00]               └-> "before all" hook
[00:00:00]               └-> should indicate that API Keys are enabled
[00:00:00]                 └-> "before each" hook: global before each
[00:00:00]                 └- ✖ fail: security (basic license) API Keys GET /internal/security/api_key/_enabled should indicate that API Keys are enabled
[00:00:00]                 │      Error: expected 200 "OK", got 400 "Bad Request"
[00:00:00]                 │       at Test._assertStatus (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:268:12)
[00:00:00]                 │       at Test._assertFunction (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:283:11)
[00:00:00]                 │       at Test.assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:173:18)
[00:00:00]                 │       at assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:131:12)
[00:00:00]                 │       at /dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:128:5
[00:00:00]                 │       at Test.Request.callback (/dev/shm/workspace/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:718:3)
[00:00:00]                 │       at /dev/shm/workspace/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:906:18
[00:00:00]                 │       at IncomingMessage.<anonymous> (/dev/shm/workspace/kibana/node_modules/supertest/node_modules/superagent/lib/node/parsers/json.js:19:7)
[00:00:00]                 │       at endReadableNT (internal/streams/readable.js:1327:12)
[00:00:00]                 │       at processTicksAndRejections (internal/process/task_queues.js:80:21)
[00:00:00]                 │ 
[00:00:00]                 │ 

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[mistic]

Skipped:

master: 16fa35c

[legrego]

This might be a regression in Elasticsearch. It looks like the call to invalidate an API Key with a single ID is failing:

DELETE /_security/api_key
{
  "id": "kibana-api-key-service-test"
}

{
  "error" : {
    "root_cause" : [
      {
        "type" : "x_content_parse_exception",
        "reason" : "[2:3] [invalidate_api_key] unknown field [id] did you mean [ids]?"
      }
    ],
    "type" : "x_content_parse_exception",
    "reason" : "[2:3] [invalidate_api_key] unknown field [id] did you mean [ids]?"
  },
  "status" : 400
}

Taking Kibana out of the equation, I'm seeing the same:

➜  ~ curl -XDELETE "https://localhost:9200/_security/api_key?pretty" -H 'Content-Type: application/json' -d'{  "id": "kibana-api-key-service-test"}' -u elastic:changeme
{
  "error" : {
    "root_cause" : [
      {
        "type" : "x_content_parse_exception",
        "reason" : "[1:4] [invalidate_api_key] unknown field [id] did you mean [ids]?"
      }
    ],
    "type" : "x_content_parse_exception",
    "reason" : "[1:4] [invalidate_api_key] unknown field [id] did you mean [ids]?"
  },
  "status" : 400
}

@elastic/es-security I know we recently added support for bulk-invalidation via a new ids field, but based on the docs, it looks like the existing id field should also work. Is there a regression here, or are we using this API incorrectly?

edit was the id field removed from master already in anticipation of 8.0, but maintained in the 7.x branches? That might explain this behavior

[mistic]

@legrego I think that would be fixed by that PR #87053

[ywangd]

Sorry folks that you have to find this change out (elastic/elasticsearch#66671) through trial and error. The id field is removed from v8.0 and deprecated (but still supported) in 7.x. Next time, I'll remember to reach out for this type of changes.