Failing test: X-Pack Alerting API Integration Tests.x-pack/test/alerting_api_integration/security_and_spaces/tests/alerting/disable·ts - alerting api integration security and spaces enabled Alerts disable space_1_all at space2 should still be able to disable alert when AAD is broken

[kibanamachine]

A test failed on a tracked branch

Error: expected 200 "OK", got 500 "Internal Server Error"
    at Test._assertStatus (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:268:12)
    at Test._assertFunction (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:283:11)
    at Test.assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:173:18)
    at assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:131:12)
    at /dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:128:5
    at Test.Request.callback (/dev/shm/workspace/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:718:3)
    at /dev/shm/workspace/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:906:18
    at IncomingMessage.<anonymous> (/dev/shm/workspace/kibana/node_modules/supertest/node_modules/superagent/lib/node/parsers/json.js:19:7)
    at endReadableNT (_stream_readable.js:1223:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

First failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

[00:13:00]                 └-> should still be able to disable alert when AAD is broken
[00:13:00]                   └-> "before each" hook: global before each
[00:13:02]                   │ proc [kibana]   log   [22:30:40.321] [info][eventLog][plugins] event logged: {"@timestamp":"2020-11-20T22:30:40.255Z","event":{"provider":"alerting","action":"execute","start":"2020-11-20T22:30:40.255Z","outcome":"success","end":"2020-11-20T22:30:40.320Z","duration":65000000},"kibana":{"saved_objects":[{"rel":"primary","type":"alert","id":"11c04c0a-d0c1-456c-9ea3-218abf4657f4","namespace":"space2"}],"alerting":{"status":"ok"},"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"alert executed: test.noop:11c04c0a-d0c1-456c-9ea3-218abf4657f4: 'abc'","ecs":{"version":"1.6.0"}}
[00:13:02]                   │ proc [kibana]   log   [22:30:40.326] [info][eventLog][plugins] event logged: {"@timestamp":"2020-11-20T22:30:40.255Z","event":{"provider":"alerting","action":"execute","start":"2020-11-20T22:30:40.255Z","outcome":"success","end":"2020-11-20T22:30:40.326Z","duration":71000000},"kibana":{"saved_objects":[{"rel":"primary","type":"alert","id":"a57d3bfc-c0f2-4b67-aba9-0a0c8cf8df7e","namespace":"space2"}],"alerting":{"status":"ok"},"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"message":"alert executed: test.noop:a57d3bfc-c0f2-4b67-aba9-0a0c8cf8df7e: 'abc'","ecs":{"version":"1.6.0"}}
[00:13:02]                   │ proc [kibana]   log   [22:30:40.328] [error][data][elasticsearch] [version_conflict_engine_exception]: [space2:alert:11c04c0a-d0c1-456c-9ea3-218abf4657f4]: version conflict, required seqNo [1230], primary term [1]. current document has seqNo [1231] and primary term [1]
[00:13:02]                   │ proc [kibana]   log   [22:30:40.329] [error][http] ResponseError: [space2:alert:11c04c0a-d0c1-456c-9ea3-218abf4657f4]: version conflict, required seqNo [1230], primary term [1]. current document has seqNo [1231] and primary term [1]: version_conflict_engine_exception
[00:13:02]                   │ proc [kibana]     at IncomingMessage.<anonymous> (/dev/shm/workspace/kibana-build-xpack-2/node_modules/@elastic/elasticsearch/lib/Transport.js:272:25)
[00:13:02]                   │ proc [kibana]     at IncomingMessage.emit (events.js:326:22)
[00:13:02]                   │ proc [kibana]     at endReadableNT (_stream_readable.js:1223:12)
[00:13:02]                   │ proc [kibana]     at processTicksAndRejections (internal/process/task_queues.js:84:21) {
[00:13:02]                   │ proc [kibana]   meta: {
[00:13:02]                   │ proc [kibana]     body: { error: [Object], status: 409 },
[00:13:02]                   │ proc [kibana]     statusCode: 409,
[00:13:02]                   │ proc [kibana]     headers: {
[00:13:02]                   │ proc [kibana]       'content-type': 'application/json; charset=UTF-8',
[00:13:02]                   │ proc [kibana]       'content-length': '623'
[00:13:02]                   │ proc [kibana]     },
[00:13:02]                   │ proc [kibana]     meta: {
[00:13:02]                   │ proc [kibana]       context: null,
[00:13:02]                   │ proc [kibana]       request: [Object],
[00:13:02]                   │ proc [kibana]       name: 'elasticsearch-js',
[00:13:02]                   │ proc [kibana]       connection: [Object],
[00:13:02]                   │ proc [kibana]       attempts: 0,
[00:13:02]                   │ proc [kibana]       aborted: false
[00:13:02]                   │ proc [kibana]     }
[00:13:02]                   │ proc [kibana]   },
[00:13:02]                   │ proc [kibana]   isBoom: true,
[00:13:02]                   │ proc [kibana]   isServer: false,
[00:13:02]                   │ proc [kibana]   data: null,
[00:13:02]                   │ proc [kibana]   output: {
[00:13:02]                   │ proc [kibana]     statusCode: 409,
[00:13:02]                   │ proc [kibana]     payload: {
[00:13:02]                   │ proc [kibana]       message: '[space2:alert:11c04c0a-d0c1-456c-9ea3-218abf4657f4]: version conflict, required seqNo [1230], primary term [1]. current document has seqNo [1231] and primary term [1]: version_conflict_engine_exception',
[00:13:02]                   │ proc [kibana]       statusCode: 409,
[00:13:02]                   │ proc [kibana]       error: 'Conflict'
[00:13:02]                   │ proc [kibana]     },
[00:13:02]                   │ proc [kibana]     headers: {}
[00:13:02]                   │ proc [kibana]   },
[00:13:02]                   │ proc [kibana]   [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/conflict'
[00:13:02]                   │ proc [kibana] }
[00:13:02]                   └- ✖ fail: alerting api integration security and spaces enabled Alerts disable space_1_all at space2 should still be able to disable alert when AAD is broken
[00:13:02]                   │      Error: expected 200 "OK", got 500 "Internal Server Error"
[00:13:02]                   │       at Test._assertStatus (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:268:12)
[00:13:02]                   │       at Test._assertFunction (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:283:11)
[00:13:02]                   │       at Test.assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:173:18)
[00:13:02]                   │       at assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:131:12)
[00:13:02]                   │       at /dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:128:5
[00:13:02]                   │       at Test.Request.callback (/dev/shm/workspace/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:718:3)
[00:13:02]                   │       at /dev/shm/workspace/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:906:18
[00:13:02]                   │       at IncomingMessage.<anonymous> (/dev/shm/workspace/kibana/node_modules/supertest/node_modules/superagent/lib/node/parsers/json.js:19:7)
[00:13:02]                   │       at endReadableNT (_stream_readable.js:1223:12)
[00:13:02]                   │       at processTicksAndRejections (internal/process/task_queues.js:84:21)
[00:13:02]                   │ 
[00:13:02]                   │

[pmuellr]

Yikes! a 409 in the alerting code resulted in a 500 being returned from an alerting endpoint. Not good. So, probably two bugs to fix here, whatever caused the 500 needs to be fixed first. And I thought we had code that would retry conflicts like this, so that we shouldn't be sending 409's back through the http endpoints. My guess is that is is the retry logic that is causing the 500's. And that there should be tests for this, but maybe there aren't, or if we do have some, they are insufficient.

[kibanamachine]

New failure: Jenkins Build