[test-failed]: Chrome UI Functional Tests1.test/functional/apps/status_page/index·ts - status page should display the server status #92299

liza-mae · 2021-02-22T22:26:54Z

Version: 7.12.0
Class: Chrome UI Functional Tests1.test/functional/apps/status_page/index·ts
Stack Trace:

Error: expected 'Red' to equal 'Green'
   at Assertion.assert (packages/kbn-expect/expect.js:100:11)
   at Assertion.be.Assertion.equal (packages/kbn-expect/expect.js:227:8)
   at Assertion.be (packages/kbn-expect/expect.js:69:22)
   at Context.<anonymous> (test/functional/apps/status_page/index.ts:41:31)
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (internal/process/task_queues.js:93:5)
   at Object.apply (packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)

Other test failures:

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/1375/testReport/

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-02-22T23:23:17Z

Pinging @elastic/kibana-core (Team:Core)

liza-mae · 2021-02-22T23:23:45Z

I am not sure what is causing the instance to go to red, until we know more I am going to label as a bug so we can investigate it with higher priority.

liza-mae · 2021-02-22T23:30:48Z

I am also seeing inconsistency in data return between /status and api/status on my cluster, why?

For example:

liza-mae · 2021-02-22T23:31:02Z

Test screenshot:

liza-mae · 2021-02-22T23:45:03Z

Also to note, before the tests start the status is green.

{"overall":{"since":"2021-02-22T13:44:18.623Z","state":"green","title":"Green","nickname":"Looking good","icon":"success","uiColor":"secondary"}

liza-mae · 2021-02-23T16:12:29Z

Cloud reports Kibana as healthy and operationally I have not seen mass failures.

liza-mae · 2021-02-23T16:22:16Z

[alerts]: Alerting framework is unavailable

elasticmachine · 2021-02-23T16:22:29Z

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

liza-mae · 2021-02-23T16:22:58Z

cc: Alerting team - would alerting framework being unavailable cause this ? How would we debug ?

pmuellr · 2021-02-23T19:54:57Z

You can do some amount of debugging by looking at the api status, via:

curl "$KBN_URLBASE/api/status?v8format=true"

I just tried that with a new-ish 7.11.0 deployment, and can see there's a 503 error for the alert health code trying to access the .kibana_task_manager index.

pmuellr · 2021-02-23T19:56:46Z

Should mention that it makes that check against the task manager index every 5 minutes, so I'd expect if the 503 is transient, it will only affect a 5 minute window of time - of course that's still awful, but it would be interesting to know if this status ends up "clearing" over time.

liza-mae · 2021-02-23T19:58:42Z

Thanks @pmuellr in my case it did not appear to clear.

liza-mae · 2021-02-23T20:00:51Z

Just want to add that my cluster is on staging, which has seen higher amounts of service unavailable.

liza-mae · 2021-02-23T20:03:00Z

How does this impact how alerting functions ?

pmuellr · 2021-02-23T20:26:04Z

How does this impact how alerting functions ?

As near as I can tell, it doesn't.

In my deployment, while seeing the red status on alerting, I can run an existing alert and see it firing. Meaning all sorts of things are in working order - alerting, task manager, saved objects, etc.

Honestly not sure that we USE the health status to gate any function - it appears we don't for normal alert operations - we're just updating the core status every couple of minutes, and somewhere along the way, if an error occurs, it will get stuck on that error.

It's also unfortunate in that status blob we don't have any dates associated with the individual plugin status - I bet we'd see the date was some time ago. We'd also want to add dates to the blob that the alerting framework provides to the status, since there's the date we query task manager, and also the date the task manager document was last updated, all of which would be good to have for diagnostic purposes.

liza-mae · 2021-03-01T16:43:59Z

#93062

liza-mae · 2021-03-02T21:46:52Z

I have a case where the status is going red on kibana but I did not receive any 503 service unavailable errors.

liza-mae · 2021-03-02T21:50:35Z

Just noting it looks like it is expected to fail on any http error code and does not clear.

ymao1 · 2021-03-03T13:13:30Z

@liza-mae I have a PR to address some of the issues with the alerting health status check here: #93282. Is there a way to run these tests on the PR branch to see if it resolves this issue?

liza-mae · 2021-03-03T17:06:18Z

Hi @ymao1 thanks for cloud we can't test the PR, however I am seeing this test fail on local testing also, maybe if we can reproduce we can test it that way. It is not service available causing the issue in what I am seeing.

liza-mae · 2021-03-03T19:08:29Z

Okay so I am seeing this occur in two different ways. 1) Is through cloud, a system running for awhile hits service unavailable and the status gets stuck in red. 2) Through our function tests locally running the test test/functional/apps/saved_objects_management/show_relationships.ts this also puts the server in red position, but I don't know enough about the test to know why. I have collected logs for this case though.

@ymao1
For 1) to test on cloud, we would need to merge your PR and wait for next available snapshot.
For 2) we an reproduce easily through running the functional test above, we may need someone from core team to take a look at the error logs, I will provide them and cc them.

liza-mae · 2021-03-03T19:13:10Z

@elastic/kibana-core and @elastic/kibana-security teams when I run this test test/functional/apps/saved_objects_management/show_relationships.ts it puts the server in red status. Can someone take a look as to why, debug logs to follow.

liza-mae · 2021-03-03T19:13:58Z

Server Log

  log   [18:52:02.134] [info][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET
  log   [18:52:02.140] [info][savedobjects-service] [.kibana] INIT -> LEGACY_SET_WRITE_BLOCK
  log   [18:52:02.235] [info][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY
  log   [18:52:02.255] [info][savedobjects-service] [.kibana] LEGACY_SET_WRITE_BLOCK -> LEGACY_CREATE_REINDEX_TARGET
  log   [18:52:02.284] [info][savedobjects-service] [.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE
  log   [18:52:02.285] [info][savedobjects-service] [.kibana_task_manager] Migration completed after 155ms
  log   [18:52:02.357] [info][savedobjects-service] [.kibana] LEGACY_CREATE_REINDEX_TARGET -> LEGACY_REINDEX
  log   [18:52:02.368] [info][savedobjects-service] [.kibana] LEGACY_REINDEX -> LEGACY_REINDEX_WAIT_FOR_TASK
  log   [18:52:02.478] [info][savedobjects-service] [.kibana] LEGACY_REINDEX_WAIT_FOR_TASK -> LEGACY_DELETE
  log   [18:52:02.559] [info][savedobjects-service] [.kibana] LEGACY_DELETE -> SET_SOURCE_WRITE_BLOCK
  log   [18:52:02.608] [info][savedobjects-service] [.kibana] SET_SOURCE_WRITE_BLOCK -> CREATE_REINDEX_TEMP
  log   [18:52:02.685] [info][savedobjects-service] [.kibana] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP
  log   [18:52:02.697] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP -> REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK
  log   [18:52:02.809] [info][savedobjects-service] [.kibana] REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK -> SET_TEMP_WRITE_BLOCK
  log   [18:52:02.852] [info][savedobjects-service] [.kibana] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET
  log   [18:52:02.970] [info][savedobjects-service] [.kibana] CLONE_TEMP_TO_TARGET -> OUTDATED_DOCUMENTS_SEARCH
  log   [18:52:02.982] [info][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH -> OUTDATED_DOCUMENTS_TRANSFORM
  log   [18:52:03.093] [warning][environment] Detected an unhandled Promise rejection.
ResponseError: Saved object index alias [.kibana_7.12.0] not found: index_not_found_exception
  log   [18:52:03.893] [info][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_TRANSFORM -> OUTDATED_DOCUMENTS_SEARCH
  log   [18:52:03.898] [info][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH -> UPDATE_TARGET_MAPPINGS
  log   [18:52:03.950] [info][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK
  log   [18:52:04.055] [info][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> MARK_VERSION_INDEX_READY
  log   [18:52:04.130] [info][savedobjects-service] [.kibana] MARK_VERSION_INDEX_READY -> DONE
  log   [18:52:04.130] [info][savedobjects-service] [.kibana] Migration completed after 2001ms
  log   [18:52:08.696] [info][plugins][routes][security] Logging in with provider "basic" (basic)
 error  [18:52:10.991] [error][client][connection] Error: 140594917275456:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl
/record/ssl3_record.c:332:

 error  [18:52:12.046] [error][client][connection] Error: 140594917275456:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl
/record/ssl3_record.c:332:

liza-mae · 2021-03-03T19:15:03Z

https://localhost:5601/api/status?v8format=true

Shows not found

{"name":"packer-virtualbox-iso-1558277077","uuid":"b594d02b-afe4-46a1-8575-ddd9cdca51b4","version":{"number":"7.12.0","build_hash":"08417cbd6c15e4c866651a7dcdfeded58845206d","build_number":39134,"build_snapshot":false},"status":{"overall":{"level":"unavailable","summary":"[11] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}},"uptime":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"apm":{"level":"unavailable","summary":"[4] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"infra":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"case":{"level":"unavailable","summary":"[securitySolution]: [3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"securitySolution":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}}}}},"infra":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"logstash":{"level":"unavailable","summary":"[monitoring]: [3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"monitoring":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"infra":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"monitoring":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"infra":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}},"securitySolution":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"stackAlerts":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"}}}},"core":{"elasticsearch":{"level":"available","summary":"Elasticsearch is available","meta":{"warningNodes":[],"incompatibleNodes":[]}},"savedObjects":{"level":"available","summary":"SavedObjects service has completed migrations and is available","meta":{"migratedIndices":{"migrated":0,"skipped":0,"patched":2}}}},"plugins":{"advancedSettings":{"level":"available","summary":"All dependencies are available"},"bfetch":{"level":"available","summary":"All dependencies are available"},"charts":{"level":"available","summary":"All dependencies are available"},"devTools":{"level":"available","summary":"All dependencies are available"},"discover":{"level":"available","summary":"All dependencies are available"},"embeddable":{"level":"available","summary":"All dependencies are available"},"esUiShared":{"level":"available","summary":"All dependencies are available"},"expressions":{"level":"available","summary":"All dependencies are available"},"indexPatternManagement":{"level":"available","summary":"All dependencies are available"},"inspector":{"level":"available","summary":"All dependencies are available"},"kibanaOverview":{"level":"available","summary":"All dependencies are available"},"kibanaReact":{"level":"available","summary":"All dependencies are available"},"kibanaUsageCollection":{"level":"available","summary":"All dependencies are available"},"kibanaUtils":{"level":"available","summary":"All dependencies are available"},"legacyExport":{"level":"available","summary":"All dependencies are available"},"management":{"level":"available","summary":"All dependencies are available"},"navigation":{"level":"available","summary":"All dependencies are available"},"presentationUtil":{"level":"available","summary":"All dependencies are available"},"savedObjects":{"level":"available","summary":"All dependencies are available"},"savedObjectsManagement":{"level":"available","summary":"All dependencies are available"},"savedObjectsTaggingOss":{"level":"available","summary":"All dependencies are available"},"share":{"level":"available","summary":"All dependencies are available"},"spacesOss":{"level":"available","summary":"All dependencies are available"},"telemetryCollectionManager":{"level":"available","summary":"All dependencies are available"},"telemetryManagementSection":{"level":"available","summary":"All dependencies are available"},"uiActions":{"level":"available","summary":"All dependencies are available"},"urlForwarding":{"level":"available","summary":"All dependencies are available"},"visDefaultEditor":{"level":"available","summary":"All dependencies are available"},"visTypeXy":{"level":"available","summary":"All dependencies are available"},"visualizations":{"level":"available","summary":"All dependencies are available"},"visualize":{"level":"available","summary":"All dependencies are available"},"canvas":{"level":"available","summary":"All dependencies are available"},"dashboardEnhanced":{"level":"available","summary":"All dependencies are available"},"embeddableEnhanced":{"level":"available","summary":"All dependencies are available"},"features":{"level":"available","summary":"All dependencies are available"},"fileUpload":{"level":"available","summary":"All dependencies are available"},"globalSearchBar":{"level":"available","summary":"All dependencies are available"},"globalSearchProviders":{"level":"available","summary":"All dependencies are available"},"grokdebugger":{"level":"available","summary":"All dependencies are available"},"ingestPipelines":{"level":"available","summary":"All dependencies are available"},"mapsLegacyLicensing":{"level":"available","summary":"All dependencies are available"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}},"painlessLab":{"level":"available","summary":"All dependencies are available"},"runtimeFields":{"level":"available","summary":"All dependencies are available"},"searchprofiler":{"level":"available","summary":"All dependencies are available"},"telemetryCollectionXpack":{"level":"available","summary":"All dependencies are available"},"transform":{"level":"available","summary":"All dependencies are available"},"translations":{"level":"available","summary":"All dependencies are available"},"uiActionsEnhanced":{"level":"available","summary":"All dependencies are available"},"uptime":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"watcher":{"level":"available","summary":"All dependencies are available"},"xpackLegacy":{"level":"available","summary":"All dependencies are available"},"urlDrilldown":{"level":"available","summary":"All dependencies are available"},"apmOss":{"level":"available","summary":"All dependencies are available"},"console":{"level":"available","summary":"All dependencies are available"},"dashboard":{"level":"available","summary":"All dependencies are available"},"data":{"level":"available","summary":"All dependencies are available"},"home":{"level":"available","summary":"All dependencies are available"},"inputControlVis":{"level":"available","summary":"All dependencies are available"},"kibanaLegacy":{"level":"available","summary":"All dependencies are available"},"mapsLegacy":{"level":"available","summary":"All dependencies are available"},"newsfeed":{"level":"available","summary":"All dependencies are available"},"regionMap":{"level":"available","summary":"All dependencies are available"},"securityOss":{"level":"available","summary":"All dependencies are available"},"telemetry":{"level":"available","summary":"All dependencies are available"},"tileMap":{"level":"available","summary":"All dependencies are available"},"timelion":{"level":"available","summary":"All dependencies are available"},"usageCollection":{"level":"available","summary":"All dependencies are available"},"visTypeMarkdown":{"level":"available","summary":"All dependencies are available"},"visTypeMetric":{"level":"available","summary":"All dependencies are available"},"visTypeTable":{"level":"available","summary":"All dependencies are available"},"visTypeTagcloud":{"level":"available","summary":"All dependencies are available"},"visTypeTimelion":{"level":"available","summary":"All dependencies are available"},"visTypeTimeseries":{"level":"available","summary":"All dependencies are available"},"visTypeVega":{"level":"available","summary":"All dependencies are available"},"visTypeVislib":{"level":"available","summary":"All dependencies are available"},"actions":{"level":"available","summary":"All dependencies are available"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"apm":{"level":"unavailable","summary":"[4] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"infra":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"banners":{"level":"available","summary":"All dependencies are available"},"beatsManagement":{"level":"available","summary":"All dependencies are available"},"case":{"level":"unavailable","summary":"[securitySolution]: [3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"securitySolution":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}}}}},"cloud":{"level":"available","summary":"All dependencies are available"},"code":{"level":"available","summary":"All dependencies are available"},"consoleExtensions":{"level":"available","summary":"All dependencies are available"},"crossClusterReplication":{"level":"available","summary":"All dependencies are available"},"dashboardMode":{"level":"available","summary":"All dependencies are available"},"dataEnhanced":{"level":"available","summary":"All dependencies are available"},"discoverEnhanced":{"level":"available","summary":"All dependencies are available"},"encryptedSavedObjects":{"level":"available","summary":"All dependencies are available"},"enterpriseSearch":{"level":"available","summary":"All dependencies are available"},"eventLog":{"level":"available","summary":"All dependencies are available"},"fleet":{"level":"available","summary":"All dependencies are available"},"globalSearch":{"level":"available","summary":"All dependencies are available"},"graph":{"level":"available","summary":"All dependencies are available"},"indexLifecycleManagement":{"level":"available","summary":"All dependencies are available"},"indexManagement":{"level":"available","summary":"All dependencies are available"},"infra":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"lens":{"level":"available","summary":"All dependencies are available"},"licenseManagement":{"level":"available","summary":"All dependencies are available"},"licensing":{"level":"available","summary":"All dependencies are available"},"lists":{"level":"available","summary":"All dependencies are available"},"logstash":{"level":"unavailable","summary":"[monitoring]: [3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"monitoring":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"infra":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"maps":{"level":"available","summary":"All dependencies are available"},"monitoring":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"infra":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}},"observability":{"level":"available","summary":"All dependencies are available"},"remoteClusters":{"level":"available","summary":"All dependencies are available"},"reporting":{"level":"available","summary":"All dependencies are available"},"rollup":{"level":"available","summary":"All dependencies are available"},"savedObjectsTagging":{"level":"available","summary":"All dependencies are available"},"security":{"level":"available","summary":"All dependencies are available"},"securitySolution":{"level":"unavailable","summary":"[3] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"ml":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}}}}}},"snapshotRestore":{"level":"available","summary":"All dependencies are available"},"spaces":{"level":"available","summary":"All dependencies are available"},"stackAlerts":{"level":"unavailable","summary":"[2] services are unavailable","detail":"See the status page for more information","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"}}}},"taskManager":{"level":"available","summary":"All dependencies are available"},"triggersActionsUi":{"level":"unavailable","summary":"[alerts]: Alerting framework is unavailable","meta":{"affectedServices":{"alerts":{"level":"unavailable","summary":"Alerting framework is unavailable","meta":{"error":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Saved object [task/Alerting-alerting_health_check] not found"},"headers":{}}}}}}},"detail":"See the status page for more information"},"upgradeAssistant":{"level":"available","summary":"All dependencies are available"}}},"metrics":{"last_updated":"2021-03-03T19:04:49.576Z","collection_interval_in_millis":5000,"os":{"platform":"linux","platformRelease":"linux-4.4.0-148-generic","load":{"1m":0.14,"5m":0.29,"15m":0.36},"memory":{"total_in_bytes":4143390720,"free_in_bytes":180555776,"used_in_bytes":3962834944},"uptime_in_millis":1987000,"distro":"Ubuntu","distroRelease":"Ubuntu-16.04","cpuacct":{"control_group":"/","usage_nanos":683655347807},"cpu":{"control_group":"/","cfs_period_micros":100000,"cfs_quota_micros":-1,"stat":{"number_of_elapsed_periods":0,"number_of_times_throttled":0,"time_throttled_nanos":0}}},"process":{"memory":{"heap":{"total_in_bytes":191401984,"used_in_bytes":173407112,"size_limit":2122055680},"resident_set_size_in_bytes":275910656},"pid":21365,"event_loop_delay":0.5441600000485778,"uptime_in_millis":397396.071549},"response_times":{"avg_in_millis":0,"max_in_millis":0},"concurrent_connections":0,"requests":{"disconnects":0,"total":0,"statusCodes":{},"status_codes":{}}}}

ymao1 · 2021-03-03T19:40:48Z

@liza-mae

It looks like the alerting framework health check is failing with Saved object [task/Alerting-alerting_health_check] not found which is causing all dependent plugins to become unavailable.

The alerting framework creates a task manager task to check the alert status. Then it periodically performs a GET on that task in order to check the output of that task. It looks to me like it's unable to find that saved object and throwing a 404 error.

I recently fixed another flaky functional test that was caused by trying to access a saved object mid migration when starting up the test and this feels very similar to me.

liza-mae · 2021-03-03T19:52:04Z

After messaging with @ymao1 looks like we have two different issues then, one is just a test issue limitation with FTR. For the cloud one, we won't be able to test it until the PR is merged and is contained in next cloud snapshot.

liza-mae · 2021-03-11T22:26:33Z

@kobelb gave me this command:

DELETE .kibana_task_manager/_doc/task:Alerting-alerting_health_check

Which does cause the status to go red and throws a not found error, is that what the functional tests are causing to happen? If so, is it a test issue and how do we fix?

cc @elastic/kibana-core

pgayvallet · 2021-03-12T13:10:12Z

Which does cause the status to go red and throws a not found error, is that what the functional tests are causing to happen?

No, the test is just accessing the status page and asserting that the status is Green. It doesn't do anything to cause a Red status.

From your logs in #92299 (comment), it seems the failure was caused by a migration issue.

  log   [18:52:03.093] [warning][environment] Detected an unhandled Promise rejection.
ResponseError: Saved object index alias [.kibana_7.12.0] not found: index_not_found_exception

How often is the test failing exactly?

liza-mae · 2021-03-12T19:50:40Z

Thanks @pgayvallet for taking a look, it is failing consistently since 7.12 and it was not failing before.

ymao1 · 2021-03-23T16:43:11Z

This PR addressed the issue of the alerting status health check never recovering from a transient failure, which was brought to light by this issue. Although that PR does not resolve the overall issue, I believe it does resolve the alerting portion of it.

liza-mae · 2021-04-19T21:32:13Z

Version: 7.12.1
Class: Chrome UI Functional Tests1.test/functional/apps/status_page/index·ts
Stack Trace:

Error: expected 'Red' to equal 'Green'
   at Assertion.assert (packages/kbn-expect/expect.js:100:11)
   at Assertion.be.Assertion.equal (packages/kbn-expect/expect.js:227:8)
   at Assertion.be (packages/kbn-expect/expect.js:69:22)
   at Context.<anonymous> (test/functional/apps/status_page/index.ts:41:31)
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (internal/process/task_queues.js:93:5)
   at Object.apply (packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)

Other test failures:

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/1653/testReport/

liza-mae · 2021-04-26T22:05:53Z

Version: 7.13.0
Class: Chrome UI Functional Tests1.test/functional/apps/status_page/index·ts
Stack Trace:

Error: expected 'Red' to equal 'Green'
   at Assertion.assert (node_modules/@kbn/expect/expect.js:100:11)
   at Assertion.be.Assertion.equal (node_modules/@kbn/expect/expect.js:227:8)
   at Assertion.be (node_modules/@kbn/expect/expect.js:69:22)
   at Context.<anonymous> (test/functional/apps/status_page/index.ts:41:31)
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (internal/process/task_queues.js:93:5)
   at Object.apply (node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)

Other test failures:

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/1685/testReport/

tylersmalley · 2021-04-28T20:56:27Z

This might be similar to the issue I described here: #94968 (comment)

gmmorris · 2021-08-11T08:05:02Z

This PR addressed the issue of the alerting status health check never recovering from a transient failure, which was brought to light by this issue. Although that PR does not resolve the overall issue, I believe it does resolve the alerting portion of it.

Given this update by @ymao1 I'm removing this bug from the Alerting feature/team.
I'm not sure what feature this really belongs to now... there doesn't seem to be a StatusPage one 😬 I'll add that for now...

liza-mae added failed-test A test failure on a tracked branch, potentially flaky-test test-cloud Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc labels Feb 22, 2021

liza-mae added the bug Fixes for quality problems that affect the customer experience label Feb 22, 2021

liza-mae added the Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) label Feb 23, 2021

ymao1 self-assigned this Feb 25, 2021

ymao1 removed their assignment Mar 23, 2021

joshdover assigned pgayvallet Mar 24, 2021

liza-mae mentioned this issue Apr 28, 2021

Unable to load status page if red #98614

Closed

gmmorris added Feature:Task Manager Feature:Alerting and removed Feature:Task Manager labels Jul 2, 2021

gmmorris removed Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels Aug 11, 2021

gmmorris added the Feature:StatusPage Issues related to the Kibana Status Page and APIs label Aug 11, 2021

liza-mae closed this as completed Mar 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[test-failed]: Chrome UI Functional Tests1.test/functional/apps/status_page/index·ts - status page should display the server status #92299

[test-failed]: Chrome UI Functional Tests1.test/functional/apps/status_page/index·ts - status page should display the server status #92299

liza-mae commented Feb 22, 2021

elasticmachine commented Feb 22, 2021

liza-mae commented Feb 22, 2021

liza-mae commented Feb 22, 2021

liza-mae commented Feb 22, 2021

liza-mae commented Feb 22, 2021

liza-mae commented Feb 23, 2021

liza-mae commented Feb 23, 2021

elasticmachine commented Feb 23, 2021

liza-mae commented Feb 23, 2021

pmuellr commented Feb 23, 2021

pmuellr commented Feb 23, 2021

liza-mae commented Feb 23, 2021

liza-mae commented Feb 23, 2021

liza-mae commented Feb 23, 2021

pmuellr commented Feb 23, 2021

liza-mae commented Mar 1, 2021

liza-mae commented Mar 2, 2021

liza-mae commented Mar 2, 2021

ymao1 commented Mar 3, 2021

liza-mae commented Mar 3, 2021

liza-mae commented Mar 3, 2021

liza-mae commented Mar 3, 2021 •

edited

Loading

liza-mae commented Mar 3, 2021

liza-mae commented Mar 3, 2021 •

edited

Loading

ymao1 commented Mar 3, 2021

liza-mae commented Mar 3, 2021

liza-mae commented Mar 11, 2021

pgayvallet commented Mar 12, 2021

liza-mae commented Mar 12, 2021

ymao1 commented Mar 23, 2021

liza-mae commented Apr 19, 2021

liza-mae commented Apr 26, 2021

tylersmalley commented Apr 28, 2021

gmmorris commented Aug 11, 2021

[test-failed]: Chrome UI Functional Tests1.test/functional/apps/status_page/index·ts - status page should display the server status #92299

[test-failed]: Chrome UI Functional Tests1.test/functional/apps/status_page/index·ts - status page should display the server status #92299

Comments

liza-mae commented Feb 22, 2021

elasticmachine commented Feb 22, 2021

liza-mae commented Feb 22, 2021

liza-mae commented Feb 22, 2021

liza-mae commented Feb 22, 2021

liza-mae commented Feb 22, 2021

liza-mae commented Feb 23, 2021

liza-mae commented Feb 23, 2021

elasticmachine commented Feb 23, 2021

liza-mae commented Feb 23, 2021

pmuellr commented Feb 23, 2021

pmuellr commented Feb 23, 2021

liza-mae commented Feb 23, 2021

liza-mae commented Feb 23, 2021

liza-mae commented Feb 23, 2021

pmuellr commented Feb 23, 2021

liza-mae commented Mar 1, 2021

liza-mae commented Mar 2, 2021

liza-mae commented Mar 2, 2021

ymao1 commented Mar 3, 2021

liza-mae commented Mar 3, 2021

liza-mae commented Mar 3, 2021

liza-mae commented Mar 3, 2021 • edited Loading

liza-mae commented Mar 3, 2021

liza-mae commented Mar 3, 2021 • edited Loading

ymao1 commented Mar 3, 2021

liza-mae commented Mar 3, 2021

liza-mae commented Mar 11, 2021

pgayvallet commented Mar 12, 2021

liza-mae commented Mar 12, 2021

ymao1 commented Mar 23, 2021

liza-mae commented Apr 19, 2021

liza-mae commented Apr 26, 2021

tylersmalley commented Apr 28, 2021

gmmorris commented Aug 11, 2021

liza-mae commented Mar 3, 2021 •

edited

Loading

liza-mae commented Mar 3, 2021 •

edited

Loading