Add timeouts and setup enforcement for custom plugins statuses #77965

joshdover · 2020-09-18T23:37:54Z

Summary

Introduces two changes to make Core's StatusService more resilient to issues caused by custom plugin status checks:

Enforces that core.status.set cannot be called after setup to help encourage plugins to register an observable early, ideally with an initial status
Adds a timeout to custom status Observables that defaults to unavailable if a status check is not emitted by a plugin within 30s.

This should help avoid problems like those found with the alerting and task_manager plugins in #101778

Plugin API Changes

Custom status checks registered by plugins on the core.status.set API must now emit a value within 30s or they will timeout to an unavailable status.

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

joshdover · 2021-06-12T14:52:14Z

src/core/server/status/plugins_status.test.ts

@@ -231,6 +253,75 @@ describe('PluginStatusService', () => {
        { a: { level: ServiceStatusLevels.available, summary: 'a available' } },
      ]);
    });
+
+    it('updates when a plugin status observable emits', async () => {


This test case was actually missing, even though it's the most likely behavior from a plugin.

elasticmachine · 2021-06-15T10:51:50Z

Pinging @elastic/kibana-core (Team:Core)

test/plugin_functional/test_suites/core_plugins/status.ts

joshdover · 2021-06-17T12:22:47Z

Setting back to draft until #102108 is done since it affects this PR

…nd-errors

test/server_integration/http/platform/status.ts

test/server_integration/http/platform/config.ts

…nd-errors

spalger

New plugin dir changes look good

…nd-errors

joshdover · 2021-06-23T15:31:18Z

@elasticmachine retest

joshdover · 2021-06-23T15:53:08Z

jenkins, test this

…ic#77965)

kibanamachine · 2021-06-23T18:00:42Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

… (#103149) Co-authored-by: Josh Dover <[email protected]>

kibanamachine · 2021-07-07T23:22:33Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 3c9735d
Storybooks Preview
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures

Failed CI Steps

Test Failures

Kibana Pipeline / general / X-Pack API Integration Tests.x-pack/test/api_integration/apis/management/index_lifecycle_management/policies·js.apis management index lifecycle management policies list should have a default policy to manage the Watcher history indices

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 3 times on tracked branches: https://github.com/elastic/kibana/issues/41237

[00:00:00]       │
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook in "apis"
[00:05:47]           └-: management
[00:05:47]             └-> "before all" hook in "management"
[00:06:11]             └-: index lifecycle management
[00:06:11]               └-> "before all" hook in "index lifecycle management"
[00:06:11]               └-: policies
[00:06:11]                 └-> "before all" hook in "policies"
[00:06:11]                 └-: list
[00:06:11]                   └-> "before all" hook for "should have a default policy to manage the Watcher history indices"
[00:06:11]                   └-> should have a default policy to manage the Watcher history indices
[00:06:11]                     └-> "before each" hook: global before each for "should have a default policy to manage the Watcher history indices"
[00:06:11]                     └- ✖ fail: apis management index lifecycle management policies list should have a default policy to manage the Watcher history indices
[00:06:11]                     │       Error: expected { version: 1,
[00:06:11]                     │   modified_date: '2019-04-30T14:30:00.000Z',
[00:06:11]                     │   policy: { phases: { delete: [Object] } },
[00:06:11]                     │   in_use_by: 
[00:06:11]                     │    { indices: [],
[00:06:11]                     │      data_streams: [],
[00:06:11]                     │      composable_templates: [ '.watch-history-14' ] },
[00:06:11]                     │   name: 'watch-history-ilm-policy' } to sort of equal { version: 1,
[00:06:11]                     │   modified_date: '2019-04-30T14:30:00.000Z',
[00:06:11]                     │   policy: { phases: { delete: [Object] } },
[00:06:11]                     │   name: 'watch-history-ilm-policy' }
[00:06:11]                     │       + expected - actual
[00:06:11]                     │ 
[00:06:11]                     │        {
[00:06:11]                     │       -  "in_use_by": {
[00:06:11]                     │       -    "composable_templates": [
[00:06:11]                     │       -      ".watch-history-14"
[00:06:11]                     │       -    ]
[00:06:11]                     │       -    "data_streams": []
[00:06:11]                     │       -    "indices": []
[00:06:11]                     │       -  }
[00:06:11]                     │          "modified_date": "2019-04-30T14:30:00.000Z"
[00:06:11]                     │          "name": "watch-history-ilm-policy"
[00:06:11]                     │          "policy": {
[00:06:11]                     │            "phases": {
[00:06:11]                     │       
[00:06:11]                     │       at Assertion.assert (/dev/shm/workspace/parallel/19/kibana/node_modules/@kbn/expect/expect.js:100:11)
[00:06:11]                     │       at Assertion.eql (/dev/shm/workspace/parallel/19/kibana/node_modules/@kbn/expect/expect.js:244:8)
[00:06:11]                     │       at Context.<anonymous> (test/api_integration/apis/management/index_lifecycle_management/policies.js:50:27)
[00:06:11]                     │       at Object.apply (/dev/shm/workspace/parallel/19/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)
[00:06:11]                     │ 
[00:06:11]                     │

Stack Trace

Error: expected { version: 1,
  modified_date: '2019-04-30T14:30:00.000Z',
  policy: { phases: { delete: [Object] } },
  in_use_by: 
   { indices: [],
     data_streams: [],
     composable_templates: [ '.watch-history-14' ] },
  name: 'watch-history-ilm-policy' } to sort of equal { version: 1,
  modified_date: '2019-04-30T14:30:00.000Z',
  policy: { phases: { delete: [Object] } },
  name: 'watch-history-ilm-policy' }
    at Assertion.assert (/dev/shm/workspace/parallel/19/kibana/node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/parallel/19/kibana/node_modules/@kbn/expect/expect.js:244:8)
    at Context.<anonymous> (test/api_integration/apis/management/index_lifecycle_management/policies.js:50:27)
    at Object.apply (/dev/shm/workspace/parallel/19/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16) {
  actual: '{\n' +
    '  "in_use_by": {\n' +
    '    "composable_templates": [\n' +
    '      ".watch-history-14"\n' +
    '    ]\n' +
    '    "data_streams": []\n' +
    '    "indices": []\n' +
    '  }\n' +
    '  "modified_date": "2019-04-30T14:30:00.000Z"\n' +
    '  "name": "watch-history-ilm-policy"\n' +
    '  "policy": {\n' +
    '    "phases": {\n' +
    '      "delete": {\n' +
    '        "actions": {\n' +
    '          "delete": {\n' +
    '            "delete_searchable_snapshot": true\n' +
    '          }\n' +
    '        }\n' +
    '        "min_age": "7d"\n' +
    '      }\n' +
    '    }\n' +
    '  }\n' +
    '  "version": 1\n' +
    '}',
  expected: '{\n' +
    '  "modified_date": "2019-04-30T14:30:00.000Z"\n' +
    '  "name": "watch-history-ilm-policy"\n' +
    '  "policy": {\n' +
    '    "phases": {\n' +
    '      "delete": {\n' +
    '        "actions": {\n' +
    '          "delete": {\n' +
    '            "delete_searchable_snapshot": true\n' +
    '          }\n' +
    '        }\n' +
    '        "min_age": "7d"\n' +
    '      }\n' +
    '    }\n' +
    '  }\n' +
    '  "version": 1\n' +
    '}',
  showDiff: true
}

Kibana Pipeline / general / X-Pack API Integration Tests.x-pack/test/api_integration/apis/management/index_lifecycle_management/policies·js.apis management index lifecycle management policies list should have a default policy to manage the Watcher history indices

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 3 times on tracked branches: https://github.com/elastic/kibana/issues/41237

[00:00:00]       │
[00:00:00]         │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-centos-tests-xxl-1625695132627315867] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ds-ilm-history-5-2021.07.07-000001][0]]])." previous.health="YELLOW" reason="shards started [[.ds-ilm-history-5-2021.07.07-000001][0]]"
[00:00:00]         │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xxl-1625695132627315867] moving index [.ds-ilm-history-5-2021.07.07-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [ilm-history-ilm-policy]
[00:00:00]         │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xxl-1625695132627315867] moving index [.ds-ilm-history-5-2021.07.07-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [ilm-history-ilm-policy]
[00:00:00]         │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-centos-tests-xxl-1625695132627315867] moving index [.ds-ilm-history-5-2021.07.07-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [ilm-history-ilm-policy]
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook in "apis"
[00:05:54]           └-: management
[00:05:54]             └-> "before all" hook in "management"
[00:06:18]             └-: index lifecycle management
[00:06:18]               └-> "before all" hook in "index lifecycle management"
[00:06:18]               └-: policies
[00:06:18]                 └-> "before all" hook in "policies"
[00:06:18]                 └-: list
[00:06:18]                   └-> "before all" hook for "should have a default policy to manage the Watcher history indices"
[00:06:18]                   └-> should have a default policy to manage the Watcher history indices
[00:06:18]                     └-> "before each" hook: global before each for "should have a default policy to manage the Watcher history indices"
[00:06:18]                     └- ✖ fail: apis management index lifecycle management policies list should have a default policy to manage the Watcher history indices
[00:06:18]                     │       Error: expected { version: 1,
[00:06:18]                     │   modified_date: '2019-04-30T14:30:00.000Z',
[00:06:18]                     │   policy: { phases: { delete: [Object] } },
[00:06:18]                     │   in_use_by: 
[00:06:18]                     │    { indices: [],
[00:06:18]                     │      data_streams: [],
[00:06:18]                     │      composable_templates: [ '.watch-history-14' ] },
[00:06:18]                     │   name: 'watch-history-ilm-policy' } to sort of equal { version: 1,
[00:06:18]                     │   modified_date: '2019-04-30T14:30:00.000Z',
[00:06:18]                     │   policy: { phases: { delete: [Object] } },
[00:06:18]                     │   name: 'watch-history-ilm-policy' }
[00:06:18]                     │       + expected - actual
[00:06:18]                     │ 
[00:06:18]                     │        {
[00:06:18]                     │       -  "in_use_by": {
[00:06:18]                     │       -    "composable_templates": [
[00:06:18]                     │       -      ".watch-history-14"
[00:06:18]                     │       -    ]
[00:06:18]                     │       -    "data_streams": []
[00:06:18]                     │       -    "indices": []
[00:06:18]                     │       -  }
[00:06:18]                     │          "modified_date": "2019-04-30T14:30:00.000Z"
[00:06:18]                     │          "name": "watch-history-ilm-policy"
[00:06:18]                     │          "policy": {
[00:06:18]                     │            "phases": {
[00:06:18]                     │       
[00:06:18]                     │       at Assertion.assert (/dev/shm/workspace/parallel/19/kibana/node_modules/@kbn/expect/expect.js:100:11)
[00:06:18]                     │       at Assertion.eql (/dev/shm/workspace/parallel/19/kibana/node_modules/@kbn/expect/expect.js:244:8)
[00:06:18]                     │       at Context.<anonymous> (test/api_integration/apis/management/index_lifecycle_management/policies.js:50:27)
[00:06:18]                     │       at Object.apply (/dev/shm/workspace/parallel/19/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)
[00:06:18]                     │ 
[00:06:18]                     │

Stack Trace

Error: expected { version: 1,
  modified_date: '2019-04-30T14:30:00.000Z',
  policy: { phases: { delete: [Object] } },
  in_use_by: 
   { indices: [],
     data_streams: [],
     composable_templates: [ '.watch-history-14' ] },
  name: 'watch-history-ilm-policy' } to sort of equal { version: 1,
  modified_date: '2019-04-30T14:30:00.000Z',
  policy: { phases: { delete: [Object] } },
  name: 'watch-history-ilm-policy' }
    at Assertion.assert (/dev/shm/workspace/parallel/19/kibana/node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/parallel/19/kibana/node_modules/@kbn/expect/expect.js:244:8)
    at Context.<anonymous> (test/api_integration/apis/management/index_lifecycle_management/policies.js:50:27)
    at Object.apply (/dev/shm/workspace/parallel/19/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16) {
  actual: '{\n' +
    '  "in_use_by": {\n' +
    '    "composable_templates": [\n' +
    '      ".watch-history-14"\n' +
    '    ]\n' +
    '    "data_streams": []\n' +
    '    "indices": []\n' +
    '  }\n' +
    '  "modified_date": "2019-04-30T14:30:00.000Z"\n' +
    '  "name": "watch-history-ilm-policy"\n' +
    '  "policy": {\n' +
    '    "phases": {\n' +
    '      "delete": {\n' +
    '        "actions": {\n' +
    '          "delete": {\n' +
    '            "delete_searchable_snapshot": true\n' +
    '          }\n' +
    '        }\n' +
    '        "min_age": "7d"\n' +
    '      }\n' +
    '    }\n' +
    '  }\n' +
    '  "version": 1\n' +
    '}',
  expected: '{\n' +
    '  "modified_date": "2019-04-30T14:30:00.000Z"\n' +
    '  "name": "watch-history-ilm-policy"\n' +
    '  "policy": {\n' +
    '    "phases": {\n' +
    '      "delete": {\n' +
    '        "actions": {\n' +
    '          "delete": {\n' +
    '            "delete_searchable_snapshot": true\n' +
    '          }\n' +
    '        }\n' +
    '        "min_age": "7d"\n' +
    '      }\n' +
    '    }\n' +
    '  }\n' +
    '  "version": 1\n' +
    '}',
  showDiff: true
}

Metrics [docs]

✅ unchanged

History

💚 Build #133637 succeeded 3c9735d
💔 Build #133482 failed 3c9735d
💔 Build #133121 failed 2163a27
💔 Build #133097 failed 4098de2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

joshdover mentioned this pull request Sep 22, 2020

Migrate status service and status page to New Platform #41983

Closed

15 tasks

joshdover mentioned this pull request Feb 5, 2021

Fix delayed status emissions in alerts and task_manager #90419

Closed

9 tasks

joshdover mentioned this pull request Jun 9, 2021

Fix delayed status API updates in alerting and task_manager #101778

Merged

2 tasks

joshdover force-pushed the status/timeouts-and-errors branch from e860142 to 7b728e7 Compare June 12, 2021 14:47

joshdover changed the title ~~Add timeout and error handling to custom plugins statuses~~ Add timeouts and setup enforcement for custom plugins statuses Jun 12, 2021

joshdover commented Jun 12, 2021

View reviewed changes

joshdover force-pushed the status/timeouts-and-errors branch 2 times, most recently from 8747772 to ebda07b Compare June 15, 2021 10:51

joshdover added release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.14.0 v8.0.0 labels Jun 15, 2021

joshdover marked this pull request as ready for review June 15, 2021 10:51

joshdover requested a review from a team as a code owner June 15, 2021 10:51

joshdover added the auto-backport Deprecated - use backport:version if exact versions are needed label Jun 15, 2021

joshdover enabled auto-merge (squash) June 15, 2021 10:52

pgayvallet approved these changes Jun 15, 2021

View reviewed changes

test/plugin_functional/test_suites/core_plugins/status.ts Show resolved Hide resolved

test/plugin_functional/test_suites/core_plugins/status.ts Show resolved Hide resolved

joshdover disabled auto-merge June 15, 2021 12:44

joshdover marked this pull request as draft June 17, 2021 12:22

joshdover added 5 commits June 17, 2021 20:17

Block custom status registrations after setup

7eaf06c

Add timeout handling for custom status registrations

e4b4df6

Add integration tests for custom status checks

bfe0bae

Add documentation about custom status timeout

1f90723

Move status integration tests to server_integration

c3d8571

joshdover force-pushed the status/timeouts-and-errors branch from ebda07b to c3d8571 Compare June 17, 2021 18:26

joshdover marked this pull request as ready for review June 17, 2021 18:26

joshdover enabled auto-merge (squash) June 17, 2021 18:26

Fix type error

54a83d2

Merge remote-tracking branch 'upstream/master' into status/timeouts-a…

0d24a92

…nd-errors

mshustov reviewed Jun 21, 2021

View reviewed changes

test/server_integration/http/platform/status.ts Outdated Show resolved Hide resolved

mshustov reviewed Jun 21, 2021

View reviewed changes

test/server_integration/http/platform/status.ts Outdated Show resolved Hide resolved

mshustov reviewed Jun 21, 2021

View reviewed changes

test/server_integration/http/platform/config.ts Outdated Show resolved Hide resolved

joshdover added 2 commits June 21, 2021 13:19

Merge remote-tracking branch 'upstream/master' into status/timeouts-a…

12605a7

…nd-errors

PR comments

e4625e5

joshdover force-pushed the status/timeouts-and-errors branch 3 times, most recently from 1657110 to e5239ff Compare June 21, 2021 12:27

Move status test to separate FTR config

1e9a557

joshdover force-pushed the status/timeouts-and-errors branch from e5239ff to 1e9a557 Compare June 21, 2021 16:53

joshdover requested a review from a team as a code owner June 21, 2021 16:53

spalger approved these changes Jun 21, 2021

View reviewed changes

joshdover added 2 commits June 22, 2021 13:55

Rename test plugins

147600f

Merge remote-tracking branch 'upstream/master' into status/timeouts-a…

4098de2

…nd-errors

joshdover force-pushed the status/timeouts-and-errors branch from 69e12c7 to 4098de2 Compare June 22, 2021 11:59

joshdover added 3 commits June 22, 2021 15:03

Please last CI fixes

2163a27

Merge remote-tracking branch 'upstream/master' into status/timeouts-a…

30b4e7a

…nd-errors

Fix config file name

3c9735d

joshdover merged commit 5244019 into elastic:master Jun 23, 2021

kibanamachine mentioned this pull request Jun 23, 2021

[7.x] Add timeouts and setup enforcement for custom plugins statuses (#77965) #103149

Merged

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jun 23, 2021

Add timeouts and setup enforcement for custom plugins statuses (elast…

18d7a6c

…ic#77965)

kibanamachine added a commit that referenced this pull request Jun 23, 2021

Add timeouts and setup enforcement for custom plugins statuses (#77965)…

a6985b2

… (#103149) Co-authored-by: Josh Dover <[email protected]>

joshdover mentioned this pull request Aug 2, 2021

Hitting /api/status successfully takes a very long time after start #107300

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add timeouts and setup enforcement for custom plugins statuses #77965

Add timeouts and setup enforcement for custom plugins statuses #77965

joshdover commented Sep 18, 2020 •

edited by gchaps

Loading

joshdover Jun 12, 2021

elasticmachine commented Jun 15, 2021

joshdover commented Jun 17, 2021

spalger left a comment •

edited

Loading

joshdover commented Jun 23, 2021

joshdover commented Jun 23, 2021

kibanamachine commented Jun 23, 2021

kibanamachine commented Jul 7, 2021 •

edited

Loading

Standard Out

Stack Trace

Standard Out

Stack Trace

Add timeouts and setup enforcement for custom plugins statuses #77965

Add timeouts and setup enforcement for custom plugins statuses #77965

Conversation

joshdover commented Sep 18, 2020 • edited by gchaps Loading

Summary

Plugin API Changes

Checklist

For maintainers

joshdover Jun 12, 2021

Choose a reason for hiding this comment

elasticmachine commented Jun 15, 2021

joshdover commented Jun 17, 2021

spalger left a comment • edited Loading

Choose a reason for hiding this comment

joshdover commented Jun 23, 2021

joshdover commented Jun 23, 2021

kibanamachine commented Jun 23, 2021

💚 Backport successful

kibanamachine commented Jul 7, 2021 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Standard Out

Stack Trace

Standard Out

Stack Trace

Metrics [docs]

History

joshdover commented Sep 18, 2020 •

edited by gchaps

Loading

spalger left a comment •

edited

Loading

kibanamachine commented Jul 7, 2021 •

edited

Loading