Add support for /api/status before Kibana completes startup #79012

joshdover · 2020-09-30T19:06:47Z

Summary

Add the /api/status endpoint to the "not ready server" that is served before Kibana starts
Makes this endpoint return a 503 until all services are degraded or available
Expose detailed information on the status API when migrations are waiting for other nodes to complete:

  {
    "id": "core:[email protected]",
    "message": "SavedObjects service is waiting for other nodes to complete the migration If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_2 and restarting Kibana.",
    "since": "2020-09-30T19:30:43.268Z",
    "state": "red",
    "icon": "danger",
    "uiColor": "danger"
  },

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
This was checked for keyboard-only and screenreader accessibility
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

src/core/server/http/http_service.ts

joshdover · 2020-09-30T19:43:32Z

I haven't updated any unit tests yet, so this is sure to break some things right now.

src/core/server/http/http_service.ts

src/core/server/http/http_server.ts

mshustov · 2020-10-01T08:44:32Z

src/core/server/http/http_service.ts

-    this.notReadyServer.route({
+    this.notReadyServer = new HttpServer(this.logger, 'NotReady');
+    const notReadySetup = await this.notReadyServer.setup(config);
+    notReadySetup.server.route({


Let's use newly added API instead of hapi one?

I was going to do that, but since we don't support some of the HTTP methods right now, I left it like this so that it covers any request.

src/core/server/status/legacy_status.ts

src/core/server/status/status_service.ts

…nodes

legrego · 2021-01-26T21:06:22Z

@joshdover if you want to bring this back up-to-date with master, I can pull this down and try to work through some of the test failures that we discussed today.

joshdover · 2021-01-27T12:58:47Z

@legrego Sure thing, I should be able to get to this tomorrow, if that's alright. I'm also thinking that we may need to do some refactoring for #89287. For one, this should probably be called something different than the "notReadyServer", maybe "initializationServer"?.

I also think we may need to formalize an API on the InternalHttpServiceSetup for the interactive setup process. I think it will be best if we don't initialize any plugins before we configure ES. This would make it more foolproof that we don't break any plugin code. Note that plugins are constructed during Server.setup and not during Server.prototype.constructor, so we should be able to block on the interactive setup before we call pluginsServer.setup.

…ady-status

legrego · 2021-04-07T18:06:02Z

@elasticmachine merge upstream

…pdate-not-ready-status

…as started listening

legrego · 2021-04-08T19:01:31Z

@joshdover I brought this PR back up to date with master, started addressing the first round of feedback from @mshustov, and fixed the test failures we discussed.

In short, there were three main reasons for the test failures:

The bypassErrorFormatting was being applied too broadly. I updated the options for the custom response handler to allow consumers to decide if error formatting should be bypassed: 78cd007 (#79012)
The status endpoint returns a 503 now if any service is unavailable. This caused the KbnClient to fail in those cases, so I introduced a mechanism for client callers to indicate that specific error codes were acceptable: e7f355b (#79012)
Now that the NotReady server is a proper HttpServer instance, it ends up logging a message indicating that it has started up. This message was being improperly interpreted by the FTR, so the FTR thought that the real ~~slim shady~~ Kibana server was up and running. I updated this check to test for the Kibana server specifically: 8791a24 (#79012)

elasticmachine · 2021-04-22T15:48:41Z

Pinging @elastic/kibana-core (Team:Core)

jbudz

Is there any expectation for supporting basic auth (from the browser) before startup completes? The login page is unavailable, and api/status will return a 401.

Have we put any thought into whether this is a breaking change, and documenting best practices for status monitoring? I'm okay with addressing this separately.

Changes LGTM otherwise.

legrego · 2021-04-22T16:49:32Z

Is there any expectation for supporting basic auth (from the browser) before startup completes? The login page is unavailable, and api/status will return a 401.

@jbudz great question -- the NotReady server can't support basic auth at this point, unless we were to hardcode credentials into kibana.yml or similar to use. It does make me wonder if we should only register the /api/status route on the NotReady server if Kibana has been configured to explicitly allow anonymous access to the status endpoint. @joshdover what are your thoughts on that?

Have we put any thought into whether this is a breaking change, and documenting best practices for status monitoring? I'm okay with addressing this separately.

I personally hadn't considered it, so thank you for raising this. Our docs only make a passing reference to /api/status, so in that regard I wouldn't consider this breaking.

That said, I don't know how other stack components consume this endpoint

joshdover · 2021-04-22T16:59:15Z

It does make me wonder if we should only register the /api/status route on the NotReady server if Kibana has been configured to explicitly allow anonymous access to the status endpoint. @joshdover what are your thoughts on that?

Yeah that would make some sense, though it doesn't feel great to have this sometimes available. We'd likely replace this with a more limited health check endpoint that's safe to be public at some point. (#46984).

…s allowed

legrego · 2021-04-23T11:41:46Z

@elasticmachine merge upstream

legrego · 2021-04-23T13:40:53Z

@mshustov would you mind reviewing one last time to get the requested codeowner approval from kibana-core?

legrego · 2021-04-26T13:56:29Z

@elasticmachine merge upstream

mshustov · 2021-04-26T15:54:39Z

@legrego I don't know why GH didn't count my approval #79012 (review) (probably because I did it on the draft state). Approved again.

kibanamachine · 2021-04-26T15:57:40Z

💚 Build Succeeded

Metrics [docs]

Unknown metric groups

API count

id	before	after	diff
`core`	2180	2182	+2

History

💚 Build #121991 succeeded e9ce811
💚 Build #121726 succeeded ecd4cbf
💚 Build #119347 succeeded c0aebe1
💚 Build #118900 succeeded 11f0a0a
💚 Build #118335 succeeded 2e0ac39

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…79012) Co-authored-by: Larry Gregory <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

kibanamachine · 2021-04-26T16:01:36Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…98329) Co-authored-by: Larry Gregory <[email protected]> Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Josh Dover <[email protected]> Co-authored-by: Larry Gregory <[email protected]>

joshdover commented Sep 30, 2020

View reviewed changes

src/core/server/http/http_service.ts Outdated Show resolved Hide resolved

joshdover force-pushed the not-ready-status branch from a7e3af4 to a0790a7 Compare September 30, 2020 19:36

mshustov reviewed Oct 1, 2020

View reviewed changes

joshdover force-pushed the not-ready-status branch from a0790a7 to 1fc08ef Compare October 2, 2020 22:29

joshdover added 5 commits December 14, 2020 16:43

Add support for /api/status before Kibana completes startup

06dd043

Serve 503s when unavailable on /api/status

70ca428

Expose detailed information when SO migrations are waiting for other …

a5addbe

…nodes

Fix 503 handling

069213f

fx ci

567da20

joshdover force-pushed the not-ready-status branch from 49cde1f to 567da20 Compare December 14, 2020 23:45

legrego mentioned this pull request Jan 26, 2021

Interactive setup mode #89287

Closed

joshdover mentioned this pull request Feb 10, 2021

Have FTR correctly wait for Kibana to be green before running tests #89828

Closed

legrego added 7 commits April 2, 2021 15:40

Merge branch 'master' of github.com:elastic/kibana into update-not-re…

2cba1e3

…ady-status

fix merge

56e1f5f

Type & test fixes

c328b35

Change custom response handling to optionally bypass the error format

78cd007

fixing http service tests

9b1d174

Merge branch 'master' of github.com:elastic/kibana into update-not-re…

5742dd9

…ady-status

allow KbnClient to ignore errors

e7f355b

kibanamachine and others added 6 commits April 7, 2021 14:06

Merge branch 'master' into not-ready-status

11aa452

wait for the kibana http server to start, not the NotReady server

8791a24

Merge branch 'not-ready-status' of github.com:joshdover/kibana into u…

8a6c767

…pdate-not-ready-status

Avoid mutating internalSetup object

efbbc7c

Create dedicated method to support router registration after server h…

fcc1853

…as started listening

Throw error if fakeContext is used unexpectedly

2e0ac39

legrego added release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.14.0 v8.0.0 labels Apr 22, 2021

legrego marked this pull request as ready for review April 22, 2021 15:48

legrego requested review from a team as code owners April 22, 2021 15:48

jbudz reviewed Apr 22, 2021

View reviewed changes

jbudz approved these changes Apr 22, 2021

View reviewed changes

only register status route on NotReady server when anonymous access i…

4a4b772

…s allowed

Merge branch 'master' into not-ready-status

e9ce811

Merge branch 'master' into not-ready-status

efa4809

mshustov approved these changes Apr 26, 2021

View reviewed changes

legrego merged commit 48523e5 into elastic:master Apr 26, 2021

kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Apr 26, 2021

Add support for /api/status before Kibana completes startup (elastic#…

4e00afd

…79012) Co-authored-by: Larry Gregory <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

kibanamachine mentioned this pull request Apr 26, 2021

[7.x] Add support for /api/status before Kibana completes startup (#79012) #98329

Merged

tylersmalley mentioned this pull request Apr 28, 2021

Unable to load status page if red #98614

Closed

mshustov mentioned this pull request May 26, 2021

RequestHandler should be able to pass additional details in the response body #100648

Closed

mshustov mentioned this pull request Aug 31, 2021

/status endpoint returns 503 when any plugin is unavailable #110583

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for /api/status before Kibana completes startup #79012

Add support for /api/status before Kibana completes startup #79012

joshdover commented Sep 30, 2020 •

edited

Loading

joshdover commented Sep 30, 2020

mshustov Oct 1, 2020

joshdover Oct 2, 2020

legrego commented Jan 26, 2021

joshdover commented Jan 27, 2021

legrego commented Apr 7, 2021

legrego commented Apr 8, 2021

elasticmachine commented Apr 22, 2021

jbudz left a comment

legrego commented Apr 22, 2021

joshdover commented Apr 22, 2021

legrego commented Apr 23, 2021

legrego commented Apr 23, 2021

legrego commented Apr 26, 2021

mshustov commented Apr 26, 2021

kibanamachine commented Apr 26, 2021

API count

kibanamachine commented Apr 26, 2021

Add support for /api/status before Kibana completes startup #79012

Add support for /api/status before Kibana completes startup #79012

Conversation

joshdover commented Sep 30, 2020 • edited Loading

Summary

Checklist

For maintainers

joshdover commented Sep 30, 2020

mshustov Oct 1, 2020

Choose a reason for hiding this comment

joshdover Oct 2, 2020

Choose a reason for hiding this comment

legrego commented Jan 26, 2021

joshdover commented Jan 27, 2021

legrego commented Apr 7, 2021

legrego commented Apr 8, 2021

elasticmachine commented Apr 22, 2021

jbudz left a comment

Choose a reason for hiding this comment

legrego commented Apr 22, 2021

joshdover commented Apr 22, 2021

legrego commented Apr 23, 2021

legrego commented Apr 23, 2021

legrego commented Apr 26, 2021

mshustov commented Apr 26, 2021

kibanamachine commented Apr 26, 2021

💚 Build Succeeded

Metrics [docs]

API count

History

kibanamachine commented Apr 26, 2021

💚 Backport successful

joshdover commented Sep 30, 2020 •

edited

Loading