[Reporting/Screenshots] Handle page setup errors and capture the page, don't fail the job #58683

tsullivan · 2020-02-27T00:57:12Z

Release note: Kibana Reporting can now tolerate timeouts when trying to read visualization information on the Kibana page, and will capture a screenshot of the page even when such errors happen. If a timeout error happens on a Reporting job, the errors will become part of the job document and will be shown in the listing of Reporting jobs in Management > Kibana > Reporting.

Opening the Kibana URL for screenshot capture and reading the Kibana visualization element info from the page involves a lot of moving parts that could fail the entire reporting job if an error happens somewhere in the flow. This PR changes that by breaking up the flow to setup setups and a capture step. If there are errors in the setup step, they are logged and added to the Reporting job metadata, but the capture step is still allowed to continue.

New configuration settings for Reporting

xpack.reporting.capture.timeouts:
  openUrl: 30000
  waitForElements: 30000
  renderComplete: 30000

Each of these settings allows the user to control how much time to allow for each of the 3 main parts of setup$.

openUrl: the time allowed for Reporting to wait for the initial data of the Kibana page in the browser (time to resolve the .application selector)
waitForElements: the time allowed for Reporting to wait for the visualization panels to load (time to resolve the data-shared-items-container selector)
renderComplete: the time allowed for Reporting to wait for each visualization to signal that rendering is done

Note that xpack.reporting.queue.timeout still exists, and represents the overall time that the Reporting job is allowed to finish, and still has a default of 120 seconds. None of the new timeout settings should exceed the xpack.reporting.queue.timeout setting.

Other changes in this PR:

Retires the scanPage module. It had the job of waiting for visualization selectors while also checking the page for errors. The latter is a troubleshooting step that is no longer needed with this PR.
Removes some obsolete troubleshooting tasks, such as logging page text when the app selector could not be found
Wraps a lot of log messages in i18n.translate

If a Reporting job completed with "warnings" it gets a new treatment on the Management > Reporting job listing page:

However, there will still be a download available:

Why does the download show an empty Kibana page? Because when the page loaded initially, there was a toast message error saying "Saved Object not found". That toast message dismissed itself after a while, but before Reporting captured the page.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
~~[] Documentation was added for features that require explanation or tutorials~~
Unit or functional tests were updated or added to match the most common scenarios
~~[ ] This was checked for keyboard-only and screenreader accessibility~~
~~[ ] This renders correctly on smaller devices using a responsive layout. (You can test this in your browser~~
~~[ ] This was checked for cross-browser compatibility, including a check against IE11~~

For maintainers

This was checked for breaking API changes and was labeled appropriately

x-pack/legacy/plugins/reporting/export_types/common/lib/screenshots/check_for_toast.ts

tsullivan · 2020-02-27T23:17:42Z

x-pack/legacy/plugins/reporting/config.ts

+          .default(30000),
+        renderComplete: Joi.number()
+          .integer()
+          .default(30000),


Note: These get passed to Puppeteer waitForSelector calls. Puppeteer's default for these is 30 seconds internally.

I think we do call https://pptr.dev/#?product=Puppeteer&version=v2.1.1&show=api-pagesetdefaulttimeouttimeout and pass in the overall job timeout, so we might be overriding the default timeouts for these APIS.

Yup, we do it here: https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/reporting/server/browsers/chromium/driver_factory/index.ts#L120. Maybe we just default these to the global job timeout in this case?

++ If we're passing the timeout to each of the calls, then setting page.setDefaultTimeout doesn't make a lot of sense.

I don't think we should use the global job timeout, which is the "queue timeout" because once that timeout hits there is no buffer room to recover a completed screenshot for the job. Hitting the "queue timeout" should be avoided as much as possible.

We need to think about this... Should the page.setDefaultTimeout be the max of these 3 new timeouts?

At any rate, server/browsers/chromium/driver_factory/index.ts#L120 is no longer valid in this PR

@joelgriffith wdyt about the changes in 926b694 ?

specifically 926b694#diff-874db4343510c2d0fb661b14af4a97e9R115

Yup, this makes sense:

// All navigation methods default to 30 seconds, // which can cause the job to fail even if we bump timeouts in // All waitFor methods have their own timeout config // the config. Help alleviate errors like page.setDefaultTimeout(this.captureConfig.timeouts.openUrl);```

Individual methods can still override this, but I like setting it to the openUrl which is where I think the bulk of the operation is for PDF reports

tsullivan · 2020-02-27T23:21:25Z

Here is a test plan on how this PR will affect the different ways that jobs have been known to fail:

Test using invalid hostname for xpack.reporting.kibanaServer.hostname
Test invalid auth (stuck on login page)
Deleted Canvas worksheet
Deleted visualization in a Dashboard
Deleted saved object for visualization report
Search on a dashboard is taking a long time

tsullivan · 2020-02-27T23:28:32Z

x-pack/legacy/plugins/reporting/export_types/common/lib/screenshots/get_number_of_items.ts

+    // know how many items to expect since gridster incrementally adds panels
+    // we have to use this hint to wait for all of them
+    await browser.waitForSelector(
+      `${renderCompleteSelector},[${itemsCountAttribute}]`,


this block of code was moved over from the scanPage function

joelgriffith

This is great, nice logging additions + try/catch blocks. Think this will help us a ton 👍

… with errors shown

elasticmachine · 2020-03-04T19:35:43Z

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

tsullivan · 2020-03-05T16:41:37Z

@elasticmachine merge upstream

joelgriffith

Rest of the changes LGTM

kibanamachine · 2020-03-06T00:48:15Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: d514aee

History

💚 Build #31150 succeeded 01cb7e4
💔 Build #31007 failed 1723523
💔 Build #30862 failed fc9d418
💔 Build #30266 failed 4ce34f6
💔 Build #29896 failed 86a9738

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…, don't fail the job (elastic#58683) * [Reporting] Handle error if intercepted request could not be continued * [Reporting/Screenshots] Handle page setup errors and capture the page with errors shown * show warnings in UI * i18n todos * Cleanup an old troubleshooting task * set the default for all new timeout settings to 30 seconds * fix some tests * update error strings * Cleanup 2 * fix tests 2 * polish the job info map status items * More error message updating * Log the error that was caught * Oops fix ts * add documentation * fix i18n * fix mocha test * use the openUrl timeout as the default for navigation * fix comment Co-authored-by: Elastic Machine <[email protected]>

…x-closed-index * 'master' of github.com:elastic/kibana: (32 commits) [ML] Use Kibana's HttpHandler for HTTP requests (elastic#59320) [APM] Create settings page to manage Custom Links (elastic#57788) [Upgrade Assistant] Server-side batch reindexing (elastic#58598) completes navigation test (elastic#59141) [SIEM] Fixes dragging entries to the Timeline while data is loading may trigger a partial page reload (elastic#59476) [Reporting/Screenshots] Handle page setup errors and capture the page, don't fail the job (elastic#58683) [SIEM] [CASES] API with io-ts validation (elastic#59265) Use camelCase rather than snakeCase for plugin name (elastic#59461) [Maps] top term percentage field property (elastic#59386) Add custom action to registry and show actions list in siem (elastic#58395) [Search service] Add enhanced ES search strategy (elastic#59224) [Logs UI] Speed up stream rendering using memoization (elastic#59163) expand max-old-space-size for xpack jest tests (elastic#59455) Added possibility to embed connectors create and edit flyouts (elastic#58514) Revert "Temporarily disabling PR project mappings (elastic#59485)" (elastic#59491) Temporarily disabling PR project mappings (elastic#59485) [Endpoint] Fix alert list functional test error (elastic#59357) Rename status_page to statusPage (elastic#59186) Fix visual baseline job (elastic#59348) Extended AlertContextValue with metadata optional property (elastic#59391) ... # Conflicts: # x-pack/plugins/upgrade_assistant/common/types.ts # x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_actions.ts # x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.test.ts # x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.ts # x-pack/plugins/upgrade_assistant/server/routes/reindex_indices/reindex_indices.test.ts # x-pack/plugins/upgrade_assistant/server/routes/reindex_indices/reindex_indices.ts

…, don't fail the job (#58683) (#59519) * [Reporting] Handle error if intercepted request could not be continued * [Reporting/Screenshots] Handle page setup errors and capture the page with errors shown * show warnings in UI * i18n todos * Cleanup an old troubleshooting task * set the default for all new timeout settings to 30 seconds * fix some tests * update error strings * Cleanup 2 * fix tests 2 * polish the job info map status items * More error message updating * Log the error that was caught * Oops fix ts * add documentation * fix i18n * fix mocha test * use the openUrl timeout as the default for navigation * fix comment Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

tsullivan added v8.0.0 v7.7.0 labels Feb 27, 2020

tsullivan requested review from joelgriffith and a team February 27, 2020 00:57

tsullivan commented Feb 27, 2020

View reviewed changes

x-pack/legacy/plugins/reporting/export_types/common/lib/screenshots/check_for_toast.ts Show resolved Hide resolved

tsullivan force-pushed the reporting/screenshot-always-capture branch 6 times, most recently from 4026286 to 4b2a977 Compare February 27, 2020 22:00

tsullivan changed the title ~~[Reporting/Screenshots] Handle page setup errors and capture the page, even if errors~~ [Reporting/Screenshots] Handle page setup errors and capture the page, don't fail the job Feb 27, 2020

tsullivan force-pushed the reporting/screenshot-always-capture branch 2 times, most recently from 14bcff1 to c96095d Compare February 27, 2020 22:58

tsullivan commented Feb 27, 2020

View reviewed changes

joelgriffith approved these changes Feb 28, 2020

View reviewed changes

tsullivan force-pushed the reporting/screenshot-always-capture branch from 2ff57cd to b41acab Compare February 28, 2020 16:46

tsullivan added 10 commits February 28, 2020 14:23

[Reporting] Handle error if intercepted request could not be continued

d953b97

[Reporting/Screenshots] Handle page setup errors and capture the page…

052dc3e

… with errors shown

show warnings in UI

32ec120

i18n todos

45ce509

Cleanup an old troubleshooting task

7fe4001

set the default for all new timeout settings to 30 seconds

bdc2ea7

fix some tests

d6fc7bc

update error strings

ad8579d

Cleanup 2

dd5d05f

fix tests 2

7dc9e51

tsullivan added 2 commits February 28, 2020 16:04

Oops fix ts

12679f5

add documentation

86a9738

tsullivan marked this pull request as ready for review February 28, 2020 23:17

tsullivan added the release_note:enhancement label Feb 28, 2020

tsullivan added 4 commits March 2, 2020 21:44

Merge branch 'master' into reporting/screenshot-always-capture

606c1f1

fix i18n

4ce34f6

fix mocha test

8ff5ced

Merge branch 'master' into reporting/screenshot-always-capture

fc9d418

tsullivan added (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead Team:Reporting Services labels Mar 4, 2020

tsullivan added 4 commits March 4, 2020 12:52

Merge branch 'master' into reporting/screenshot-always-capture

d8a3d0c

Merge branch 'master' into reporting/screenshot-always-capture

99066b2

use the openUrl timeout as the default for navigation

926b694

fix comment

1723523

Merge branch 'master' into reporting/screenshot-always-capture

01cb7e4

joelgriffith approved these changes Mar 5, 2020

View reviewed changes

Merge branch 'master' into reporting/screenshot-always-capture

d514aee

tsullivan merged commit 893d8da into elastic:master Mar 6, 2020

tsullivan deleted the reporting/screenshot-always-capture branch March 6, 2020 05:28

tsullivan mentioned this pull request Mar 6, 2020

[7.x] [Reporting/Screenshots] Handle page setup errors and capture the page, don't fail the job (#58683) #59519

Merged

tsullivan mentioned this pull request Sep 23, 2020

[Reporting] report generation timeout when visualisation has error #50084

Closed

tsullivan mentioned this pull request Dec 30, 2020

[Reporting] Failure to take a screenshot of a dashboard or visualization should retry #87074

Closed

tsullivan mentioned this pull request Oct 7, 2021

[Reporting] Revisit handling timeouts for different phases of screenshot capture #113807

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Reporting/Screenshots] Handle page setup errors and capture the page, don't fail the job #58683

[Reporting/Screenshots] Handle page setup errors and capture the page, don't fail the job #58683

tsullivan commented Feb 27, 2020 •

edited

Loading

tsullivan Feb 27, 2020

joelgriffith Feb 28, 2020 •

edited

Loading

joelgriffith Feb 28, 2020

tsullivan Mar 4, 2020

tsullivan Mar 5, 2020 •

edited

Loading

joelgriffith Mar 5, 2020

joelgriffith Mar 5, 2020

tsullivan commented Feb 27, 2020

tsullivan Feb 27, 2020

joelgriffith left a comment

elasticmachine commented Mar 4, 2020

tsullivan commented Mar 5, 2020

joelgriffith left a comment

kibanamachine commented Mar 6, 2020

[Reporting/Screenshots] Handle page setup errors and capture the page, don't fail the job #58683

[Reporting/Screenshots] Handle page setup errors and capture the page, don't fail the job #58683

Conversation

tsullivan commented Feb 27, 2020 • edited Loading

Checklist

For maintainers

tsullivan Feb 27, 2020

Choose a reason for hiding this comment

joelgriffith Feb 28, 2020 • edited Loading

Choose a reason for hiding this comment

joelgriffith Feb 28, 2020

Choose a reason for hiding this comment

tsullivan Mar 4, 2020

Choose a reason for hiding this comment

tsullivan Mar 5, 2020 • edited Loading

Choose a reason for hiding this comment

joelgriffith Mar 5, 2020

Choose a reason for hiding this comment

joelgriffith Mar 5, 2020

Choose a reason for hiding this comment

tsullivan commented Feb 27, 2020

tsullivan Feb 27, 2020

Choose a reason for hiding this comment

joelgriffith left a comment

Choose a reason for hiding this comment

elasticmachine commented Mar 4, 2020

tsullivan commented Mar 5, 2020

joelgriffith left a comment

Choose a reason for hiding this comment

kibanamachine commented Mar 6, 2020

💚 Build Succeeded

History

tsullivan commented Feb 27, 2020 •

edited

Loading

joelgriffith Feb 28, 2020 •

edited

Loading

tsullivan Mar 5, 2020 •

edited

Loading