Failing test: X-Pack Reporting Functional Tests: Reports and Timeout Handling.x-pack/test/reporting_functional/reporting_and_timeout - Reporting Functional Tests with forced timeout adds a visual warning in the report output

[kibanamachine]

A test failed on a tracked branch

Error: expected 0.01084983087942698 to be below 0.01
    at Assertion.assert (node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.lessThan.Assertion.below (node_modules/@kbn/expect/expect.js:336:8)
    at Function.lessThan (node_modules/@kbn/expect/expect.js:531:15)
    at Context.<anonymous> (x-pack/test/reporting_functional/reporting_and_timeout/index.ts:53:92)
    at Object.apply (node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)

First failure: CI Build - main

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[tsullivan]

Looking to see if the "session" image that was generated at test time can be retrieved

[tsullivan]

The session image didn't match, because the "loading" indicators weren't on the screen. Therefore, in this case it is appropriate to increase the allowance for visual differences in the test.

Session image:

Baseline image:

[tsullivan]

PR fix: #135356

[kibanamachine]

New failure: CI Build - main

[spalger]

14 failures in the last week, I appreciate that you're close with #135356 @tsullivan but I'm going to skip until it's done. Please make sure to unskip in your PR after merging main.

[kibanamachine]

New failure: CI Build - main

[tsullivan]

I'm sending up a new PR to add more allowance to the visual difference, to unblock further flaky failures.

baseline image:

actual image that caused the failure:

the part needed for test validation is just the warning message at the top. If further flakiness happens then a good solution would be to crop the image area to the important part before the image comparison.

[kibanamachine]

New failure: CI Build - main

[mistic]

Skipped.

main: 0a03a52

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[jbudz]

/skip

[jbudz]

Skipped

main: 37c23b4

[tsullivan]

The latest failures look like a new type of error, which is probably a regression from the last round of maintenance:

2022-11-10 07:52:23 MST | └-: Reporting Functional Tests with forced timeout
-- | --
  | 2022-11-10 07:52:23 MST | └-> "before all" hook: beforeTestSuite.trigger for "adds a visual warning in the report output"
  | 2022-11-10 07:52:23 MST | └-> "before all" hook for "adds a visual warning in the report output"
  | 2022-11-10 07:52:23 MST | └-> adds a visual warning in the report output
  | 2022-11-10 07:53:04 MST | └-> "before each" hook: global before each for "adds a visual warning in the report output"
  | 2022-11-10 07:53:04 MST | └- ✖ fail: Reporting Functional Tests with forced timeout adds a visual warning in the report output
  | 2022-11-10 07:53:04 MST | │      Error: Unrecognised filter type - 196
  | 2022-11-10 07:53:04 MST | │       at module.exports.Filter._reverseFilterLine (node_modules/pngjs/lib/filter-parse.js:147:15)
  | 2022-11-10 07:53:04 MST | │       at module.exports.SyncReader.process (node_modules/pngjs/lib/sync-reader.js:34:17)
  | 2022-11-10 07:53:04 MST | │       at Object.exports.process (node_modules/pngjs/lib/filter-parse-sync.js:21:10)
  | 2022-11-10 07:53:04 MST | │       at module.exports (node_modules/pngjs/lib/parser-sync.js:93:35)
  | 2022-11-10 07:53:04 MST | │       at Object.read (node_modules/pngjs/lib/png-sync.js:10:10)
  | 2022-11-10 07:53:04 MST | │       at comparePngs (test/functional/services/lib/compare_pngs.ts:106:35)
  | 2022-11-10 07:53:04 MST | │       at PngService.comparePngs (test/functional/services/common/png.ts:32:12)
  | 2022-11-10 07:53:04 MST | │       at PngService.checkIfPngsMatch (test/functional/services/common/png.ts:69:18)
  | 2022-11-10 07:53:04 MST | │       at Context.<anonymous> (x-pack/test/reporting_functional/reporting_and_timeout/index.ts:82:9)
  | 2022-11-10 07:53:04 MST | │       at Object.apply (node_modules/@kbn/test/target_node/src/functional_test_runner/lib/mocha/wrap_function.js:78:16)
  | 2022-11-10 07:53:04 MST | │
  | 2022-11-10 07:53:04 MST | │

[petrklapka]

@tsullivan - Can you please set an LOE estimate? Thanks!