Failing test: Chrome UI Functional Tests.test/functional/apps/visualize/group2/_gauge_chart·ts - visualize app gauge chart should format the metric correctly in percentage mode

[kibanamachine]

A test failed on a tracked branch

Error: timed out waiting for rendering count to be greater than or equal to [3]
    at onFailure (retry_for_truthy.ts:39:13)
    at retryForSuccess (retry_for_success.ts:84:7)
    at retryForTruthy (retry_for_truthy.ts:27:3)
    at RetryService.waitFor (retry.ts:87:5)
    at VisualizeChartPageObject.waitForRenderingCount (visualize_chart_page.ts:215:5)
    at VisualizeEditorPageObject.clickGo (visualize_editor_page.ts:74:5)
    at Context.<anonymous> (_gauge_chart.ts:59:7)
    at Object.apply (wrap_function.js:73:16)

First failure: CI Build - main

[elasticmachine]

Pinging @elastic/kibana-visualizations (Team:Visualizations)

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[mistic]

Skipped.

main: c8398aa

[nickofthyme]

I was able to reproduce this once running the tests locally with CI=true but never again even after several attempts with Marco's util script.

I put a log on both the onContainerLoading and onContainerRender function to see if they are called the same between the pass & fail runs.

kibana/src/plugins/visualizations/public/embeddable/visualize_embeddable.tsx

Lines 373 to 397 in 48e2b57

	onContainerLoading = () => {
	this.renderComplete.dispatchInProgress();
	this.updateOutput({
	...this.getOutput(),
	loading: true,
	rendered: false,
	error: undefined,
	});
	};
	onContainerData = () => {
	this.handleWarnings();
	this.updateOutput({
	...this.getOutput(),
	loading: false,
	});
	};
	onContainerRender = () => {
	this.renderComplete.dispatchComplete();
	this.updateOutput({
	...this.getOutput(),
	rendered: true,
	});
	};

Looking at the log from both tests they are exactly the same up until the call to waitForRenderingCount from inside the clickGo method.

While both tests make a call to onContainerLoading after the first try, the failed test never calls onContainerRender which would increment the counter. See below.

Passing test logs

│ debg TestSubjects.find(visualizationLoader)
│ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
│ debg -- currentRenderingCount=2
│ debg -- expectedCount=3
│ debg browser[INFO] http://localhost:5620/XXXXXXXXXXXX/bundles/plugin/visualizations/1.0.0/visualizations.chunk.7.js 1221:14 "onContainerLoading"
│ debg browser[INFO] http://localhost:5620/XXXXXXXXXXXX/bundles/plugin/visualizations/1.0.0/visualizations.chunk.7.js 1238:14 "onContainerRender"
│ debg TestSubjects.find(visualizationLoader)
│ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
│ debg -- currentRenderingCount=3
│ debg -- expectedCount=3
│ debg Find.allByCssSelector('[data-test-subj="visualizationLoader"] .chart svg text') with timeout=10000
└- ✓ pass  (10.1s)

Failing test log

│ debg TestSubjects.find(visualizationLoader)
│ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
│ debg -- currentRenderingCount=2
│ debg -- expectedCount=3
│ debg browser[INFO] http://localhost:5620/325677afb049/bundles/core/core.entry.js 15:78214 "onContainerLoading - renderComplete"
│ debg TestSubjects.find(visualizationLoader)
│ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
│ debg -- currentRenderingCount=2
│ debg -- expectedCount=3
│ debg TestSubjects.find(visualizationLoader)
│ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
│ debg -- currentRenderingCount=2
│ debg -- expectedCount=3
| [...removed duplicates (4 lines)]
│ info Taking window screenshot "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1720724252798929239/elastic/kibana-flaky-test-suite-runner/kibana/test/functional/screenshots/failure/visualize app gauge chart should format the metric correctly in percentage mode-c04e9b096275f91bc00a390653d79e70b48b3bb5e153567cc7fa59cc71390d31.png"
│ info Current URL is: http://localhost:5620/app/visualize#/create?type=gauge&indexPattern=logstash-*&_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:%272015-09-19T06:31:44.000Z%27,to:%272015-09-23T18:31:44.000Z%27))&_a=(filters:!(),linked:!f,query:(language:kuery,query:%27%27),uiState:(),vis:(aggs:!((enabled:!t,id:%271%27,params:(field:bytes),schema:metric,type:avg)),params:(addLegend:!t,addTooltip:!t,gauge:(alignment:automatic,backStyle:Full,colorSchema:%27Green%20to%20Red%27,colorsRange:!((from:0,to:50),(from:50,to:75),(from:75,to:10000)),extendRange:!t,gaugeColorMode:Labels,gaugeStyle:Full,gaugeType:Arc,invertColors:!f,labels:(color:black,show:!t),orientation:vertical,percentageFormatPattern:%270.0%25%27,percentageMode:!t,scale:(color:%27rgba(105,112,125,0.2)%27,labels:!f,show:!t),style:(bgColor:!t,bgFill:%27rgba(105,112,125,0.2)%27,bgMask:!f,bgWidth:0.9,fontSize:60,mask:!f,maskBars:50,subText:%27%27,width:0.9),type:meter),isDisplayWarning:!f,type:gauge),title:%27%27,type:gauge))
│ info Saving page source to: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1720724252798929239/elastic/kibana-flaky-test-suite-runner/kibana/test/functional/failure_debug/html/visualize app gauge chart should format the metric correctly in percentage mode-c04e9b096275f91bc00a390653d79e70b48b3bb5e153567cc7fa59cc71390d31.html
└- x fail: visualize app gauge chart should format the metric correctly in percentage mode
│      Error: timed out waiting for rendering count to be greater than or equal to [3]
│       at onFailure (retry_for_truthy.ts:39:13)
│       at retryForSuccess (retry_for_success.ts:84:7)
│       at retryForTruthy (retry_for_truthy.ts:27:3)
│       at RetryService.waitFor (retry.ts:87:5)
│       at VisualizeChartPageObject.waitForRenderingCount (visualize_chart_page.ts:215:5)
│       at VisualizeEditorPageObject.clickGo (visualize_editor_page.ts:90:5)
│       at Context.<anonymous> (_gauge_chart.ts:60:7)
│       at Object.apply (wrap_function.js:73:16)

See full test log diff

--- failed.log
+++ passed.log
@@ -1,4 +1,4 @@
-Failed test log
+Passed test log

  └-: visualize app
    └-: gauge chart
@@ -101,173 +101,11 @@ Failed test log
        │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
        │ debg -- currentRenderingCount=2
        │ debg -- expectedCount=3
-       │ debg browser[INFO] http://localhost:5620/325677afb049/bundles/core/core.entry.js 15:78214 "onContainerLoading - renderComplete"
+       │ debg browser[INFO] http://localhost:5620/XXXXXXXXXXXX/bundles/plugin/visualizations/1.0.0/visualizations.chunk.7.js 1221:14 "onContainerLoading"
+       │ debg browser[INFO] http://localhost:5620/XXXXXXXXXXXX/bundles/plugin/visualizations/1.0.0/visualizations.chunk.7.js 1238:14 "onContainerRender"
        │ debg TestSubjects.find(visualizationLoader)
        │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
-       │ debg -- expectedCount=3
-       │ debg TestSubjects.find(visualizationLoader)
-       │ debg Find.findByCssSelector('[data-test-subj="visualizationLoader"]') with timeout=10000
-       │ debg -- currentRenderingCount=2
+       │ debg -- currentRenderingCount=3
        │ debg -- expectedCount=3
-       │ info Taking window screenshot "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1720724252798929239/elastic/kibana-flaky-test-suite-runner/kibana/test/functional/screenshots/failure/visualize app gauge chart should format the metric correctly in percentage mode-c04e9b096275f91bc00a390653d79e70b48b3bb5e153567cc7fa59cc71390d31.png"
-       │ info Current URL is: http://localhost:5620/app/visualize#/create?type=gauge&indexPattern=logstash-*&_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:%272015-09-19T06:31:44.000Z%27,to:%272015-09-23T18:31:44.000Z%27))&_a=(filters:!(),linked:!f,query:(language:kuery,query:%27%27),uiState:(),vis:(aggs:!((enabled:!t,id:%271%27,params:(field:bytes),schema:metric,type:avg)),params:(addLegend:!t,addTooltip:!t,gauge:(alignment:automatic,backStyle:Full,colorSchema:%27Green%20to%20Red%27,colorsRange:!((from:0,to:50),(from:50,to:75),(from:75,to:10000)),extendRange:!t,gaugeColorMode:Labels,gaugeStyle:Full,gaugeType:Arc,invertColors:!f,labels:(color:black,show:!t),orientation:vertical,percentageFormatPattern:%270.0%25%27,percentageMode:!t,scale:(color:%27rgba(105,112,125,0.2)%27,labels:!f,show:!t),style:(bgColor:!t,bgFill:%27rgba(105,112,125,0.2)%27,bgMask:!f,bgWidth:0.9,fontSize:60,mask:!f,maskBars:50,subText:%27%27,width:0.9),type:meter),isDisplayWarning:!f,type:gauge),title:%27%27,type:gauge))
-       │ info Saving page source to: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1720724252798929239/elastic/kibana-flaky-test-suite-runner/kibana/test/functional/failure_debug/html/visualize app gauge chart should format the metric correctly in percentage mode-c04e9b096275f91bc00a390653d79e70b48b3bb5e153567cc7fa59cc71390d31.html
-       └- x fail: visualize app gauge chart should format the metric correctly in percentage mode
-       │      Error: timed out waiting for rendering count to be greater than or equal to [3]
-       │       at onFailure (retry_for_truthy.ts:39:13)
-       │       at retryForSuccess (retry_for_success.ts:84:7)
-       │       at retryForTruthy (retry_for_truthy.ts:27:3)
-       │       at RetryService.waitFor (retry.ts:87:5)
-       │       at VisualizeChartPageObject.waitForRenderingCount (visualize_chart_page.ts:215:5)
-       │       at VisualizeEditorPageObject.clickGo (visualize_editor_page.ts:90:5)
-       │       at Context.<anonymous> (_gauge_chart.ts:60:7)
-       │       at Object.apply (wrap_function.js:73:16)
+       │ debg Find.allByCssSelector('[data-test-subj="visualizationLoader"] .chart svg text') with timeout=10000
+       └- ✓ pass  (10.1s)

---

In the end I think there is flakiness with the ExpressionLoader handlers.

A trivial solution would be to trigger another click on go within the waitForRenderingCount retries. But this is dirty and at that point the render assertion looses it's utility.

I think the real solution here is to avoid the render count and just assert the format is applied to the chart.

Thoughts?

[dej611]

I think the real solution here is to avoid the render count and just assert the format is applied to the chart.

While we could remove the render-count logic from the test, we then have 2 problems:

• a potential issue with reporting - rendering-complete update goes side by side with the rendering-count, so this issue could be a hint of a bug that can affect it
• find an alternative to check a re-render (i.e. charts debug log? )

While for the second point I think it would be fine, but the former is what makes me a feel that it won't solve it completely.

[mbondyra]

Removing the blocker label as it's a legacy code so it's justifiable it's not blocking.