Automation Plan: Kibana Visual Testing

[liza-mae]

This describes the initial test plan and infrastructure work for Kibana visual testing. Evaluating using a commercial tool for visual testing. Also researching cypress.io for visual testing to see if it helps with test flakiness.

We currently have close to 800 functional tests in Kibana repo for selenium and x-pack.

I went through build statistics to see how many runs per month over the last year.

Month	Build Runs
Jan 2018	1422
Feb 2018	983
Mar 2018	1517
Apr 2018	1530
May 2018	2408
Jun 2018	2001
Jul 2018	3180
Aug 2018	2999
Sep 2018	2609
Oct 2018	4008
Nov 2018*	1171
Dec 2018	3870
Jan 2019	5576
Feb 2019	6242

I went through existing test count on latest kibana repo master, excluding tests for: intake, plugin, interpreter, graph, home, getting started, context, dev tools, management, logstash pipeline, rollup, watcher, security, API, upgrade, spaces.

Existing test count should be filtered down further for what will be part of a visual test suite and we need to add any missing coverage. We can group different types of visualizations into a dashboard or different canvas elements into one workpad for consolidated checks to reduce number of tests.

Functional Area	Existing Tests	Visual Tests	New Tests	Total	Eng
*Dashboard	177
*Visualize	223				@liza-mae
*Canvas	3				@liza-mae
*Timelion	7
*Discover	75
*Maps	72				@nreese
Uptime	2				@justinkambic
*Reporting	15
Stack Monitoring	70				@chrisronline
Infrastructure	2				@weltenwort
Logs	1				@weltenwort
*Localization	-				@liza-mae
Machine Learning	-
APM	-
Elastic Charts	-				@markov00
Canvas	-				@clintandrewhall

(More than one engineer can be assigned to work on a functional area and can be directly or indirectly involved in the test dev)
(*feature means it is owned by Kibana team )

Estimating snapshot checks: (No of tests) x (3 browsers available) x (No of runs) x (growth factor)
Only Chrome and Firefox currently available in commercial tools, IE to be added later, not sure about Safari.

Test Plan

TBD

[liza-mae]

@spalger would you like to get this PR ready visual testing: #32953 or would you like us to help?

cc: @wayneseymour @dmlemeshko

[liza-mae]

This should be a separate test suite, need to figure out organization.

[liza-mae]

Development of features for visual testing has been available/open to add tests across each Kibana functional area for a few months now, at that time we had PR checking limited to only visual testing PRs, however now we have decided to enable visual testing against every PR as part of #42989

I have been running stability tests for Percy with our existing tests and so far Canvas and Maps seem pretty stable. There are still issues I am investigating with the other areas, more updates to follow.