[discuss] Build on top of synthetics or use playwright directly?

[spalger]

@elastic/synthetics is a package which wraps microsoft/playwright by allowing users to define journey()s which are made up of step()s, and run against an instance of headless Chrome. Beyond defining the test structure, the fundamental API is provided by playwright via the page, browser, client, and context parameters passed into journey() callbacks.

We need to weigh the pros and cons of consuming @elastic/synthetics vs using playwright directly. I've tried to get a simple POC together of writing journeys in the Kibana repo and it's been a challenge to adapt the current service system to something that works with synthetics. In the meantime I'm planning to start writing some simple synthetics but there was a question about wether it's worth it to use a wrapper around playwright or if we should just consume it directly... I think there's value in using synthetics but much of it is kind of "potential" value.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[spalger]

Journeys/Synthetics wrapper around playwright

Pros:

• Good docs, benefits from great docs of playwright
• Runner is distinct from "test" runners, already exists, and the nouns are pretty great IMO (journey/step vs describe/it)
• APM data is automatically correlated once [Proposal] Crosslinking Synthetics with APM synthetics#265 is complete
• Runner state is automatically indexed once [Heartbeat] One shot mode  beats#25972 is complete so we can build a UI that consumes the runner data and exposes run data to users online.
• Data follows the observability model, giving us
  • more options for automatically consuming this data in the observability applications, maybe someday from a first-class Kibana app for CI builds or something
  • automatic index lifecycle management
  • system index
  • etc.
• Synthetics is still an Elastic project, so we have a lot more influence over the way that project evolves compared to many other solutions out there (particularly if we're willing to contribute the code)

Cons:

• We need to position all changes in a way that makes sense for synthetics/observability/uptime (retries, setup logic, customized journey context)
• Our wrapper logic needs to download and extract heartbeat, which will download and extract synthetics, which will download and extract chromium, which will run our tests in a child process once we've started Kibana and ES... smells a little like it's overcomplicated somewhere
• The setup logic that starts ES and Kibana will be disconnected from the services running in the synthetics process and need to send info like Kibana/ES URL to the process through ENV vars or something
• The correct heartbeat version is another dependency version we would need to track and maintain. I'm not sure what the version relationship between HB and ES is right now, but I'm pretty sure that HB works with a large range of ES versions
• If we end up reusing FTR services in synthetics then we will have to expose a getService function (and similar) to journey's

[spalger]

Consume playwright directly

Pros:

• Well documented API
• Starting Kibana/ES, services, runner, can all coexist in the same process
• No need to download heartbeat or track versions there
• There is a test runner package available from playwright, but we would likely build our own with the semantics we want for functional/performance testing

Cons:

• If we wanted to provide a report online somewhere about what happened we would need to implement a good deal of the logic already implemented by synthetics and build the UI ourselves
• [Proposal] Crosslinking Synthetics with APM synthetics#265 would need to be implemented correctly
• Very low chance that observability will be able to help us build a UI on top of data we collect outside of HB and index in our own format
• Need to manage our own ES data lifecycle, system indexes, etc.

[dmlemeshko]

I might be wrong, but If we move test runner into Synthetics, it becomes quite Kibana-specific and more complex project. Plus FTR roadmap should be considered as a part of Synthetics project. Will it ok with Observability team?

By comparing listed cons I think consuming playright directly is less painful, but still concerning. Some thoughts:

• Maybe there is a way to use elastic/synthetics as an extension on top of migrated to playright FTR?
• If not, can we consider it as an option and think of how much effort it may take?

[spalger]

If we move test runner into Synthetics

To be clear, that's not what we're looking to do. We're looking to add new types of tests and probably deprecate usage of the FTR or try to reduce it dramatically in the Kibana repo. We'll likely have both solutions for a long while and the FTR roadmap won't become the Observability team's responsibility. We'll still have our own tooling around synthetics to accomplish what we need.

[tylersmalley]

Closing this for now - we're moving forward with using Playwright with the plan to move to Synthetics. This will be a good test case for anything to follow.