Performance Testing in CI

[cdamus]

Feature Description:

Another feature request, #10440, adds a node.js script that measures the start-up time of the Electron frontend application, with statistics computed over several runs. Together with the similar browser frontend start-up performance script, it could be helpful to run it in the continuous integration (CI) build to check for performance regressions in this critical measure of time to readiness of the Theia application.

At a minimum, a scheduled build of a new performance workflow in GitHub should run the start-up performance measurement scripts defined in the monorepo package (as renamed by PR #10442):

• yarn performance:startup:browser
• yarn performance:startup:electron

These scripts would be enhanced to detect that they are running in a GitHub build (probably via the GITHUB_ACTIONS environment variable) and write out their performance results to be recorded and compared against the recent performance history to check for regression. In the event of a regression, on worsening performance by some threshold to be determined (initial suggestion a 150% increase in the LCP time measurement captured by these scripts), the build may comment on the commit that was tested to indicate that it may have caused a performance regression. It is not suggested in this feature request that the build should fail.

A prototype is implemented here that uses a third-party action “github-action-benchmark” to record the history of performance measurements in the project’s GitHub Pages, with charts for visualization of the performance history and JSON data that may be used in future enhancements for more rigorous regression analysis.

An alternative to tracking the performance history on the gh-pages branch, which implies publishing the charts to the website on each run, is to use a separate branch dedicated to the purpose. This branch would keep the history of measurements and be used to check for regression (reported, if applicable, as a comment on the commit that was tested) but then would not be published to the Pages.