Add basic performance testing via Cypress #8586
Conversation
This adds a basic approach to performance testing within Cypress tests. You sprinkle `cy.startMeasuring` / `cy.stopMeasuring` pairs in the areas of interest. The measurements are added to the Cypress log shown in interactive runs and they are also assembled into a dashboard (https://matrix-org.github.io/matrix-react-sdk/cypress/bench/) for tracking changes across time. Fixes element-hq/element-web#21738
jryans
added
the
T-Task
| @@ -40,17 +40,16 @@ jobs: | |||
| test/end-to-end-tests/synapse/installations/consent/homeserver.log | |||
| retention-days: 14 | |||
|
|
|||
| - name: Download previous benchmark data | |||
There was a problem hiding this comment.
This is a dead block, as we aren't using the "separate file" version of this benchmark action.
|
The measure commands have upset Cypress in a few spots, so I'm leaving this as a draft for the moment. I should be able to sort those out in the next few days. |
| // Needed to make this file a module | ||
| export { }; |
There was a problem hiding this comment.
Ah that's the thing I was missing for the two files with dummy imports 🤦
|
Okay, seems to be working now! 😄 |
| # The dashboard is available at https://matrix-org.github.io/matrix-react-sdk/cypress/bench/ | ||
| benchmark-data-dir-path: cypress/bench | ||
| fail-on-alert: false | ||
| comment-on-alert: false |
There was a problem hiding this comment.
why not enable either of these?
There was a problem hiding this comment.
Could do, up to the team really. The current configuration is just copied from a previous usage of this benchmarking action. Perhaps it would be good to get a few data points in the dashboard and then define an appropriate threshold for alerting?
This adds a basic approach to performance testing within Cypress tests. You sprinkle
cy.startMeasuring/cy.stopMeasuringpairs in the areas of interest. The measurements are added to the Cypress log shown in interactiveruns and they are also assembled into a dashboard (https://matrix-org.github.io/matrix-react-sdk/cypress/bench/, only updated for develop runs, so nothing there yet) for tracking changes across time.
Fixes element-hq/element-web#21738
This change is marked as an internal change (Task), so will not be included in the changelog.