Creating performance test to validate UCX at scale

[william-conti]

This pull request introduces new methods in the Makefile for running performance tests. It also adds a new test suite for performance in the tests/performance directory, including a conftest.py file for setting up fixtures and test_performance.py for running performance tests. The changes related to tests include creating and configuring various Databricks objects such as groups, pipelines, jobs, experiments, models, pools, warehouses, clusters, policies, queries, alerts, scopes, dashboards, repos, directories, and notebooks, and then verifying the successful creation and assignment of permissions for these objects. The performance tests measure the time taken to create and configure these objects and save the results in a SQL database for further analysis.

• Populate 1k groups in workspace / account
• Populate objects in scope of group migration
• Verify and Identify any failure creation - at this stage it should not fail, but it has to be captured
• Run assessment / group migration
• Compare before/after group migration

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.82%. Comparing base (e1b7a27) to head (257dc3b).
Report is 367 commits behind head on main.

❗ Current head 257dc3b differs from pull request most recent head 5e246da. Consider uploading reports for the commit 5e246da to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #607      +/-   ##
==========================================
+ Coverage   78.60%   82.82%   +4.21%     
==========================================
  Files          41       35       -6     
  Lines        4272     3493     -779     
  Branches      800      650     -150     
==========================================
- Hits         3358     2893     -465     
+ Misses        711      454     -257     
+ Partials      203      146      -57     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[CLAassistant]

All committers have signed the CLA.

[nfx]

Make timeout a parameter of run_workflow(), setting 1 day timeout here is a hidden disaster

[nfx]

If you're running this as a dedicated runner, it might make sense to route logs to a file as well.

[nfx]

Log errors into a file as well, ide would have trouble rendering it

[nfx]

closing this PR as the last time it had commit was 7 months ago.