Update existing and add new SQL queries to the assessment step dashboard #269

larsgeorge-db · 2023-09-22T12:51:37Z

Fixes #268

larsgeorge-db · 2023-09-22T12:52:06Z

codecov · 2023-09-22T12:52:49Z

Codecov Report

Merging #269 (17347f0) into main (f7eab58) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #269      +/-   ##
==========================================
+ Coverage   83.37%   83.40%   +0.02%     
==========================================
  Files          29       29              
  Lines        1979     1982       +3     
  Branches      337      337              
==========================================
+ Hits         1650     1653       +3     
  Misses        261      261              
  Partials       68       68

Files Changed	Coverage Δ
src/databricks/labs/ucx/framework/dashboards.py	`76.88% <100.00%> (+0.35%)`	⬆️

larsgeorge-db · 2023-09-22T12:53:34Z

nfx

You're also adding more queries?

src/databricks/labs/ucx/assessment/queries/database_summary.sql

src/databricks/labs/ucx/assessment/queries/clusters.sql

src/databricks/labs/ucx/assessment/queries/all_tables.sql

README.md

src/databricks/labs/ucx/assessment/queries/database_summary.sql

src/databricks/labs/ucx/assessment/queries/all_tables.sql

nfx · 2023-09-22T23:37:49Z

tests/integration/framework/test_dashboards.py

@@ -15,7 +18,7 @@
 # logging.getLogger("databricks").setLevel("DEBUG")


-def test_creating_widgets(ws: WorkspaceClient):
+def test_creating_widgets(ws: WorkspaceClient, make_warehouse, make_schema):


You can use the preprovisioned warehouse through environment variable, no need to create new warehouse to test a dashboard.

nfx · 2023-09-22T23:40:45Z

src/databricks/labs/ucx/assessment/queries/clusters.sql

+  creator,
+  CASE WHEN success=1 THEN "Compatible" ELSE "Incompatible" END AS compatible,
+  failures
+FROM $inventory.clusters


Add NOT STARTSWITH(cluster_name, "job-") and also change the etl code

As a WHERE clause? And what ETL code?

I am confused where job- is used... looking at our E2 env:

yes, in the WHERE clause

nfx · 2023-09-22T23:43:53Z

@larsgeorge-db please also make the PR title very verbose - release scripts create a changelog line from the PR title.

nfx · 2023-09-24T18:25:30Z

src/databricks/labs/ucx/assessment/queries/all_tables.sql

+       CASE
+           WHEN STARTSWITH(location, "/dbfs/")
+                AND NOT STARTSWITH(location, "/dbfs/mnt") THEN "DBFS ROOT"
+           WHEN STARTSWITH(location, "/dbfs/")


First startswith is redundant

See comment below.

STARTSWITH(location, "/dbfs/") AND STARTSWITH(location, "/dbfs/mnt") is redundant by boolean logic.

src/databricks/labs/ucx/assessment/queries/all_tables.sql

src/databricks/labs/ucx/assessment/queries/database_summary.sql

src/databricks/labs/ucx/assessment/queries/jobs.sql

larsgeorge-db · 2023-09-25T08:22:11Z

Fixes #280

nfx

last few comments remaining

nfx · 2023-09-25T10:01:07Z

src/databricks/labs/ucx/assessment/queries/all_tables.sql

+       CASE
+           WHEN STARTSWITH(location, "/dbfs/")
+                AND NOT STARTSWITH(location, "/dbfs/mnt") THEN "DBFS ROOT"
+           WHEN STARTSWITH(location, "/dbfs/")


STARTSWITH(location, "/dbfs/") AND STARTSWITH(location, "/dbfs/mnt") is redundant by boolean logic.

nfx · 2023-09-25T10:01:53Z

src/databricks/labs/ucx/assessment/queries/clusters.sql

+  creator,
+  CASE WHEN success=1 THEN "Compatible" ELSE "Incompatible" END AS compatible,
+  failures
+FROM $inventory.clusters


yes, in the WHERE clause

src/databricks/labs/ucx/assessment/queries/database_summary.sql

larsgeorge-db · 2023-09-25T12:25:54Z

# Version changelog ## 0.2.0 * Added retrieving for all account-level groups with matching names to workspace-level groups in case no explicit configuration ([#277](#277)). * Added crawler for Azure Service principals used for direct storage access ([#305](#305)). * Added more SQL queries to the assessment step dashboard ([#269](#269)). * Added filtering out for job clusters in the clusters crawler ([#298](#298)). * Added recording errors from `crawl_tables` step in `$inventory.table_failures` table and display counter on the dashboard ([#300](#300)). * Added comprehensive introduction user manual ([#273](#273)). * Added interactive tutorial for local group migration readme ([#291](#291)). * Added tutorial links to the landing page of documentation ([#290](#290)). * Added (internal) support for account-level configuration and multi-cloud workspace list ([#264](#264)). * Improved order of tasks in the README notebook ([#286](#286)). * Improved installation script to run in a Windows Git Bash terminal ([#282](#282)). * Improved installation script by setting log level to uppercase by default ([#271](#271)). * Improved installation finish messages within installer script ([#267](#267)). * Improved automation for `MANAGED` table migration and continued building tables migration component ([#295](#295)). * Fixed debug notebook code with refactored package structure ([#250](#250)) ([#265](#265)). * Fixed replacement of custom configured database to replicate in the report for external locations ([#296](#296)). * Removed redundant `notebooks` top-level folder ([#263](#263)). * Split checking for test failures and linting errors into independent GitHub Actions checks ([#287](#287)). * Verify query metadata for assessment dashboards during unit tests ([#294](#294)).

larsgeorge-db added the pr/do-not-merge this pull request is not ready to merge label Sep 22, 2023

larsgeorge-db requested a review from nfx as a code owner September 22, 2023 12:51

larsgeorge-db marked this pull request as draft September 22, 2023 12:51

nfx reviewed Sep 22, 2023

View reviewed changes

FastLee requested a review from nfx September 22, 2023 18:21

FastLee added enhancement New feature or request and removed pr/do-not-merge this pull request is not ready to merge labels Sep 22, 2023

FastLee marked this pull request as ready for review September 22, 2023 18:25

FastLee approved these changes Sep 22, 2023

View reviewed changes

nfx requested changes Sep 22, 2023

View reviewed changes

larsgeorge-db changed the title ~~Enhance Assessment Dashboard (#268)~~ Update existing and add new SQL queries to the assessment step dashboard (#268) Sep 24, 2023

larsgeorge-db changed the title ~~Update existing and add new SQL queries to the assessment step dashboard (#268)~~ Update existing and add new SQL queries to the assessment step dashboard Sep 24, 2023

nfx reviewed Sep 24, 2023

View reviewed changes

larsgeorge-db mentioned this pull request Sep 25, 2023

Add support for COUNTER dashboard visualization #280

Closed

larsgeorge-db and others added 10 commits September 25, 2023 11:51

Enhance Assessment Dashboard (#268)

3edefad

Added Dashboard testing

5c97fe2

Added Dashboard testing and two more queries

713a41b

Removed integration test and added queries.

4fdbff3

Added Clusters and Jobs queries

437f083

Added Clusters and Jobs queries

a61cd6c

Important Commit!!!

7030b35

Fixed a couple of report issues.

b0e469c

Fix viz columns and format SQL code.

8564de8

Applied feedback and added counter viz support.

9488221

larsgeorge-db force-pushed the feature/issue_268_improve_assessment_dashboard branch from 9240bac to 9488221 Compare September 25, 2023 09:51

nfx reviewed Sep 25, 2023

View reviewed changes

Applied feedback, rearranged dashboard, fixed empty widget title

17347f0

nfx approved these changes Sep 25, 2023

View reviewed changes

nfx added this pull request to the merge queue Sep 25, 2023

Merged via the queue into main with commit dfa350a Sep 25, 2023

nfx linked an issue Sep 25, 2023 that may be closed by this pull request

Add support for COUNTER dashboard visualization #280

Closed

nfx deleted the feature/issue_268_improve_assessment_dashboard branch September 26, 2023 23:03

nfx mentioned this pull request Sep 29, 2023

Release v0.2.0 #328

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update existing and add new SQL queries to the assessment step dashboard #269

Update existing and add new SQL queries to the assessment step dashboard #269

larsgeorge-db commented Sep 22, 2023

larsgeorge-db commented Sep 22, 2023

codecov bot commented Sep 22, 2023 •

edited

Loading

larsgeorge-db commented Sep 22, 2023

nfx left a comment

nfx Sep 22, 2023

nfx Sep 22, 2023

larsgeorge-db Sep 25, 2023

larsgeorge-db Sep 25, 2023

nfx Sep 25, 2023

nfx commented Sep 22, 2023

nfx Sep 24, 2023

larsgeorge-db Sep 25, 2023

nfx Sep 25, 2023

larsgeorge-db commented Sep 25, 2023

nfx left a comment

nfx Sep 25, 2023

nfx Sep 25, 2023

larsgeorge-db commented Sep 25, 2023

Update existing and add new SQL queries to the assessment step dashboard #269

Update existing and add new SQL queries to the assessment step dashboard #269

Conversation

larsgeorge-db commented Sep 22, 2023

larsgeorge-db commented Sep 22, 2023

codecov bot commented Sep 22, 2023 • edited Loading

Codecov Report

larsgeorge-db commented Sep 22, 2023

nfx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nfx commented Sep 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

larsgeorge-db commented Sep 25, 2023

nfx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

larsgeorge-db commented Sep 25, 2023

codecov bot commented Sep 22, 2023 •

edited

Loading