Code

Add dashboard crawlers#3397

Code

Merged

gueniai merged 182 commits intomaindatabrickslabs/ucx:mainfrom feat/add-dashboard-crawlersdatabrickslabs/ucx:feat/add-dashboard-crawlersCopy head branch name to clipboardDec 13, 2024

+1,933-654

Commits

Commits on Dec 13, 2024

• Setup framework for crawling dashboards

  JCZuurmond

  committed
  4d98773

  Copy full SHA for 4d98773

• Move lint related dashboard integration test to source code directory

  JCZuurmond

  committed
  a930bf7

  Copy full SHA for a930bf7

• Start with empty crawl

  JCZuurmond

  committed
  482e968

  Copy full SHA for 482e968

• Add integration test for crawling dashboard

  JCZuurmond

  committed
  ac7fb08

  Copy full SHA for ac7fb08

• Crawl Redash dashboards

  JCZuurmond

  committed
  19638fa

  Copy full SHA for 19638fa

• Test include dashboard parameter

  JCZuurmond

  committed
  3db33b5

  Copy full SHA for 3db33b5

• Add include dashboard parameter

  JCZuurmond

  committed
  401679f

  Copy full SHA for 401679f

• Rename crawler variable

  JCZuurmond

  committed
  d4be79b

  Copy full SHA for d4be79b

• Test crawl LakeviewDashboards

  JCZuurmond

  committed
  2788888

  Copy full SHA for 2788888

• Crawl lakeview dashboards

  JCZuurmond

  committed
  ba0ce24

  Copy full SHA for ba0ce24

• Test include Lakeview dashboard ids

  JCZuurmond

  committed
  364bb1b

  Copy full SHA for 364bb1b

• Implement include Lakeview dashboards

  JCZuurmond

  committed
  0936e02

  Copy full SHA for 0936e02

• Add Redash or Lakeview to log messages to differentiate

  JCZuurmond

  committed
  9f5853e

  Copy full SHA for 9f5853e

• Differentiate between Redash and Lakeview dashboards

  JCZuurmond

  committed
  304f911

  Copy full SHA for 304f911

• Format

  JCZuurmond

  committed
  395e099

  Copy full SHA for 395e099

• List dashboard

  JCZuurmond

  committed
  44aa79b

  Copy full SHA for 44aa79b

• Assert mock dashboards to have an ID

  JCZuurmond

  committed
  dfa0f7a

  Copy full SHA for dfa0f7a

• Add dashboard crawlers to RuntimeContext

  JCZuurmond

  committed
  8c297e0

  Copy full SHA for 8c297e0

• Update comment on include_dashboards_ids

  Show description for 9de0611
  JCZuurmond

  committed
  9de0611

  Copy full SHA for 9de0611

• Move Redash dashboard crawler to global context

  JCZuurmond

  committed
  e8fbb78

  Copy full SHA for e8fbb78

• Add dashboard crawlers to assessment workflow

  JCZuurmond

  committed
  0d23447

  Copy full SHA for 0d23447

• Add Redash and Lakeview dashboard tables to install

  JCZuurmond

  committed
  b31b83f

  Copy full SHA for b31b83f

• Update Redash migration to use RedashDashboardCrawler

  JCZuurmond

  committed
  0c67e3a

  Copy full SHA for 0c67e3a

• Make get queries from dashboard protected

  JCZuurmond

  committed
  aa48e6c

  Copy full SHA for aa48e6c

• Force Redash dashboard crawler key word arguments

  JCZuurmond

  committed
  430806d

  Copy full SHA for 430806d

• Lower case B in dashboard

  JCZuurmond

  committed
  a31d643

  Copy full SHA for a31d643

• Handle non specified dashboard id

  JCZuurmond

  committed
  92a26c3

  Copy full SHA for 92a26c3

• Pass Redash crawler in global context

  JCZuurmond

  committed
  3cf04e4

  Copy full SHA for 3cf04e4

• Handle no dashboard id in revert DBSQL dashboards cli command

  JCZuurmond

  committed
  67c47a6

  Copy full SHA for 67c47a6

• Fix Redash integration test

  JCZuurmond

  committed
  6a59a08

  Copy full SHA for 6a59a08

• Add parent field to RedashDashboard

  JCZuurmond

  committed
  60fc9f9

  Copy full SHA for 60fc9f9

• Extend RedashDashboard from sdk Dashboard

  JCZuurmond

  committed
  9d86e7d

  Copy full SHA for 9d86e7d

• Skip dashboard without id

  JCZuurmond

  committed
  7308f7e

  Copy full SHA for 7308f7e

• Get dashboard in fixture to update widget

  JCZuurmond

  committed
  645043f

  Copy full SHA for 645043f

• Skip dashboards that miss ids

  JCZuurmond

  committed
  3da399d

  Copy full SHA for 3da399d

• Fix unit tests

  JCZuurmond

  committed
  c56c6bf

  Copy full SHA for c56c6bf

• Fix integration test

  JCZuurmond

  committed
  e763ee4

  Copy full SHA for e763ee4

• Add TODO for linting lakeview dashboards

  JCZuurmond

  committed
  4487039

  Copy full SHA for 4487039

• Fix integration test including dashboard

  JCZuurmond

  committed
  8abcd50

  Copy full SHA for 8abcd50

• Refactor while condition

  JCZuurmond

  committed
  bd5b2d2

  Copy full SHA for bd5b2d2

• Ignore too many public methods

  JCZuurmond

  committed
  1829f78

  Copy full SHA for 1829f78

• Fix protected access in unit tests

  JCZuurmond

  committed
  02872fb

  Copy full SHA for 02872fb

• Add assert to mock

  JCZuurmond

  committed
  3b6f0e4

  Copy full SHA for 3b6f0e4

• Format

  JCZuurmond

  committed
  03e4f61

  Copy full SHA for 03e4f61

• Shorten variable name

  JCZuurmond

  committed
  dbbbfd4

  Copy full SHA for dbbbfd4

• Update cli tests

  JCZuurmond

  committed
  022a7fd

  Copy full SHA for 022a7fd

• Add attributes to LakeviewDashboard

  JCZuurmond

  committed
  fef9d5d

  Copy full SHA for fef9d5d

• Lint Lakeview dashboards in QueryLinter

  JCZuurmond

  committed
  00066c6

  Copy full SHA for 00066c6

• Expect tags on LakeviewDashboard

  JCZuurmond

  committed
  2731a67

  Copy full SHA for 2731a67

• Remove tags from LakeviewDashboard

  JCZuurmond

  committed
  584062e

  Copy full SHA for 584062e

• Test attributes on LakeviewDashboard

  JCZuurmond

  committed
  2b1f1ba

  Copy full SHA for 2b1f1ba

• Create LakeviewDashboard from SdkLakeviewDashboard

  JCZuurmond

  committed
  85e21b9

  Copy full SHA for 85e21b9

• Test redash Dashboard crawler to persist dashboards

  JCZuurmond

  committed
  0c5feb2

  Copy full SHA for 0c5feb2

• Test handling DatabricksError on list

  JCZuurmond

  committed
  111c1bf

  Copy full SHA for 111c1bf

• Test handling DatabricksError on iterate

  JCZuurmond

  committed
  de53c11

  Copy full SHA for de53c11

• Test debug listing upper limit

  JCZuurmond

  committed
  5be1a45

  Copy full SHA for 5be1a45

• Fix condition for debug listing upper limit

  JCZuurmond

  committed
  263d6f7

  Copy full SHA for 263d6f7

• Test warning logs

  JCZuurmond

  committed
  434fbed

  Copy full SHA for 434fbed

• Test getting dashboard with dashboard ids

  JCZuurmond

  committed
  89b7063

  Copy full SHA for 89b7063

• Test for dashboard id not found

  JCZuurmond

  committed
  b48d646

  Copy full SHA for b48d646

• Test for skipping dashboard without id

  JCZuurmond

  committed
  f05bf24

  Copy full SHA for f05bf24

• Add comment about API listing for Lakeview dashboards

  JCZuurmond

  committed
  e431c22

  Copy full SHA for e431c22

• Duplicate dashboard crawler tests for Lakeview

  JCZuurmond

  committed
  ab36522

  Copy full SHA for ab36522

• Skip Lakeview dashboards without id

  JCZuurmond

  committed
  b98af1b

  Copy full SHA for b98af1b

• Fix crawling lakeview dashboards

  JCZuurmond

  committed
  5afe53f

  Copy full SHA for 5afe53f

• Force include dashboard ids to be keyword argument

  JCZuurmond

  committed
  d98c808

  Copy full SHA for d98c808

• Fix typo

  JCZuurmond

  committed
  3442918

  Copy full SHA for 3442918

• Force Lakeview dashboard fixture to have keyword arguments

  JCZuurmond

  committed
  5c99bfd

  Copy full SHA for 5c99bfd

• Add query parameter to make_lakeview_dashboard fixture

  JCZuurmond

  committed
  bc7bf54

  Copy full SHA for bc7bf54

• Fix return type

  JCZuurmond

  committed
  3ca8656

  Copy full SHA for 3ca8656

• Rename dashboard classes

  JCZuurmond

  committed
  27efdc4

  Copy full SHA for 27efdc4

• Rename Lakeview fixture dashboard query

  JCZuurmond

  committed
  695338d

  Copy full SHA for 695338d

• Add Lakeview dashboard fixture to mock runtime

  JCZuurmond

  committed
  a59cd3a

  Copy full SHA for a59cd3a

• Fix reference to dataset

  JCZuurmond

  committed
  f1a65df

  Copy full SHA for f1a65df

• Fix getting dashboard ids

  JCZuurmond

  committed
  98a5ccb

  Copy full SHA for 98a5ccb

• Add get query to RedashDashboardCrawler

  JCZuurmond

  committed
  c61c9b0

  Copy full SHA for c61c9b0

• Test getting Redash query

  JCZuurmond

  committed
  8eae204

  Copy full SHA for 8eae204

• Test getting non-existing query

  JCZuurmond

  committed
  59c48f9

  Copy full SHA for 59c48f9

• Add list queries to RedashDashboardCrawler

  JCZuurmond

  committed
  04282e9

  Copy full SHA for 04282e9

• Test listing Redash queries

  JCZuurmond

  committed
  a81062e

  Copy full SHA for a81062e

• Implement listing redash queries

  JCZuurmond

  committed
  f04303b

  Copy full SHA for f04303b

• Test handling permission denied error when listing Redash queries

  JCZuurmond

  committed
  dc0120b

  Copy full SHA for dc0120b

• Handle Databricks error when listing Redash queries

  JCZuurmond

  committed
  95b27aa

  Copy full SHA for 95b27aa

• Assert warning when listing Redash queries

  JCZuurmond

  committed
  88c0f7a

  Copy full SHA for 88c0f7a

• Add list queries method to LakeviewDashboardCrawler

  JCZuurmond

  committed
  d6158b6

  Copy full SHA for d6158b6

• Test listing Lakeview queries

  JCZuurmond

  committed
  b55031a

  Copy full SHA for b55031a

• Add listing Lakeview queries

  JCZuurmond

  committed
  ce3ad35

  Copy full SHA for ce3ad35

• Fix test for handling permission denied in Lakeview queries list

  JCZuurmond

  committed
  5c32c6f

  Copy full SHA for 5c32c6f

• Remove redundant dashboard id

  JCZuurmond

  committed
  74a06ca

  Copy full SHA for 74a06ca

• Test handling corrupted serialized dashboard

  JCZuurmond

  committed
  7a5d02d

  Copy full SHA for 7a5d02d

• Explain difference between Lakeview and Redash list queries

  JCZuurmond

  committed
  f4f62cb

  Copy full SHA for f4f62cb

• Add get_query method to LakeviewDashboardCrawler

  JCZuurmond

  committed
  f8c74a9

  Copy full SHA for f8c74a9

• Test get Lakeview query

  JCZuurmond

  committed
  b45b8af

  Copy full SHA for b45b8af

• Implement get Lakeview query

  JCZuurmond

  committed
  e8b43d0

  Copy full SHA for e8b43d0

• Fix get query type hint

  JCZuurmond

  committed
  a886a2e

  Copy full SHA for a886a2e

• Extract converting sdk to lsql Lakeview dashboard

  JCZuurmond

  committed
  8e9d7a6

  Copy full SHA for 8e9d7a6

• Let Redash query methods return strings

  JCZuurmond

  committed
  e914814

  Copy full SHA for e914814

• Refactor get_query to get_queries

  JCZuurmond

  committed
  111cbf3

  Copy full SHA for 111cbf3

• Merge get_queries with list_queries

  JCZuurmond

  committed
  52e615c

  Copy full SHA for 52e615c

• Handle query is None

  JCZuurmond

  committed
  4711fa3

  Copy full SHA for 4711fa3

• Format

  JCZuurmond

  committed
  364d5a6

  Copy full SHA for 364d5a6

• Fix type hint

  JCZuurmond

  committed
  68fc0ab

  Copy full SHA for 68fc0ab

• Remove redudant json.JsonDecodeError

  JCZuurmond

  committed
  da17cd7

  Copy full SHA for da17cd7

• Fix iterator import

  JCZuurmond

  committed
  4376dc3

  Copy full SHA for 4376dc3

• Assert len queries

  JCZuurmond

  committed
  280da01

  Copy full SHA for 280da01

• Add make Lakeview dashboard fixture to MockInstallationContext

  JCZuurmond

  committed
  73299c2

  Copy full SHA for 73299c2

• Rename variables

  JCZuurmond

  committed
  b3ebdc6

  Copy full SHA for b3ebdc6

• Add missing caplog

  JCZuurmond

  committed
  76fffcf

  Copy full SHA for 76fffcf

• Test caplog.messages instead of caplog.text

  JCZuurmond

  committed
  4293ed4

  Copy full SHA for 4293ed4

• Test invalid serialized json

  JCZuurmond

  committed
  166c200

  Copy full SHA for 166c200

• Ignore too-many-locals

  JCZuurmond

  committed
  1176c40

  Copy full SHA for 1176c40

• Add Query dataclass

  JCZuurmond

  committed
  6c84fb3

  Copy full SHA for 6c84fb3

• Add parent to Query

  JCZuurmond

  committed
  3a3d481

  Copy full SHA for 3a3d481

• Fix type hints

  JCZuurmond

  committed
  d70fbd7

  Copy full SHA for d70fbd7

• Format

  JCZuurmond

  committed
  b062f84

  Copy full SHA for b062f84

• Add debug limit to Redash queries listing

  JCZuurmond

  committed
  8898d3e

  Copy full SHA for 8898d3e

• Let QueryLinter use the dashboard crawlers

  JCZuurmond

  committed
  18aa565

  Copy full SHA for 18aa565

• Let Redash dashboard migration use legacy queries from crawler

  JCZuurmond

  committed
  d8f72d4

  Copy full SHA for d8f72d4

• Fix integration test

  JCZuurmond

  committed
  54bddd2

  Copy full SHA for 54bddd2

• Update has_calls to assert_has_calls

  JCZuurmond

  committed
  e4374a8

  Copy full SHA for e4374a8

• Merge Dashboard dataclasses

  JCZuurmond

  committed
  7619b1a

  Copy full SHA for 7619b1a

• Remove DashboardCrawlerType

  JCZuurmond

  committed
  d9621c1

  Copy full SHA for d9621c1

• Fix dashboard tests

  JCZuurmond

  committed
  24f0908

  Copy full SHA for 24f0908

• Add include_query_ids to RedashDashboardCrawler

  JCZuurmond

  committed
  92f0d7e

  Copy full SHA for 92f0d7e

• Add include_query_ids to LakeviewDashboardCrawler

  JCZuurmond

  committed
  9b4a6ff

  Copy full SHA for 9b4a6ff

• Pass include query ids from config

  JCZuurmond

  committed
  c3380f1

  Copy full SHA for c3380f1

• Expose make query and include created query ids

  JCZuurmond

  committed
  cd70491

  Copy full SHA for cd70491

• Fix query linter integration test

  JCZuurmond

  committed
  2158c59

  Copy full SHA for 2158c59

• Fix variable should be same type

  JCZuurmond

  committed
  d1ecfea

  Copy full SHA for d1ecfea

• Add missing return

  JCZuurmond

  committed
  b621ef2

  Copy full SHA for b621ef2

• Fix wrong name in unit test

  JCZuurmond

  committed
  fd95523

  Copy full SHA for fd95523

• Handle query id being None

  JCZuurmond

  committed
  c186e9e

  Copy full SHA for c186e9e

• Set back disable too-many-public-methods

  JCZuurmond

  committed
  6aad514

  Copy full SHA for 6aad514

• Avoid duplicate queries in query id

  JCZuurmond

  committed
  5204b12

  Copy full SHA for 5204b12

• Split Redash and Lakeview DFSA ownership test

  JCZuurmond

  committed
  93672a1

  Copy full SHA for 93672a1

• Mark Lakeview ownership to fail

  JCZuurmond

  committed
  7f4a6ed

  Copy full SHA for 7f4a6ed

• Scope queries in MockRuntimeContext

  JCZuurmond

  committed
  072f2e6

  Copy full SHA for 072f2e6

• Do not refresh dashboard

  JCZuurmond

  committed
  3bb6091

  Copy full SHA for 3bb6091

• Clarify migrate dashboard integration test

  JCZuurmond

  committed
  a7e3b1f

  Copy full SHA for a7e3b1f

• Shorten for-loop

  JCZuurmond

  committed
  0e52991

  Copy full SHA for 0e52991

• Move dashboards out of cached property

  JCZuurmond

  committed
  7b21f52

  Copy full SHA for 7b21f52

• Test dashboard migration tags to be set

  JCZuurmond

  committed
  a34a244

  Copy full SHA for a34a244

• Fix filtering dashboards

  JCZuurmond

  committed
  abe1a93

  Copy full SHA for abe1a93

• Refresh dashboards when reverting to get latest tags

  JCZuurmond

  committed
  2e6eefe

  Copy full SHA for 2e6eefe

• Wait for dashboard migration tag to be present in integration test

  JCZuurmond

  committed
  9c9796a

  Copy full SHA for 9c9796a

• Remove redundant for-loop

  JCZuurmond

  committed
  b76dffa

  Copy full SHA for b76dffa

• Add dashboard tables to table persistence docs

  JCZuurmond

  committed
  bcb5445

  Copy full SHA for bcb5445

• Add tags to Query

  JCZuurmond

  committed
  295559d

  Copy full SHA for 295559d

• Add catalog and schema to query

  JCZuurmond

  committed
  5667196

  Copy full SHA for 5667196

• Rewrite Redash to use Query instead of LegacyQuery

  JCZuurmond

  committed
  a462860

  Copy full SHA for a462860

• Shorten filtering dashboards

  JCZuurmond

  committed
  73c63e2

  Copy full SHA for 73c63e2

• Remove redundant return

  JCZuurmond

  committed
  af0e3f1

  Copy full SHA for af0e3f1

• Add a tile for the dashboards

  JCZuurmond

  committed
  12e3c31

  Copy full SHA for 12e3c31

• Force change on test running real assessment job

  JCZuurmond

  committed
  5516693

  Copy full SHA for 5516693

• Make dashboard name and parent optional

  JCZuurmond

  committed
  a1521ea

  Copy full SHA for a1521ea

• Assert dashboard id

  JCZuurmond

  committed
  b45dcc8

  Copy full SHA for b45dcc8

• Let dashboard from_ methods not use cls.<attribute>

  JCZuurmond

  committed
  1ee945d

  Copy full SHA for 1ee945d

• Let query attributes be optional

  JCZuurmond

  committed
  08d0340

  Copy full SHA for 08d0340

• Let _include attributes on DashboardCrawlers be None

  JCZuurmond

  committed
  8d0d041

  Copy full SHA for 8d0d041

• Remove note about public method

  JCZuurmond

  committed
  fd8c335

  Copy full SHA for fd8c335

• Fix unit test

  JCZuurmond

  committed
  41c56a8

  Copy full SHA for 41c56a8

• Simplify get dashboards

  JCZuurmond

  committed
  2ed977e

  Copy full SHA for 2ed977e

• Move force refresh of Redash dashboards to cli

  JCZuurmond

  committed
  db04793

  Copy full SHA for db04793

• Created issue for TODO

  Show description for dea142c
  JCZuurmond

  committed
  dea142c

  Copy full SHA for dea142c

• Add creator to dashboard

  JCZuurmond

  committed
  2408958

  Copy full SHA for 2408958

• Add dashboard ownership

  JCZuurmond

  committed
  59dc713

  Copy full SHA for 59dc713

• Test variants of dashboard ownership

  JCZuurmond

  committed
  fd7b11a

  Copy full SHA for fd7b11a

• Format

  JCZuurmond

  committed
  d13823d

  Copy full SHA for d13823d

• Fix user id being an integer

  JCZuurmond

  committed
  ad5bb5e

  Copy full SHA for ad5bb5e

• Retrieve dashboard creator using creator id

  JCZuurmond

  committed
  81be495

  Copy full SHA for 81be495

• Mock Redash crawler in cli test

  JCZuurmond

  committed
  0769e6a

  Copy full SHA for 0769e6a

• Update dashboard tags in integration test

  JCZuurmond

  committed
  ad7aa43

  Copy full SHA for ad7aa43

• Avoid cls.<attribute> in Query classmethods

  JCZuurmond

  committed
  2bb5b2a

  Copy full SHA for 2bb5b2a

• Log databricks error on dashboard list as error

  JCZuurmond

  committed
  a211fe2

  Copy full SHA for a211fe2

• Log databricks error on legacy query list as error

  JCZuurmond

  committed
  76eb477

  Copy full SHA for 76eb477

• Update assert

  JCZuurmond

  committed
  630a88b

  Copy full SHA for 630a88b

• Refactor created queries

  JCZuurmond

  committed
  afc88f7

  Copy full SHA for afc88f7

• Import datetime as dt

  JCZuurmond

  committed
  ce546cb

  Copy full SHA for ce546cb

• Change asserts

  JCZuurmond

  committed
  d7c0d24

  Copy full SHA for d7c0d24

• Update list comprehension

  JCZuurmond

  committed
  65732ee

  Copy full SHA for 65732ee

• Fix integration test

  JCZuurmond

  committed
  6ade39b

  Copy full SHA for 6ade39b

• Log error when cannot list dashboard

  JCZuurmond

  committed
  125a916

  Copy full SHA for 125a916