Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dashboard crawlers #3397

Merged
merged 182 commits into from
Dec 13, 2024
Merged
Show file tree
Hide file tree
Changes from 181 commits
Commits
Show all changes
182 commits
Select commit Hold shift + click to select a range
4d98773
Setup framework for crawling dashboards
JCZuurmond Nov 28, 2024
a930bf7
Move lint related dashboard integration test to source code directory
JCZuurmond Nov 28, 2024
482e968
Start with empty crawl
JCZuurmond Nov 28, 2024
ac7fb08
Add integration test for crawling dashboard
JCZuurmond Nov 28, 2024
19638fa
Crawl Redash dashboards
JCZuurmond Nov 28, 2024
3db33b5
Test include dashboard parameter
JCZuurmond Nov 28, 2024
401679f
Add include dashboard parameter
JCZuurmond Nov 28, 2024
d4be79b
Rename crawler variable
JCZuurmond Nov 28, 2024
2788888
Test crawl LakeviewDashboards
JCZuurmond Nov 28, 2024
ba0ce24
Crawl lakeview dashboards
JCZuurmond Nov 28, 2024
364bb1b
Test include Lakeview dashboard ids
JCZuurmond Nov 28, 2024
0936e02
Implement include Lakeview dashboards
JCZuurmond Nov 28, 2024
9f5853e
Add Redash or Lakeview to log messages to differentiate
JCZuurmond Nov 28, 2024
304f911
Differentiate between Redash and Lakeview dashboards
JCZuurmond Nov 28, 2024
395e099
Format
JCZuurmond Nov 28, 2024
44aa79b
List dashboard
JCZuurmond Nov 28, 2024
dfa0f7a
Assert mock dashboards to have an ID
JCZuurmond Nov 28, 2024
8c297e0
Add dashboard crawlers to RuntimeContext
JCZuurmond Nov 28, 2024
9de0611
Update comment on include_dashboards_ids
JCZuurmond Nov 28, 2024
e8fbb78
Move Redash dashboard crawler to global context
JCZuurmond Nov 28, 2024
0d23447
Add dashboard crawlers to assessment workflow
JCZuurmond Nov 28, 2024
b31b83f
Add Redash and Lakeview dashboard tables to install
JCZuurmond Nov 28, 2024
0c67e3a
Update Redash migration to use RedashDashboardCrawler
JCZuurmond Nov 28, 2024
aa48e6c
Make get queries from dashboard protected
JCZuurmond Nov 29, 2024
430806d
Force Redash dashboard crawler key word arguments
JCZuurmond Nov 29, 2024
a31d643
Lower case B in dashboard
JCZuurmond Nov 29, 2024
92a26c3
Handle non specified dashboard id
JCZuurmond Nov 29, 2024
3cf04e4
Pass Redash crawler in global context
JCZuurmond Nov 29, 2024
67c47a6
Handle no dashboard id in revert DBSQL dashboards cli command
JCZuurmond Nov 29, 2024
6a59a08
Fix Redash integration test
JCZuurmond Nov 29, 2024
60fc9f9
Add parent field to RedashDashboard
JCZuurmond Nov 29, 2024
9d86e7d
Extend RedashDashboard from sdk Dashboard
JCZuurmond Nov 29, 2024
7308f7e
Skip dashboard without id
JCZuurmond Nov 29, 2024
645043f
Get dashboard in fixture to update widget
JCZuurmond Dec 2, 2024
3da399d
Skip dashboards that miss ids
JCZuurmond Dec 2, 2024
c56c6bf
Fix unit tests
JCZuurmond Dec 2, 2024
e763ee4
Fix integration test
JCZuurmond Dec 2, 2024
4487039
Add TODO for linting lakeview dashboards
JCZuurmond Dec 2, 2024
8abcd50
Fix integration test including dashboard
JCZuurmond Dec 2, 2024
bd5b2d2
Refactor while condition
JCZuurmond Dec 2, 2024
1829f78
Ignore too many public methods
JCZuurmond Dec 2, 2024
02872fb
Fix protected access in unit tests
JCZuurmond Dec 2, 2024
3b6f0e4
Add assert to mock
JCZuurmond Dec 2, 2024
03e4f61
Format
JCZuurmond Dec 2, 2024
dbbbfd4
Shorten variable name
JCZuurmond Dec 2, 2024
022a7fd
Update cli tests
JCZuurmond Dec 2, 2024
fef9d5d
Add attributes to LakeviewDashboard
JCZuurmond Dec 2, 2024
00066c6
Lint Lakeview dashboards in QueryLinter
JCZuurmond Dec 2, 2024
2731a67
Expect tags on LakeviewDashboard
JCZuurmond Dec 2, 2024
584062e
Remove tags from LakeviewDashboard
JCZuurmond Dec 2, 2024
2b1f1ba
Test attributes on LakeviewDashboard
JCZuurmond Dec 3, 2024
85e21b9
Create LakeviewDashboard from SdkLakeviewDashboard
JCZuurmond Dec 3, 2024
0c5feb2
Test redash Dashboard crawler to persist dashboards
JCZuurmond Dec 3, 2024
111c1bf
Test handling DatabricksError on list
JCZuurmond Dec 3, 2024
de53c11
Test handling DatabricksError on iterate
JCZuurmond Dec 3, 2024
5be1a45
Test debug listing upper limit
JCZuurmond Dec 3, 2024
263d6f7
Fix condition for debug listing upper limit
JCZuurmond Dec 3, 2024
434fbed
Test warning logs
JCZuurmond Dec 3, 2024
89b7063
Test getting dashboard with dashboard ids
JCZuurmond Dec 3, 2024
b48d646
Test for dashboard id not found
JCZuurmond Dec 3, 2024
f05bf24
Test for skipping dashboard without id
JCZuurmond Dec 3, 2024
e431c22
Add comment about API listing for Lakeview dashboards
JCZuurmond Dec 3, 2024
ab36522
Duplicate dashboard crawler tests for Lakeview
JCZuurmond Dec 3, 2024
b98af1b
Skip Lakeview dashboards without id
JCZuurmond Dec 3, 2024
5afe53f
Fix crawling lakeview dashboards
JCZuurmond Dec 3, 2024
d98c808
Force include dashboard ids to be keyword argument
JCZuurmond Dec 3, 2024
3442918
Fix typo
JCZuurmond Dec 3, 2024
5c99bfd
Force Lakeview dashboard fixture to have keyword arguments
JCZuurmond Dec 3, 2024
bc7bf54
Add query parameter to make_lakeview_dashboard fixture
JCZuurmond Dec 3, 2024
3ca8656
Fix return type
JCZuurmond Dec 3, 2024
27efdc4
Rename dashboard classes
JCZuurmond Dec 3, 2024
695338d
Rename Lakeview fixture dashboard query
JCZuurmond Dec 3, 2024
a59cd3a
Add Lakeview dashboard fixture to mock runtime
JCZuurmond Dec 3, 2024
f1a65df
Fix reference to dataset
JCZuurmond Dec 3, 2024
98a5ccb
Fix getting dashboard ids
JCZuurmond Dec 3, 2024
c61c9b0
Add get query to RedashDashboardCrawler
JCZuurmond Dec 3, 2024
8eae204
Test getting Redash query
JCZuurmond Dec 3, 2024
59c48f9
Test getting non-existing query
JCZuurmond Dec 3, 2024
04282e9
Add list queries to RedashDashboardCrawler
JCZuurmond Dec 3, 2024
a81062e
Test listing Redash queries
JCZuurmond Dec 3, 2024
f04303b
Implement listing redash queries
JCZuurmond Dec 3, 2024
dc0120b
Test handling permission denied error when listing Redash queries
JCZuurmond Dec 3, 2024
95b27aa
Handle Databricks error when listing Redash queries
JCZuurmond Dec 3, 2024
88c0f7a
Assert warning when listing Redash queries
JCZuurmond Dec 3, 2024
d6158b6
Add list queries method to LakeviewDashboardCrawler
JCZuurmond Dec 3, 2024
b55031a
Test listing Lakeview queries
JCZuurmond Dec 3, 2024
ce3ad35
Add listing Lakeview queries
JCZuurmond Dec 3, 2024
5c32c6f
Fix test for handling permission denied in Lakeview queries list
JCZuurmond Dec 3, 2024
74a06ca
Remove redundant dashboard id
JCZuurmond Dec 3, 2024
7a5d02d
Test handling corrupted serialized dashboard
JCZuurmond Dec 3, 2024
f4f62cb
Explain difference between Lakeview and Redash list queries
JCZuurmond Dec 3, 2024
f8c74a9
Add get_query method to LakeviewDashboardCrawler
JCZuurmond Dec 3, 2024
b45b8af
Test get Lakeview query
JCZuurmond Dec 3, 2024
e8b43d0
Implement get Lakeview query
JCZuurmond Dec 3, 2024
a886a2e
Fix get query type hint
JCZuurmond Dec 3, 2024
8e9d7a6
Extract converting sdk to lsql Lakeview dashboard
JCZuurmond Dec 3, 2024
e914814
Let Redash query methods return strings
JCZuurmond Dec 3, 2024
111cbf3
Refactor get_query to get_queries
JCZuurmond Dec 3, 2024
52e615c
Merge get_queries with list_queries
JCZuurmond Dec 3, 2024
4711fa3
Handle query is None
JCZuurmond Dec 3, 2024
364d5a6
Format
JCZuurmond Dec 3, 2024
68fc0ab
Fix type hint
JCZuurmond Dec 3, 2024
da17cd7
Remove redudant json.JsonDecodeError
JCZuurmond Dec 3, 2024
4376dc3
Fix iterator import
JCZuurmond Dec 3, 2024
280da01
Assert len queries
JCZuurmond Dec 3, 2024
73299c2
Add make Lakeview dashboard fixture to MockInstallationContext
JCZuurmond Dec 3, 2024
b3ebdc6
Rename variables
JCZuurmond Dec 3, 2024
76fffcf
Add missing caplog
JCZuurmond Dec 3, 2024
4293ed4
Test caplog.messages instead of caplog.text
JCZuurmond Dec 3, 2024
166c200
Test invalid serialized json
JCZuurmond Dec 3, 2024
1176c40
Ignore too-many-locals
JCZuurmond Dec 3, 2024
6c84fb3
Add Query dataclass
JCZuurmond Dec 3, 2024
3a3d481
Add parent to Query
JCZuurmond Dec 3, 2024
d70fbd7
Fix type hints
JCZuurmond Dec 3, 2024
b062f84
Format
JCZuurmond Dec 3, 2024
8898d3e
Add debug limit to Redash queries listing
JCZuurmond Dec 3, 2024
18aa565
Let QueryLinter use the dashboard crawlers
JCZuurmond Dec 3, 2024
d8f72d4
Let Redash dashboard migration use legacy queries from crawler
JCZuurmond Dec 3, 2024
54bddd2
Fix integration test
JCZuurmond Dec 3, 2024
e4374a8
Update has_calls to assert_has_calls
JCZuurmond Dec 6, 2024
7619b1a
Merge Dashboard dataclasses
JCZuurmond Dec 6, 2024
d9621c1
Remove DashboardCrawlerType
JCZuurmond Dec 6, 2024
24f0908
Fix dashboard tests
JCZuurmond Dec 6, 2024
92f0d7e
Add include_query_ids to RedashDashboardCrawler
JCZuurmond Dec 6, 2024
9b4a6ff
Add include_query_ids to LakeviewDashboardCrawler
JCZuurmond Dec 6, 2024
c3380f1
Pass include query ids from config
JCZuurmond Dec 6, 2024
cd70491
Expose make query and include created query ids
JCZuurmond Dec 6, 2024
2158c59
Fix query linter integration test
JCZuurmond Dec 6, 2024
d1ecfea
Fix variable should be same type
JCZuurmond Dec 6, 2024
b621ef2
Add missing return
JCZuurmond Dec 6, 2024
fd95523
Fix wrong name in unit test
JCZuurmond Dec 6, 2024
c186e9e
Handle query id being None
JCZuurmond Dec 6, 2024
6aad514
Set back disable too-many-public-methods
JCZuurmond Dec 6, 2024
5204b12
Avoid duplicate queries in query id
JCZuurmond Dec 6, 2024
93672a1
Split Redash and Lakeview DFSA ownership test
JCZuurmond Dec 6, 2024
7f4a6ed
Mark Lakeview ownership to fail
JCZuurmond Dec 6, 2024
072f2e6
Scope queries in MockRuntimeContext
JCZuurmond Dec 6, 2024
3bb6091
Do not refresh dashboard
JCZuurmond Dec 6, 2024
a7e3b1f
Clarify migrate dashboard integration test
JCZuurmond Dec 6, 2024
0e52991
Shorten for-loop
JCZuurmond Dec 9, 2024
7b21f52
Move dashboards out of cached property
JCZuurmond Dec 9, 2024
a34a244
Test dashboard migration tags to be set
JCZuurmond Dec 9, 2024
abe1a93
Fix filtering dashboards
JCZuurmond Dec 9, 2024
2e6eefe
Refresh dashboards when reverting to get latest tags
JCZuurmond Dec 9, 2024
9c9796a
Wait for dashboard migration tag to be present in integration test
JCZuurmond Dec 9, 2024
b76dffa
Remove redundant for-loop
JCZuurmond Dec 9, 2024
bcb5445
Add dashboard tables to table persistence docs
JCZuurmond Dec 9, 2024
295559d
Add tags to Query
JCZuurmond Dec 9, 2024
5667196
Add catalog and schema to query
JCZuurmond Dec 9, 2024
a462860
Rewrite Redash to use Query instead of LegacyQuery
JCZuurmond Dec 9, 2024
73c63e2
Shorten filtering dashboards
JCZuurmond Dec 9, 2024
af0e3f1
Remove redundant return
JCZuurmond Dec 9, 2024
12e3c31
Add a tile for the dashboards
JCZuurmond Dec 9, 2024
5516693
Force change on test running real assessment job
JCZuurmond Dec 9, 2024
a1521ea
Make dashboard name and parent optional
JCZuurmond Dec 10, 2024
b45dcc8
Assert dashboard id
JCZuurmond Dec 10, 2024
1ee945d
Let dashboard from_ methods not use cls.<attribute>
JCZuurmond Dec 10, 2024
08d0340
Let query attributes be optional
JCZuurmond Dec 10, 2024
8d0d041
Let _include attributes on DashboardCrawlers be None
JCZuurmond Dec 10, 2024
fd8c335
Remove note about public method
JCZuurmond Dec 10, 2024
41c56a8
Fix unit test
JCZuurmond Dec 10, 2024
2ed977e
Simplify get dashboards
JCZuurmond Dec 10, 2024
db04793
Move force refresh of Redash dashboards to cli
JCZuurmond Dec 10, 2024
dea142c
Created issue for TODO
JCZuurmond Dec 10, 2024
2408958
Add creator to dashboard
JCZuurmond Dec 10, 2024
59dc713
Add dashboard ownership
JCZuurmond Dec 10, 2024
fd7b11a
Test variants of dashboard ownership
JCZuurmond Dec 10, 2024
d13823d
Format
JCZuurmond Dec 10, 2024
ad5bb5e
Fix user id being an integer
JCZuurmond Dec 10, 2024
81be495
Retrieve dashboard creator using creator id
JCZuurmond Dec 10, 2024
0769e6a
Mock Redash crawler in cli test
JCZuurmond Dec 10, 2024
ad7aa43
Update dashboard tags in integration test
JCZuurmond Dec 10, 2024
2bb5b2a
Avoid cls.<attribute> in Query classmethods
JCZuurmond Dec 12, 2024
a211fe2
Log databricks error on dashboard list as error
JCZuurmond Dec 12, 2024
76eb477
Log databricks error on legacy query list as error
JCZuurmond Dec 12, 2024
630a88b
Update assert
JCZuurmond Dec 12, 2024
afc88f7
Refactor created queries
JCZuurmond Dec 12, 2024
ce546cb
Import datetime as dt
JCZuurmond Dec 12, 2024
d7c0d24
Change asserts
JCZuurmond Dec 12, 2024
65732ee
Update list comprehension
JCZuurmond Dec 12, 2024
6ade39b
Fix integration test
JCZuurmond Dec 12, 2024
125a916
Log error when cannot list dashboard
JCZuurmond Dec 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/table_persistence.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ Table utilization per workflow:
| udfs | RW | RW | RO | | | | |
| logs | RW | | RW | RW | | RW | RW |
| recon_results | | | | | | | RW |
| redash_dashboards | RW | | | | | | RW |
| lakeview_dashboards | RW | | | | | | RW |

**RW** - Read/Write, the job generates or updates the table.<br/>
**RO** - Read Only
Expand Down Expand Up @@ -139,3 +141,16 @@ This is used by the permission crawler.
| object_type | string | type of object (NOTEBOOK, DIRECTORY, REPO, FILE, LIBRARY) |
| path | string | full path of the object in the workspace |
| language | string | language of the object (applicable for notebooks only) |


#### _$inventory_.redash_dashboards and _$inventory_.lakeview_dashboards

Holds a list of all Redash or Lakeview dashboards. This is used by the `QueryLinter` and `Redash` migration.

| Column | Datatype | Description | Comments |
|-----------|--------------|---------------------------------------------------------------------------------------------|----------|
| id | string | The ID for this dashboard. | |
| name | string | The title of the dashboard that appears in list views and at the top of the dashboard page. | |
| parent | string | The identifier of the workspace folder containing the object. | |
| query_ids | list[string] | The IDs of the queries referenced by this dashboard. | |
| tags | list[string] | The tags set on this dashboard. | |
Loading
Loading