Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure that table exists, even when crawlers produce zero records #373

Merged
merged 2 commits into from
Oct 4, 2023
Merged

Ensure that table exists, even when crawlers produce zero records #373

merged 2 commits into from
Oct 4, 2023

Conversation

FastLee
Copy link
Contributor

@FastLee FastLee commented Oct 3, 2023

Close #348

@codecov
Copy link

codecov bot commented Oct 3, 2023 •
edited
Loading

Codecov Report

Merging #373 (5715c57) into main (b1c3488) will decrease coverage by 0.03%.
The diff coverage is 95.23%.

❗ Current head 5715c57 differs from pull request most recent head f3015fe. Consider uploading reports for the commit f3015fe to get more accurate results

@@            Coverage Diff             @@
##             main     #373      +/-   ##
==========================================
- Coverage   83.76%   83.73%   -0.03%     
==========================================
  Files          30       30              
  Lines        2335     2337       +2     
  Branches      411      410       -1     
==========================================
+ Hits         1956     1957       +1     
- Misses        292      293       +1     
  Partials       87       87
Files Coverage Δ
src/databricks/labs/ucx/assessment/crawlers.py 73.65% <100.00%> (+0.14%) ⬆️
...databricks/labs/ucx/hive_metastore/data_objects.py 80.85% <100.00%> (ø)
src/databricks/labs/ucx/hive_metastore/grants.py 100.00% <100.00%> (ø)
src/databricks/labs/ucx/hive_metastore/mounts.py 88.00% <100.00%> (ø)
src/databricks/labs/ucx/hive_metastore/tables.py 94.65% <100.00%> (ø)
...rc/databricks/labs/ucx/workspace_access/manager.py 95.23% <100.00%> (ø)
src/databricks/labs/ucx/framework/crawlers.py 86.79% <90.90%> (-0.55%) ⬇️

nfx
nfx requested changes Oct 3, 2023
View reviewed changes
src/databricks/labs/ucx/framework/crawlers.py Outdated Show resolved Hide resolved
src/databricks/labs/ucx/framework/crawlers.py Outdated Show resolved Hide resolved
tests/unit/framework/mocks.py Outdated Show resolved Hide resolved
src/databricks/labs/ucx/framework/crawlers.py Outdated Show resolved Hide resolved
src/databricks/labs/ucx/framework/crawlers.py Outdated Show resolved Hide resolved
nfx
nfx approved these changes Oct 4, 2023
View reviewed changes
src/databricks/labs/ucx/framework/crawlers.py
if len(rows) == 0:
return

klass = rows[0].__class__
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously we got the klass by looking at the first record, now we add it as the explicit argument. Do we really want to add explicit argument?

@nfx nfx changed the title Added capability for adding empty tables to SqlBackend Ensure that table exists, even when crawlers produce zero records Oct 4, 2023
@nfx nfx added this pull request to the merge queue Oct 4, 2023
Merged via the queue into main with commit 47123a9 Oct 4, 2023
zpappa pushed a commit that referenced this pull request Oct 4, 2023
@FastLee @zpappa
Ensure that table exists, even when crawlers produce zero records (#373)
8d011ab 
Close #348
@nfx nfx mentioned this pull request Oct 12, 2023
Merged
nfx added a commit that referenced this pull request Oct 12, 2023
@nfx
Release v0.4.0 (#447)
1d6377e 
# Version changelog

## 0.4.0

* Added exception handling for secret scope not found.
([#418](#418)).
* Added a crawler for creating an inventory of Azure Service Principals
([#326](#326)).
* Added check if account group already exists during failure recovery
([#446](#446)).
* Added checking for index out of range.
([#429](#429)).
* Added hyperlink to UCX releases in the main readme
([#408](#408)).
* Added integration test to check backup groups get deleted
([#387](#387)).
* Added logging of errors during threadpool operations.
([#376](#376)).
* Added recovery mode for workspace-local groups from temporary groups
([#435](#435)).
* Added support for migrating Legacy Table ACLs from workspace-local to
account-level groups
([#412](#412)).
* Added detection for installations of unreleased versions
([#399](#399)).
* Decoupled `PermissionsManager` from `GroupMigrationToolkit`
([#407](#407)).
* Enabled debug logging for every job task run through a file, which is
accessible from both workspace UI and Databricks CLI
([#426](#426)).
* Ensured that table exists, even when crawlers produce zero records
([#373](#373)).
* Extended test suite for HMS->HMS TACL migration
([#439](#439)).
* Fixed handling of secret scope responses
([#431](#431)).
* Fixed `crawl_permissions` task to respect 'workspace_start_path'
config ([#444](#444)).
* Fixed broken logic in `parallel` module and applied hardened error
handling design for parallel code
([#405](#405)).
* Fixed codecov.io reporting
([#403](#403)).
* Fixed integration tests for crawlers
([#379](#379)).
* Improved README.py and logging messages
([#433](#433)).
* Improved cleanup for workspace backup groups by adding more retries on
errors ([#375](#375)).
* Improved dashboard queries to show unsupported storage types.
([#398](#398)).
* Improved documentation for readme notebook
([#257](#257)).
* Improved test coverage for installer
([#371](#371)).
* Introduced deterministic `env_or_skip` fixture for integration tests
([#396](#396)).
* Made HMS & UC fixtures return `CatalogInfo`, `SchemaInfo`, and
`TableInfo` ([#409](#409)).
* Merge `workspace_access.Crawler` and `workspace_access.Applier`
interfaces to `workspace_access.AclSupport`
([#436](#436)).
* Moved examples to docs
([#404](#404)).
* Properly isolated integration testing for workflows on an existing
shared cluster ([#414](#414)).
* Removed thread pool for any IAM Group removals and additions
([#394](#394)).
* Replace plus char with minus in version tag for GCP dev installation
of UCX ([#420](#420)).
* Run integration tests on shared clusters for a faster devloop
([#397](#397)).
* Show difference between serverless and PRO warehouses during
installation ([#385](#385)).
* Split `migrate-groups` workflow into three different stages for
reliability ([#442](#442)).
* Use groups instead of usernames in code owners file
([#389](#389)).
@nfx nfx deleted the fix/empty_table branch October 17, 2023 22:44
FastLee pushed a commit that referenced this pull request Oct 25, 2023
@nfx @FastLee
Release v0.4.0 (#447)
7fff6ac 
# Version changelog

## 0.4.0

* Added exception handling for secret scope not found.
([#418](#418)).
* Added a crawler for creating an inventory of Azure Service Principals
([#326](#326)).
* Added check if account group already exists during failure recovery
([#446](#446)).
* Added checking for index out of range.
([#429](#429)).
* Added hyperlink to UCX releases in the main readme
([#408](#408)).
* Added integration test to check backup groups get deleted
([#387](#387)).
* Added logging of errors during threadpool operations.
([#376](#376)).
* Added recovery mode for workspace-local groups from temporary groups
([#435](#435)).
* Added support for migrating Legacy Table ACLs from workspace-local to
account-level groups
([#412](#412)).
* Added detection for installations of unreleased versions
([#399](#399)).
* Decoupled `PermissionsManager` from `GroupMigrationToolkit`
([#407](#407)).
* Enabled debug logging for every job task run through a file, which is
accessible from both workspace UI and Databricks CLI
([#426](#426)).
* Ensured that table exists, even when crawlers produce zero records
([#373](#373)).
* Extended test suite for HMS->HMS TACL migration
([#439](#439)).
* Fixed handling of secret scope responses
([#431](#431)).
* Fixed `crawl_permissions` task to respect 'workspace_start_path'
config ([#444](#444)).
* Fixed broken logic in `parallel` module and applied hardened error
handling design for parallel code
([#405](#405)).
* Fixed codecov.io reporting
([#403](#403)).
* Fixed integration tests for crawlers
([#379](#379)).
* Improved README.py and logging messages
([#433](#433)).
* Improved cleanup for workspace backup groups by adding more retries on
errors ([#375](#375)).
* Improved dashboard queries to show unsupported storage types.
([#398](#398)).
* Improved documentation for readme notebook
([#257](#257)).
* Improved test coverage for installer
([#371](#371)).
* Introduced deterministic `env_or_skip` fixture for integration tests
([#396](#396)).
* Made HMS & UC fixtures return `CatalogInfo`, `SchemaInfo`, and
`TableInfo` ([#409](#409)).
* Merge `workspace_access.Crawler` and `workspace_access.Applier`
interfaces to `workspace_access.AclSupport`
([#436](#436)).
* Moved examples to docs
([#404](#404)).
* Properly isolated integration testing for workflows on an existing
shared cluster ([#414](#414)).
* Removed thread pool for any IAM Group removals and additions
([#394](#394)).
* Replace plus char with minus in version tag for GCP dev installation
of UCX ([#420](#420)).
* Run integration tests on shared clusters for a faster devloop
([#397](#397)).
* Show difference between serverless and PRO warehouses during
installation ([#385](#385)).
* Split `migrate-groups` workflow into three different stages for
reliability ([#442](#442)).
* Use groups instead of usernames in code owners file
([#389](#389)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nfx nfx nfx approved these changes

@larsgeorge-db larsgeorge-db Awaiting requested review from larsgeorge-db

Assignees
No one assigned
Labels
None yet
Projects
UCX
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Upgrade fails the Assessment Report
2 participants
@FastLee @nfx