Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added recovery mode for workspace-local groups from temporary groups #435

Merged
merged 2 commits into from
Oct 11, 2023
Merged

Added recovery mode for workspace-local groups from temporary groups #435

merged 2 commits into from
Oct 11, 2023

Conversation

nfx
Copy link
Collaborator

@nfx nfx commented Oct 11, 2023 •
edited
Loading

This fix helps to recover from a state where

  • the temporary groups were created,
  • the account groups were not,
  • the original workspace-local groups were deleted.
  • The fix recreates the original state, with recreating all the workspace-local groups, which does have a temp pair but not an account level pair attached to the workspace.

This helps to recreate the state of the migration with the already existing group_manager.migration_groups_provider

fannijako and others added 2 commits October 10, 2023 14:46
@fannijako
disaster recovery from temp state
07343b7 
This fix helps to recover from a state, where the temporary groups were created, the account groups were not and the original workspac-local groups were deleted.

The fix recreates the original state, with recreating all the workspace-local groups, which does have a temp pair but not an account level pair attached to the workspace.

This helps to recreate the state of the migration with the already existing group_manager.migration_groups_provider
@nfx
simplify logic and add documentation
d221a20
@nfx nfx requested review from a team October 11, 2023 14:44
@codecov
Copy link

codecov bot commented Oct 11, 2023

Codecov Report

Merging #435 (d221a20) into main (bc752e6) will decrease coverage by 1.07%.
Report is 4 commits behind head on main.
The diff coverage is 10.00%.

@@            Coverage Diff             @@
##             main     #435      +/-   ##
==========================================
- Coverage   84.22%   83.16%   -1.07%     
==========================================
  Files          31       31              
  Lines        2847     2899      +52     
  Branches      538      545       +7     
==========================================
+ Hits         2398     2411      +13     
- Misses        326      368      +42     
+ Partials      123      120       -3
Files Coverage Δ
src/databricks/labs/ucx/workspace_access/groups.py 85.86% <10.00%> (-4.37%) ⬇️

... and 5 files with indirect coverage changes

@nfx nfx merged commit 11753d9 into main Oct 11, 2023
@nfx nfx deleted the fix/issue_386_disaster_recovery branch October 11, 2023 14:47
@nfx nfx mentioned this pull request Oct 12, 2023
Merged
nfx added a commit that referenced this pull request Oct 12, 2023
@nfx
Release v0.4.0 (#447)
1d6377e 
# Version changelog

## 0.4.0

* Added exception handling for secret scope not found.
([#418](#418)).
* Added a crawler for creating an inventory of Azure Service Principals
([#326](#326)).
* Added check if account group already exists during failure recovery
([#446](#446)).
* Added checking for index out of range.
([#429](#429)).
* Added hyperlink to UCX releases in the main readme
([#408](#408)).
* Added integration test to check backup groups get deleted
([#387](#387)).
* Added logging of errors during threadpool operations.
([#376](#376)).
* Added recovery mode for workspace-local groups from temporary groups
([#435](#435)).
* Added support for migrating Legacy Table ACLs from workspace-local to
account-level groups
([#412](#412)).
* Added detection for installations of unreleased versions
([#399](#399)).
* Decoupled `PermissionsManager` from `GroupMigrationToolkit`
([#407](#407)).
* Enabled debug logging for every job task run through a file, which is
accessible from both workspace UI and Databricks CLI
([#426](#426)).
* Ensured that table exists, even when crawlers produce zero records
([#373](#373)).
* Extended test suite for HMS->HMS TACL migration
([#439](#439)).
* Fixed handling of secret scope responses
([#431](#431)).
* Fixed `crawl_permissions` task to respect 'workspace_start_path'
config ([#444](#444)).
* Fixed broken logic in `parallel` module and applied hardened error
handling design for parallel code
([#405](#405)).
* Fixed codecov.io reporting
([#403](#403)).
* Fixed integration tests for crawlers
([#379](#379)).
* Improved README.py and logging messages
([#433](#433)).
* Improved cleanup for workspace backup groups by adding more retries on
errors ([#375](#375)).
* Improved dashboard queries to show unsupported storage types.
([#398](#398)).
* Improved documentation for readme notebook
([#257](#257)).
* Improved test coverage for installer
([#371](#371)).
* Introduced deterministic `env_or_skip` fixture for integration tests
([#396](#396)).
* Made HMS & UC fixtures return `CatalogInfo`, `SchemaInfo`, and
`TableInfo` ([#409](#409)).
* Merge `workspace_access.Crawler` and `workspace_access.Applier`
interfaces to `workspace_access.AclSupport`
([#436](#436)).
* Moved examples to docs
([#404](#404)).
* Properly isolated integration testing for workflows on an existing
shared cluster ([#414](#414)).
* Removed thread pool for any IAM Group removals and additions
([#394](#394)).
* Replace plus char with minus in version tag for GCP dev installation
of UCX ([#420](#420)).
* Run integration tests on shared clusters for a faster devloop
([#397](#397)).
* Show difference between serverless and PRO warehouses during
installation ([#385](#385)).
* Split `migrate-groups` workflow into three different stages for
reliability ([#442](#442)).
* Use groups instead of usernames in code owners file
([#389](#389)).
FastLee pushed a commit that referenced this pull request Oct 25, 2023
@nfx @fannijako @FastLee
Added recovery mode for workspace-local groups from temporary groups (#…
78e573f 
…435)

This fix helps to recover from a state where

- the temporary groups were created,
- the account groups were not,
- the original workspace-local groups were deleted.
- The fix recreates the original state, with recreating all the
workspace-local groups, which does have a temp pair but not an account
level pair attached to the workspace.

This helps to recreate the state of the migration with the already
existing group_manager.migration_groups_provider

---------

Co-authored-by: Fanni Jakó <[email protected]>
FastLee pushed a commit that referenced this pull request Oct 25, 2023
@nfx @FastLee
Release v0.4.0 (#447)
7fff6ac 
# Version changelog

## 0.4.0

* Added exception handling for secret scope not found.
([#418](#418)).
* Added a crawler for creating an inventory of Azure Service Principals
([#326](#326)).
* Added check if account group already exists during failure recovery
([#446](#446)).
* Added checking for index out of range.
([#429](#429)).
* Added hyperlink to UCX releases in the main readme
([#408](#408)).
* Added integration test to check backup groups get deleted
([#387](#387)).
* Added logging of errors during threadpool operations.
([#376](#376)).
* Added recovery mode for workspace-local groups from temporary groups
([#435](#435)).
* Added support for migrating Legacy Table ACLs from workspace-local to
account-level groups
([#412](#412)).
* Added detection for installations of unreleased versions
([#399](#399)).
* Decoupled `PermissionsManager` from `GroupMigrationToolkit`
([#407](#407)).
* Enabled debug logging for every job task run through a file, which is
accessible from both workspace UI and Databricks CLI
([#426](#426)).
* Ensured that table exists, even when crawlers produce zero records
([#373](#373)).
* Extended test suite for HMS->HMS TACL migration
([#439](#439)).
* Fixed handling of secret scope responses
([#431](#431)).
* Fixed `crawl_permissions` task to respect 'workspace_start_path'
config ([#444](#444)).
* Fixed broken logic in `parallel` module and applied hardened error
handling design for parallel code
([#405](#405)).
* Fixed codecov.io reporting
([#403](#403)).
* Fixed integration tests for crawlers
([#379](#379)).
* Improved README.py and logging messages
([#433](#433)).
* Improved cleanup for workspace backup groups by adding more retries on
errors ([#375](#375)).
* Improved dashboard queries to show unsupported storage types.
([#398](#398)).
* Improved documentation for readme notebook
([#257](#257)).
* Improved test coverage for installer
([#371](#371)).
* Introduced deterministic `env_or_skip` fixture for integration tests
([#396](#396)).
* Made HMS & UC fixtures return `CatalogInfo`, `SchemaInfo`, and
`TableInfo` ([#409](#409)).
* Merge `workspace_access.Crawler` and `workspace_access.Applier`
interfaces to `workspace_access.AclSupport`
([#436](#436)).
* Moved examples to docs
([#404](#404)).
* Properly isolated integration testing for workflows on an existing
shared cluster ([#414](#414)).
* Removed thread pool for any IAM Group removals and additions
([#394](#394)).
* Replace plus char with minus in version tag for GCP dev installation
of UCX ([#420](#420)).
* Run integration tests on shared clusters for a faster devloop
([#397](#397)).
* Show difference between serverless and PRO warehouses during
installation ([#385](#385)).
* Split `migrate-groups` workflow into three different stages for
reliability ([#442](#442)).
* Use groups instead of usernames in code owners file
([#389](#389)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nfx @fannijako