v0.5.0

• Added make install-dev and a stronger make clean for easier dev on-boarding and release upgrades (#458).
• Added failure summary in the assessment dashboard (#455).
• Added test for checking grants in default schema (#470).
• Added unit tests for generic permissions (#457).
• Enabled integration tests via OIDC for every pull request (#378).
• Added check if permissions are up to date (#421).
• Fixed casing in all_tables.sql query. (#464).
• Fixed missed scans for empty databases and views in crawl_grants (#469).
• Improved logging colors for dark terminal backgrounds (#468).
• Improved local group migration state handling and made log files flush every 10 minutes (#449).
• Moved workspace listing as a separate task for an assessment workflow (#437).
• Removed rate limit for get or create backup group to speed up the prepare environment (#453).
• Updated databricks-sdk requirement from ~=0.10.0 to ~=0.11.0 (#448).

v0.4.0

• Added exception handling for secret scope not found. (#418).
• Added a crawler for creating an inventory of Azure Service Principals (#326).
• Added check if account group already exists during failure recovery (#446).
• Added checking for index out of range. (#429).
• Added hyperlink to UCX releases in the main readme (#408).
• Added integration test to check backup groups get deleted (#387).
• Added logging of errors during threadpool operations. (#376).
• Added recovery mode for workspace-local groups from temporary groups (#435).
• Added support for migrating Legacy Table ACLs from workspace-local to account-level groups (#412).
• Added detection for installations of unreleased versions (#399).
• Decoupled PermissionsManager from GroupMigrationToolkit (#407).
• Enabled debug logging for every job task run through a file, which is accessible from both workspace UI and Databricks CLI (#426).
• Ensured that table exists, even when crawlers produce zero records (#373).
• Extended test suite for HMS->HMS TACL migration (#439).
• Fixed handling of secret scope responses (#431).
• Fixed crawl_permissions task to respect 'workspace_start_path' config (#444).
• Fixed broken logic in parallel module and applied hardened error handling design for parallel code (#405).
• Fixed codecov.io reporting (#403).
• Fixed integration tests for crawlers (#379).
• Improved README.py and logging messages (#433).
• Improved cleanup for workspace backup groups by adding more retries on errors (#375).
• Improved dashboard queries to show unsupported storage types. (#398).
• Improved documentation for readme notebook (#257).
• Improved test coverage for installer (#371).
• Introduced deterministic env_or_skip fixture for integration tests (#396).
• Made HMS & UC fixtures return CatalogInfo, SchemaInfo, and TableInfo (#409).
• Merge workspace_access.Crawler and workspace_access.Applier interfaces to workspace_access.AclSupport (#436).
• Moved examples to docs (#404).
• Properly isolated integration testing for workflows on an existing shared cluster (#414).
• Removed thread pool for any IAM Group removals and additions (#394).
• Replace plus char with minus in version tag for GCP dev installation of UCX (#420).
• Run integration tests on shared clusters for a faster devloop (#397).
• Show difference between serverless and PRO warehouses during installation (#385).
• Split migrate-groups workflow into three different stages for reliability (#442).
• Use groups instead of usernames in code owners file (#389).

v0.3.0

• Added inventory_database name check during installation (#275).
• Added a column to $inventory.tables to specify if a table might have been synchronised to Unity Catalog already or not (#306).
• Added a migration state to skip already migrated tables (#325).
• Fixed appending to tables by adding filtering of None rows (#356).
• Fixed handling of missing but linked cluster policies. (#361).
• Ignore errors for Redash widgets and queries redeployment during installation (#367).
• Remove exception and added proper logging for groups in the list that… (#357).
• Skip group migration when no groups are available after preparation step. (#363).
• Update databricks-sdk requirement from ~=0.9.0 to ~=0.10.0 (#362).

Kudos to @william-conti @FastLee @larsgeorge-db @nfx @dmoore247

v0.2.0

• Added retrieving for all account-level groups with matching names to workspace-level groups in case no explicit configuration (#277).
• Added crawler for Azure Service principals used for direct storage access (#305).
• Added more SQL queries to the assessment step dashboard (#269).
• Added filtering out for job clusters in the clusters crawler (#298).
• Added recording errors from crawl_tables step in $inventory.table_failures table and display counter on the dashboard (#300).
• Added comprehensive introduction user manual (#273).
• Added interactive tutorial for local group migration readme (#291).
• Added tutorial links to the landing page of documentation (#290).
• Added (internal) support for account-level configuration and multi-cloud workspace list (#264).
• Improved order of tasks in the README notebook (#286).
• Improved installation script to run in a Windows Git Bash terminal (#282).
• Improved installation script by setting log level to uppercase by default (#271).
• Improved installation finish messages within installer script (#267).
• Improved automation for MANAGED table migration and continued building tables migration component (#295).
• Fixed debug notebook code with refactored package structure (#250) (#265).
• Fixed replacement of custom configured database to replicate in the report for external locations (#296).
• Removed redundant notebooks top-level folder (#263).
• Split checking for test failures and linting errors into independent GitHub Actions checks (#287).
• Verify query metadata for assessment dashboards during unit tests (#294).

Kudos to @tamilselvanveeramani @larsgeorge-db @saraivdbx @dmoore247 @nfx @FastLee @william-conti @dipankarkush-db

v0.1.1

• Added batched iteration for INSERT INTO queries in StatementExecutionBackend with default max_records_per_batch=1000 (#237).
• Added crawler for mount points (#209).
• Added crawlers for compatibility of jobs and clusters, along with basic recommendations for external locations (#244).
• Added safe return on grants (#246).
• Added ability to specify empty group filter in the installer script (#216) (#217).
• Added ability to install application by multiple different users on the same workspace (#235).
• Added dashboard creation on installation and a requirement for warehouse_id in config, so that the assessment dashboards are refreshed automatically after job runs (#214).
• Added reliance on rate limiting from Databricks SDK for listing workspace (#258).
• Fixed errors in corner cases where Azure Service Principal Credentials were not available in Spark context (#254).
• Fixed DESCRIBE TABLE throwing errors when listing Legacy Table ACLs (#238).
• Fixed file already exists error in the installer script (#219) (#222).
• Fixed guess_external_locations failure with AttributeError: as_dict and added an integration test (#259).
• Fixed error handling edge cases in crawl_tables task (#243) (#251).
• Fixed crawl_permissions task failure on folder names containing a forward slash (#234).
• Improved README notebook documentation (#260, #228, #252, #223, #225).
• Removed redundant .python-version file (#221).
• Removed discovery of account groups from crawl_permissions task (#240).
• Updated databricks-sdk requirement from ~=0.8.0 to ~=0.9.0 (#245).

Kudos to @larsgeorge-db @william-conti @dmoore247 @tamilselvanveeramani @nfx @FastLee

v0.1.0

Version changelog

0.1.0

Features

• Added interactive installation wizard (#184, #117).
• Added schedule of jobs as part of install.sh flow and created some documentation (#187).
• Added debug notebook companion to troubleshoot the installation (#191).
• Added support for Hive Metastore Table ACLs inventory from all databases (#78, #122, #151).
• Created $inventory.tables from Scala notebook (#207).
• Added local group migration support for ML-related objects (#56).
• Added local group migration support for SQL warehouses (#57).
• Added local group migration support for all compute-related resources (#53).
• Added local group migration support for security-related objects (#58).
• Added local group migration support for workflows (#54).
• Added local group migration support for workspace-level objects (#59).
• Added local group migration support for dashboards, queries, and alerts (#144).

Stability

• Added codecov.io publishing (#204).
• Added more tests to group.py (#148).
• Added tests for group state (#133).
• Added tests for inventorizer and typed (#125).
• Added tests WorkspaceListing (#110).
• Added make_*_permissions fixtures (#159).
• Added reusable fixtures module (#119).
• Added testing for permissions (#126).
• Added inventory table manager tests (#153).
• Added product_info to track as SDK integration (#76).
• Added failsafe permission get operations (#65).
• Always install the latest pip version in ./install.sh (#201).
• Always store inventory in hive_metastore and make only inventory_database configurable (#178).
• Changed default logging level from TRACE to DEBUG log level (#124).
• Consistently use WorkspaceClient from databricks.sdk (#120).
• Convert pipeline code to use fixtures. (#166).
• Exclude mixins from coverage (#130).
• Fixed codecov.io reporting (#212).
• Fixed configuration path in job task install code (#210).
• Fixed a bug with dependency definitions (#70).
• Fixed failing test_jobs (#140).
• Fixed the issues with experiment listing (#64).
• Fixed integration testing configuration (#77).
• Make project runnable on nightly testing infrastructure (#75).
• Migrated cluster policies to new fixtures (#174).
• Migrated clusters to the new fixture framework (#162).
• Migrated instance pool to the new fixture framework (#161).
• Migrated to databricks.labs.ucx package (#90).
• Migrated token authorization to new fixtures (#175).
• Migrated experiment fixture to standard one (#168).
• Migrated jobs test to fixture based one. (#167).
• Migrated model fixture to the standard fixtures (#169).
• Migrated warehouse fixture to standard one (#170).
• Organise modules by domain (#197).
• Prefetch all account-level and workspace-level groups (#192).
• Programmatically create a dashboard (#121).
• Properly integrate Python logging facility (#118).
• Refactored code to use Databricks SDK for Python (#27).
• Refactored configuration and remove global provider state (#71).
• Removed pydantic dependency (#138).
• Removed redundant pyspark, databricks-connect, delta-spark, and pandas dependencies (#193).
• Removed redundant typer[all] dependency and its usages (#194).
• Renamed MigrationGroupsProvider to GroupMigrationState (#81).
• Replaced ratelimit and tenacity dependencies with simpler implementations (#195).
• Reorganised integration tests to align more with unit tests (#206).
• Run build workflow also on main branch (#211).
• Run integration test with a single group (#152).
• Simplify SqlBackend and table creation logic (#203).
• Updated migration_config.yml (#179).
• Updated legal information (#196).
• Use make_secret_scope fixture (#163).
• Use fixture factory for make_table, make_schema, and make_catalog (#189).
• Use new fixtures for notebooks and folders (#176).
• Validate toolkit notebook test (#183).

Contributing

• Added a note on external dependencies (#139).
• Added ability to run SQL queries on Spark when in Databricks Runtime (#108).
• Added some ground rules for contributing (#82).
• Added contributing instructions link from main readme (#109).
• Added info about environment refreshes (#155).
• Clarified documentation (#137).
• Enabled merge queue (#146).
• Improved CONTRIBUTING.md guide (#135, #145).

Kudos to @dependabot @nsenno-dbr @renardeinside @nfx @william-conti @larsgeorge-db @HariGS-DB @saraivdbx

v0.0.2

Lates version before SDK version

v0.0.1

With this release we introduce the first public version of the UC Upgrade repo. There will be a lot of changes further, therefore anybody who is looking for the latest working version shall use release 0.0.1 unless a different release comes out.