Delta-Kusto is a Command-line interface (CLI) enabling Continuous Integration / Continuous Deployment (CI / CD) automation with Kusto objects (e.g. tables, functions, policies, security roles, etc.) in Azure Data Explorer (ADX) databases. It can work on a single database, multiple databases, or an entire cluster. It also supports multi-tenant scenarios.
Delta-Kusto is doing what SQL Database projects do for Microsoft SQL: enabling CI/CD, change management and source control of Kusto databases. It works with and produces Kusto scripts so it doesn't require a new language / serialization format and can therefore be used with other tools of the ADX ecosystem.
Delta-Kusto runs on Linux, Mac OS & Windows as a stand-alone executable. It is meant to be used in headless mode.
Delta-Kusto works on database structure, not data:
- Functions
- Tables / Columns
- Ingestion Mapping
- Policies
- Materialized Views
- Security roles
- External tables
- Continuous Export
See the documentation for details and tutorial section for different tutorials on Delta-Kusto.
The high-level view of delta-kusto is the following:
The green boxes (current and target) represent sources. A source can be:
- ADX Database
- Kusto script
Delta-Kusto computes the delta between the two sources. The delta is a Kusto script containing the kusto commands required to take the current source and bring it to target source. The delta script can be exported as a stand alone file or as a folder hierarchy of scripts for easier readability and git-diff. It can also be applied to the current ADX Database.
Human validation often are required, especially if .drop commands are issued (to prevent unwanted data lost).
Using different combinations of sources can enable different scenarios:
| Current | Target | Scenario | Description |
|---|---|---|---|
| ADX Database | Kusto scripts | CI / CD scenario | Push the state if a script set to an ADX database. |
| Kusto scripts | ADX Database | Determine gap between a script set and existing DB | Computed Delta script shows what would need to be added to the script set to obtain the state of the target database. |
| Empty | ADX Database | Reverse engineer a database | Special case of the previous scenario. The delta becomes the entire state of the target. |
| Kusto scripts | Kusto scripts | Offline sync | Compute a delta between two script sets. No live databases needed. |
| ADX Database | Kusto scripts | Live Sync | Find gap between two databases. |
The current release of Delta-Kusto includes functions, tables, ingestion mappings & the following policies:
- Auto Delete
- Caching
- Ingestion Batching
- Merge
- Retention
- Sharding
- Update
See the list of issues for details on upcoming features and bugs found.
- Sync Kusto - The Sync Kusto tool was built to help create a maintainable development process around Kusto.
- Azure DevOps Task for Azure Data Explorer - Azure Pipelines task that enables you to create release pipelines and deploy your database changes to your Azure Data Explorer databases.