Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs improvements #115

Merged
merged 32 commits into from
Sep 5, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
28f256b
update readme
graciegoheen Aug 16, 2023
f27587b
update main docs page
graciegoheen Aug 16, 2023
1fdda5e
add template examples page
graciegoheen Aug 16, 2023
db35ab7
initial create-group example
graciegoheen Aug 17, 2023
2899aa7
add version example
graciegoheen Aug 17, 2023
f524125
add add-contract example
graciegoheen Aug 17, 2023
0aed2ab
add note on connection multiple pre-existing projects
graciegoheen Aug 17, 2023
e64ee54
finish create-group example
graciegoheen Aug 17, 2023
4d32714
add group example
graciegoheen Aug 17, 2023
ffd9717
started on split example
graciegoheen Aug 18, 2023
60fefce
add split example
graciegoheen Aug 18, 2023
228ea6d
start connect example
graciegoheen Aug 18, 2023
8cee281
first pass at connect docs
graciegoheen Aug 18, 2023
ece1a72
Merge branch 'main' into docs_improvements
graciegoheen Aug 29, 2023
89a3133
fix merge conflict
graciegoheen Aug 29, 2023
f103d78
add note about leaf nodes to split command example
graciegoheen Aug 29, 2023
8fc78b8
updated group commands to "protected" access
graciegoheen Aug 29, 2023
ab70e42
Update public -> protected for group command
graciegoheen Aug 29, 2023
df25d96
docs updates
graciegoheen Aug 29, 2023
2be23c0
Merge branch 'docs_improvements' of github.com:dbt-labs/dbt-meshify i…
graciegoheen Aug 29, 2023
17ded0b
Update selector syntax flags
graciegoheen Aug 29, 2023
85d9d02
add "What dbt-meshify does not handle" section
graciegoheen Aug 29, 2023
f44b65b
update index and readme
graciegoheen Sep 5, 2023
21b6b3d
Update docs/examples.md
graciegoheen Sep 5, 2023
390c0af
add group clarification
graciegoheen Sep 5, 2023
798e25f
add create-path flag callout
graciegoheen Sep 5, 2023
d3adf95
Update docs/examples.md
graciegoheen Sep 5, 2023
b40458a
Update docs/index.md
graciegoheen Sep 5, 2023
32ac6b4
Merge branch 'main' into docs_improvements
graciegoheen Sep 5, 2023
743358e
add missing screenshots
graciegoheen Sep 5, 2023
410dedd
add example page, apply formatting fixes
dave-connors-3 Sep 5, 2023
3fe8f5a
Merge branch 'main' into docs_improvements
dave-connors-3 Sep 5, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 3 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,20 +17,10 @@ These dbt-core features include:
3. __[Access](https://docs.getdbt.com/docs/collaborate/govern/model-access)__ - control the `access` level of models within groups
4. __[Versions](https://docs.getdbt.com/docs/collaborate/govern/model-versions)__ - create and increment versions of particular models.

Additionally, `dbt-meshify` automates the code development required to split a monolithic dbt project into component projects using cross-project `ref`.

## Installation

```bash
pip install dbt-meshify
```

## Basic Usage

```bash
# create a group of all models tagged with "finance"
# leaf nodes and nodes with cross-group dependencies will be `public`
# public nodes will also have contracts added to them
dbt-meshify group finance --owner-name "Monopoly Man" -s +tag:finance

# optionally use the add-version operation to add a new version to a model
dbt-meshify operation add-version -s fct_orders
```
```
35 changes: 35 additions & 0 deletions docs/examples.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Examples

For consistency and clarity of the following examples, we're going to use a simplified dbt project. In practice, the model governance features describe are _most_ beneficial for large dbt projects that are struggling to scale.

Let's imagine a dbt project with the following models:

TO DO: add DAG from example project here
graciegoheen marked this conversation as resolved.
Show resolved Hide resolved

## Component commands

### Create a new group

TO DO: add example here

### Add/increment model versions

TO DO: add example here

### Add contract(s)

TO DO: add example here

## Global commands

## Group together a subset of models

TO DO: add example here

## Split out a new subproject

TO DO: add example here

## Connect multiple dbt projects

TO DO: add example here
44 changes: 25 additions & 19 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# dbt_meshify

`dbt-meshify` is a dbt-core plugin that automates the management and creation of dbt-core model governance features introduced in dbt-core v1.5. Each command in the package will leverage your dbt project metadata to create and/or edit the files in your project to properly configure the models in your project with these governance features.
`dbt-meshify` is a dbt-core plugin that automates the creation of dbt-core model governance features introduced in dbt-core v1.5. This package will leverage your dbt project metadata to create and/or edit the files in your project to properly configure the models in your project with these governance features.
graciegoheen marked this conversation as resolved.
Show resolved Hide resolved

These dbt-core features include:

Expand All @@ -9,31 +9,37 @@ These dbt-core features include:
3. __[Access](https://docs.getdbt.com/docs/collaborate/govern/model-access)__ - control the `access` level of models within groups
4. __[Versions](https://docs.getdbt.com/docs/collaborate/govern/model-versions)__ - create and increment versions of particular models.

Additionally, `dbt-meshify` automates the code development required to split a monolithic dbt project into component projects using cross-project `ref`.
graciegoheen marked this conversation as resolved.
Show resolved Hide resolved

This package leverages the dbt-core Python API to allow users to use standard dbt selection syntax for each of the commands in this package (unless otherwise noted). See details on each of the specific commands available on the [commands page](commands.md)

## Basic Usage
## Getting Started

This package helps automate the code development required for adding the dbt-core model governance features mentioned above.

The first question to ask yourself is "which of these features do I want to add to my project"? Do you want to add contracts, create a new group, split your monolithic dbt project in two? Your answer to this question will establish which `dbt-meshify` command is right for you!

This package consists of **component** and **global** commands - so you can decide how to best break apart your work.

Each of the available [commands page](commands.md) allows you to add one (or many) of the above features to a set of models specified by the selection syntax in the command.
The **component** commands allow you to do a single step at a time and begin with `dbt-meshify operation`. For example, if you wanted to add a new version to a model, you would run something like `dbt-meshify operation add-version --select fct_orders`. This command would:
1. add a new version to `fct_orders`

The goal of this package is to make it more straightforward to apply to your project so that splitting apart a monolithic project into component projects is a more automated, dbt-tonic experience.
and that's it!

The process of splitting a dbt monolith apart roughly requires you to:
The **global** commands combine _multiple_ **component** commands to complete a larger set of work and begin with `dbt-meshify`. For example, if you wanted to define a group for a subset of your models, you would run something like `dbt-meshify group finance --owner-name "Monopoly Man" --select +tag:finance`. This command would:
1. define a new group named "finance" in your dbt project, setting the owner name to "Monopoly Man"
2. add all models tagged with "finance" to that new group
3. set `access` to public for all leaf nodes and nodes with cross-group dependencies
graciegoheen marked this conversation as resolved.
Show resolved Hide resolved
4. add contracts to all public nodes

1. Determine what parts of your project should be grouped together into subprojects
2. Determine the access-level for the members of that group
3. Add model contracts to the elements that are public and accessed my members outside the group specified in (1)
4. (Optional) Add model versions to the public models to allow for development without impacting downstream stakeholders.
all at once!

Here's how that might look for the process of creating a separate `finance` subproject in your dbt monolith.
The next question to ask yourself is "which of my models do I want to add these feature(s) to?". This informs the selection syntax you provide to the `dbt-meshify` command of choice. `dbt-meshify` uses the same selection syntax as `dbt`, so you can `--select` based on model names, tags, and so on!
graciegoheen marked this conversation as resolved.
Show resolved Hide resolved

```bash
# create a group of all models tagged with "finance"
# leaf nodes and nodes with cross-group dependencies will be `public`
# public nodes will also have contracts added to them
dbt-meshify group finance --owner-name "Monopoly Man" -s +tag:finance
Once you've decided:
1. which governance feature(s) you want to add to your dbt project
2. which subset of models you want to add those feature(s) to

# optionally use the add-version operation to add a new version to a model
dbt-meshify operation add-version -s fct_orders
```
you're ready to use `dbt-meshify`!

Future releases of this package may also include features that allow users to fully split off groups of models into entirely new dbt projects.
For further information, check out the available [commands](commands.md) or read through some [examples](examples.md).