Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

a rule check specification #6

Open
georgim0 opened this issue Dec 22, 2020 · 2 comments
Open

a rule check specification #6

georgim0 opened this issue Dec 22, 2020 · 2 comments
Assignees
@georgim0

Comments

@georgim0
Copy link
Contributor

georgim0 commented Dec 22, 2020 •
edited
Loading

add the following rules:
lineage checks:

  • 🆗 Disabled Scripts
  • 🆗 Orphan models
  • 🆗 Rejoined models. Given following model dependency: A -> B -> C. Model C should not depend on A
  • 🆗 Staging scripts referencing a staging script in a different area
  • 🆗 Staging scripts referencing a mart
  • 🆗 Marts directly referencing a source

meta checks on physical models(model that is not ephemeral):

advanced lineage check:

  • detect similar sub graphs which they originate from the same set of sources to figure out if there is a potential code reuse(enforce single source of truth, thinking that it would highlight duplicate code)
georgim0 added a commit that referenced this issue Dec 22, 2020
@georgim0
add rule to detect when a staging script references a mart #6
7202aa5
@georgim0
Copy link
Contributor Author

TODO:

  • think about seeds

georgim0 added a commit that referenced this issue Dec 22, 2020
@georgim0
add rule to detect when a staging script references a mart #6
5270644
georgim0 added a commit that referenced this issue Dec 22, 2020
@georgim0
add rule to find mart references to source #6
921cf26
@georgim0 georgim0 mentioned this issue Dec 22, 2020
Merged
georgim0 added a commit that referenced this issue Dec 22, 2020
@georgim0
add rule to detect when a staging script references a mart #6
99d2cbb
georgim0 added a commit that referenced this issue Dec 22, 2020
@georgim0
add rule to find mart references to source #6
f990152
georgim0 added a commit that referenced this issue Dec 22, 2020
@georgim0
add rule to detect when a staging script references a mart #6
e9828c7
georgim0 added a commit that referenced this issue Dec 22, 2020
@georgim0
add rule to find mart references to source #6
1be1cbe
georgim0 added a commit that referenced this issue Dec 22, 2020
@georgim0
add lineage rules to enforce data flow from source -> staging -> marts
9288ca5 
adding rules for:
- Staging scripts referencing a staging script in a different area
- Staging scripts referencing a mart
- Marts directly referencing a source

add docs and diagrams using the mermaid module

 issue #6
georgim0 added a commit that referenced this issue Dec 23, 2020
@georgim0
add rule to check if there is designated owner for physical models
6867d51 
* add rule to enforce the assignment of owner on physical models #6 
* modify all manifest input to properly assign source node definitions in the expected place in manifest.json

physical models are:

- sources
- views
- tables
- incrementals
@dorzey
Copy link
Contributor

dorzey commented Jan 20, 2021

@georgim0 where are you up to with this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@georgim0 georgim0

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dorzey @georgim0