Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat!: add model blueprinting #3848

Merged
merged 5 commits into from
Feb 18, 2025
Merged

Feat!: add model blueprinting #3848

merged 5 commits into from
Feb 18, 2025

Conversation

georgesittas
Copy link
Contributor

@georgesittas georgesittas commented Feb 15, 2025

This PR adds a new feature: the ability to implement model blueprints based on a single template.

This feature is useful when multiple models need to share the same logic, but parameterized differently. Today, this is generally possible by either duplicating model definitions, thus violating the DRY principle, or creating a custom loader, which involves knowing the inner workings of the codebase.

A simple example that demonstrates the syntax to achieve model blueprinting is shown below:

MODEL (
  name @blueprint.some_table,
  blueprints (
    (blueprint := customer1),
    (blueprint := customer2),
    (blueprint := customer3),
  ),
);

SELECT
  *
FROM @blueprint.some_source

This definition essentially expands into three different models, each one sharing the same logic but using a different value for the @blueprint variable. A blueprint can be further customized by specifying additional variables:

MODEL (
  name @customer.some_table,
  blueprints (
    (customer := customer1, field_a := x, field_b := y),
    (customer := customer2, field_a := z, field_b := w),
  )
);

SELECT
  @field_a,
  @field_b
FROM @customer.some_source

Similarly, this will expand into two different models, where each one will contain the variables specified in the corresponding blueprints tuple. For example, the first model would look like:

MODEL (
  name customer1.some_table,
);

SELECT
  x,
  y
FROM customer1.some_source

The blueprint variable mappings can be combined with custom macros to implement even more complicated patterns, e.g., where the model logic itself changes completely depending on a variable's value.

@georgesittas georgesittas requested review from izeigerman, tobymao and a team February 15, 2025 01:09
@georgesittas georgesittas changed the title Feat: add model blueprinting Feat!: add model blueprinting Feb 15, 2025
@georgesittas georgesittas force-pushed the jo/model_blueprinting branch 4 times, most recently from de99e43 to dac9c19 Compare February 18, 2025 00:58
@georgesittas georgesittas requested review from izeigerman, tobymao and a team February 18, 2025 11:58
@georgesittas georgesittas requested a review from tobymao February 18, 2025 18:37
@georgesittas georgesittas merged commit e9b3441 into main Feb 18, 2025
21 checks passed
@georgesittas georgesittas deleted the jo/model_blueprinting branch February 18, 2025 20:49
justinjoseph89 added a commit to trygforsikring/sqlmesh that referenced this pull request Feb 20, 2025
* Chore: replace mysql-connector-python with pymysql (TobikoData#3788)

* feat: don't force db connect if using serverless (TobikoData#3786)

* Feat: Add 'auditing' and 'promoting' runtime stages (TobikoData#3791)

* fix: databricks set default catalog for both connections (TobikoData#3793)

* Docs: Prod Env Observability updates (TobikoData#3790)

Co-authored-by: Trey Spiller <[email protected]>
Co-authored-by: Trey Spiller <[email protected]>

* Docs: update jinja gateway variable syntax (TobikoData#3795)

* Fix: Only promote all snapshots if the target environment expired (TobikoData#3797)

* Chore: remove [missing dates] from CLI model backfills title (TobikoData#3796)

* Fix: Make sure that physical tables exist for promoted snapshots (TobikoData#3798)

* fix: db properly support `with_log_level` (TobikoData#3799)

* fix: run integration tests in session (TobikoData#3801)

* Fix: specify init duckdb database so quickstart works (TobikoData#3800)

* Docs: clarify in faq that run ignores local definitions  (TobikoData#3794)

* feat: add airflow operator and hook for ClickHouse (TobikoData#3699)

* Feat: Allow macros in python model properties (TobikoData#3740)

* Docs: update CLI quickstart's CLI output (TobikoData#3802)

* Fix: Ensure diff sample displays when table names have been uppercased (TobikoData#3806)

* Fix: update github links for pdoc api docs (TobikoData#3807)

* Feat: Run audits as non-blocking on dev previews (TobikoData#3809)

* Chore!: bump sqlglot to v26.6.0 (TobikoData#3810)

* Feat!: Adjust physical_properties evaluation and add macro to resolve physical table names (TobikoData#3772)

* Chore: Use 'dev' suffix instead of 'temp' for non-deployable physical tables (TobikoData#3803)

* Fix!: Propagate the grain attribute when converting dbt models (TobikoData#3804)

* Docs(dagster): Fix installation instructions (TobikoData#3812)

* fix: add kwargs to build_table_properties_exp (TobikoData#3817)

* Feat: include alter statements in destructive change error message (TobikoData#3805)

* Dagster demo and tutorial video (TobikoData#3822)

* fix: databricks with_log_level (TobikoData#3823)

* Feat: improve audit error message formatting (TobikoData#3818)

* Fix(postgres): Quote role names if required when running SET ROLE on cursor init (TobikoData#3825)

* Fix: handle quoted projects properly in bigquery adapter (TobikoData#3820)

* Chore: Fix flaky test (TobikoData#3828)

* docs: adding self hosted executor docs (TobikoData#3816)

* Revert "Fix(postgres): Quote role names if required when running SET ROLE on cursor init" (TobikoData#3834)

* Fix: Don't fail because of an unrestorable change if the target model is forward-only (TobikoData#3835)

* Chore: improve metadata update console printing (TobikoData#3824)

* fix: respect disable_restatement remove intervals across env (TobikoData#3838)

* fix: respect disable restate dev unpaused snapshots (TobikoData#3840)

* Feat: add [WARNING] to console warning messages (TobikoData#3826)

* Feat: Extend support of project wide model properties (TobikoData#3832)

* Fix: Streamline execution of pre- / post- statements when creating a physical table (TobikoData#3837)

* fix: signals that return an empty list are considered ready (TobikoData#3841)

* Chore: Add the ingress section to the self-hosted executor docs

* Fix: Snapshots promoted in prod shouldn't be restated in dev (TobikoData#3843)

* Fix: Inference of python model names from the file system (TobikoData#3844)

* feat: add support for datetime/date in macros (TobikoData#3846)

* fix: only expand restatement range if incremental (TobikoData#3847)

* feat: improve gcp postgres connection config options (TobikoData#3842)

* Fix: Pin PyGithub to 2.5.0 so tests can run (TobikoData#3851)

* Fix: Warn when SQLMesh automatically adjusts a restatement range to cover the whole model (TobikoData#3850)

* fix!: normalize catalog override name (TobikoData#3849)

* Feat: allow different warning messages for logger and console (TobikoData#3836)

* Chore: fix audit doc typos (TobikoData#3856)

* Add airflow tutorial video (TobikoData#3860)

* Feat!: add model blueprinting (TobikoData#3848)

* Fix: Make sure that pending restatement intervals are always recorded last during compaction (TobikoData#3862)

* Chore: Break up the plan_builder method in Context (TobikoData#3867)

* fix: give better error message when object not serializable (TobikoData#3861)

* Feat: make date_spine macro less strict to allow dynamic behavior (TobikoData#3865)

* Add Tcloud SSO docs (TobikoData#3827)

* Fix: Unexpected backfill of a parent of a changed forward-only child when the child runs before the parent (TobikoData#3871)

* Feat: Allow CustomKind subclasses for custom materializations (TobikoData#3863)

* Chore: consolidate `make install-*` (TobikoData#3874)

---------

Co-authored-by: Trey Spiller <[email protected]>
Co-authored-by: Ryan Eakman <[email protected]>
Co-authored-by: Iaroslav Zeigerman <[email protected]>
Co-authored-by: Marisa Smith <[email protected]>
Co-authored-by: Trey Spiller <[email protected]>
Co-authored-by: Anton Parfenyuk <[email protected]>
Co-authored-by: Themis Valtinos <[email protected]>
Co-authored-by: Gerasimos Kounadis <[email protected]>
Co-authored-by: Erin Drummond <[email protected]>
Co-authored-by: Jo <[email protected]>
Co-authored-by: Sung Won Chung <[email protected]>
Co-authored-by: Ben <[email protected]>
Co-authored-by: Philippe Laflamme <[email protected]>
Co-authored-by: Toby Mao <[email protected]>
Co-authored-by: Afzal Jasani <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants