Skip to content

Commit

Permalink
Merge pull request #7 from GeneralMills/not_null_constraint
Browse files Browse the repository at this point in the history
Add table constraints macro
  • Loading branch information
ohiosonia authored Nov 7, 2022
2 parents 06186cb + 456fdfd commit cf358f2
Show file tree
Hide file tree
Showing 3 changed files with 89 additions and 1 deletion.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ venv/
ENV/
env.bak/
venv.bak/
.devcontainer

# Spyder project settings
.spyderproject
Expand Down
41 changes: 40 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ To update this repo, you will need write access to the General Mills public repo
- [materialized_views](#materialized_views) [(source)](./macros/bigquery)
- [save_test_results](#save_test_results) [(source)](./macros/save_test_results.sql)
- uses macros [generate_schema_name](#generate_schema_name) [(source)](./macros/generate_schema_name.sql) and `get_full_model_name` [source](./macros/helpers/generate_schema_name.sql)
- [big_query_catalog_macro](#big_query_catalog_macro) [(source)](./macros/bq_catalog)
- [not_null_constraint](#not_null) [(source)](./macros/bigquery)


### Usage
Expand Down Expand Up @@ -74,7 +76,7 @@ select * from renamed
```

#### materialized_views
Materialized views are powerful but they can be costly, so please consult with the Analytics team if you are thinking about using them. This feature is also in beta with dbt. To use a materialized view, in the the `dbt_project.yml` file, add a `materialized_views` block in the models section, similar to this:
Materialized views are powerful but they can be costly, so please consult with the Analytics team if you are thinking about using them. This feature is also in _beta_ with dbt. To use a materialized view, in the the `dbt_project.yml` file, add a `materialized_views` block in the models section, similar to this:

output:
+materialized: table
Expand Down Expand Up @@ -111,3 +113,40 @@ you must specify the Big Query region where your data is housed. Add the followi
project variables, substituting the Big Query region of your project.
vars:
bq_region: 'region-US'



#### not_null constraint
The purpose of this macro is to require a column _not_ to be `NULLABLE`. (By default, if the mode is not specified BigQuery defaults to `NULLABLE`.)

This feature is also in _beta_ with dbt. More documentation from dbt is available [here](https://gist.github.com/sungchun12/f7ea081773ae824a83294649530d6e41).

To utilize this, you must add a `config` at the top of your model (`.sql` file), like this:

```sql
{{
config(
materialized = "table_with_constraints"
)
}}
```

Then in the .yml file, you _must_ specify every column. Each time fields are added or removed after this, the `yml` file must be updated, or the run will fail.
```yml
version: 2
models:
- name: table_constraints_demo
config:
has_constraints: true
columns:
- name: id
data_type: int64
description: I want to describe this one, but I don't want to list all the columns
meta:
constraint: not null
- name: color
data_type: string
- name: date_day
data_type: date
```
48 changes: 48 additions & 0 deletions macros/table_with_constraints.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
{%- materialization table_with_constraints, adapter='bigquery' -%}

--prepare database for new model
{%- set identifier = this.identifier -%}

{%- set old_relation = adapter.get_relation(database=database, schema=schema, identifier=identifier) -%}

-- drop original table if it exists
{% if old_relation is not none %}
{% do adapter.drop_relation(old_relation) %}
{% endif %}

--set the new target table
{%- set target_relation = api.Relation.create(
identifier=identifier, schema=schema, database=database,
type='table') -%}

--optional configs
{%- set raw_partition_by = config.get('partition_by', none) -%}
{%- set raw_cluster_by = config.get('cluster_by', none) -%}
{%- set sql_header = config.get('sql_header', none) -%}

{%- set partition_config = adapter.parse_partition_by(raw_partition_by) -%}

{{ sql_header if sql_header is not none }}
{%- set user_provided_columns = model['columns'] -%}

{% call statement('main') -%}
create or replace table {{ target_relation }}
{% if config.get('has_constraints', False) %}
(
{% for i in user_provided_columns %}
{% set col = user_provided_columns[i] %}
{{ col['name'] }} {{ col['data_type'] }} {{ col['meta']['constraint'] or "" }} {{ "," if not loop.last }}
{% endfor %}
)
{% endif %}
{{ partition_by(partition_config) }}
{{ cluster_by(raw_cluster_by) }}
{{ bigquery_table_options(config, model, temporary) }}
as (
{{ sql }}
);
{%- endcall %}

{{ return({'relations': [target_relation]}) }}

{%- endmaterialization -%}

0 comments on commit cf358f2

Please sign in to comment.