Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store relation name in manifest's node and source objects #2837

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f638a3d
Store relation name in manifest's node object
Oct 16, 2020
01331ed
Update CHANGELOG.md
Oct 16, 2020
a0370a6
Add relation_name to node object in docs generation tests
Oct 20, 2020
244d5d2
Merge remote-tracking branch 'upstream/dev/kiyoshi-kuromiya' into dev…
Oct 20, 2020
a8809ba
Merge branch 'dev/kiyoshi-kuromiya' into feature/2647-relation-name-i…
Oct 20, 2020
40370e1
Fix wrong schema name in test and add missing relation_name in node
Oct 20, 2020
7ee78e8
Add missing relation_name fields in doc generation test manifests
Oct 20, 2020
c3bf0f8
Add relation_name to missing tests in test_docs_generate
Oct 25, 2020
b079545
Adapt relation_name for Bigquery and Snowflake in docs generation tests
Oct 25, 2020
c9e01bc
Fix quotes in relation name for Bigquery docs generate tests
Oct 25, 2020
09c37f5
Adapt relation_name to expected_run_results parameters
Oct 25, 2020
900298b
Fix database name in relation_name in expected_run_results
Oct 25, 2020
4203985
Adapt expected_seeded_manifest method to Snowflake identifier quoting
Oct 25, 2020
e1097f1
Define relation_name only for non-ephemeral models, seeds and snapshots
Oct 27, 2020
92cedf8
Fix Flake8 style issue
Oct 27, 2020
52ed4aa
Fix tests which are missing snapshot nodes
Oct 27, 2020
7115d86
Modify snapshot path for docs generation tests
Oct 27, 2020
a9901c4
Disable snapshot documentation testing for Redshift and Bigquery
Oct 27, 2020
6251d19
Use is_ephemeral_model property instead of config.materialized
franloza Oct 28, 2020
784616e
Add relation name to source object in manifest
Oct 28, 2020
3e5d901
Add snapshot to additional Redshift and Bigquery manifest tests
Oct 28, 2020
21fd75b
Fix parent_map object in tests
Oct 28, 2020
852990e
Fix child_map in tests
Oct 28, 2020
b741679
Add missing key to child map in expected_bigquery_complex_manifest
Oct 29, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
- Save cli and rpc arguments in run_results.json ([#2510](https://github.com/fishtown-analytics/dbt/issues/2510), [#2813](https://github.com/fishtown-analytics/dbt/pull/2813))
- Added support for BigQuery connections using refresh tokens ([#2344](https://github.com/fishtown-analytics/dbt/issues/2344), [#2805](https://github.com/fishtown-analytics/dbt/pull/2805))
- Remove injected_sql from manifest nodes ([#2762](https://github.com/fishtown-analytics/dbt/issues/2762), [#2834](https://github.com/fishtown-analytics/dbt/pull/2834))
- Store resolved node names in manifest ([#2647](https://github.com/fishtown-analytics/dbt/issues/2647), [#2837](https://github.com/fishtown-analytics/dbt/pull/2837))

### Under the hood
- Added strategy-specific validation to improve the relevancy of compilation errors for the `timestamp` and `check` snapshot strategies. (([#2787](https://github.com/fishtown-analytics/dbt/issues/2787), [#2791](https://github.com/fishtown-analytics/dbt/pull/2791))
Expand All @@ -32,6 +33,7 @@ Contributors:
- [@zmac12](https://github.com/zmac12) ([#2871](https://github.com/fishtown-analytics/dbt/pull/2817))
- [@Mr-Nobody99](https://github.com/Mr-Nobody99) ([docs#138](https://github.com/fishtown-analytics/dbt-docs/pull/138))
- [@jplynch77](https://github.com/jplynch77) ([docs#139](https://github.com/fishtown-analytics/dbt-docs/pull/139))
- [@franloza](https://github.com/franloza) ([#2837](https://github.com/fishtown-analytics/dbt/pull/2837))


## dbt 0.18.1 (October 13, 2020)
Expand Down
11 changes: 11 additions & 0 deletions core/dbt/compilation.py
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,15 @@ def add_ephemeral_prefix(self, name: str):
relation_cls = adapter.Relation
return relation_cls.add_ephemeral_prefix(name)

def _get_relation_name(self, node: ParsedNode):
relation_name = None
if (node.resource_type in NodeType.refable() and
not node.is_ephemeral_model):
Comment on lines +177 to +178
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

adapter = get_adapter(self.config)
relation_cls = adapter.Relation
relation_name = str(relation_cls.create_from(self.config, node))
return relation_name

def _inject_ctes_into_sql(self, sql: str, ctes: List[InjectedCTE]) -> str:
"""
`ctes` is a list of InjectedCTEs like:
Expand Down Expand Up @@ -395,6 +404,8 @@ def _compile_node(
node,
)

compiled_node.relation_name = self._get_relation_name(node)

compiled_node.compiled = True

# add ctes for specific test nodes, and also for
Expand Down
1 change: 1 addition & 0 deletions core/dbt/contracts/graph/compiled.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ class CompiledNode(ParsedNode, CompiledNodeMixin):
compiled_sql: Optional[str] = None
extra_ctes_injected: bool = False
extra_ctes: List[InjectedCTE] = field(default_factory=list)
relation_name: Optional[str] = None

def set_cte(self, cte_id: str, sql: str):
"""This is the equivalent of what self.extra_ctes[cte_id] = sql would
Expand Down
1 change: 1 addition & 0 deletions core/dbt/contracts/graph/parsed.py
Original file line number Diff line number Diff line change
Expand Up @@ -555,6 +555,7 @@ class ParsedSourceDefinition(
config: SourceConfig = field(default_factory=SourceConfig)
patch_path: Optional[Path] = None
unrendered_config: Dict[str, Any] = field(default_factory=dict)
relation_name: Optional[str] = None

def same_database_representation(
self, other: 'ParsedSourceDefinition'
Expand Down
12 changes: 11 additions & 1 deletion core/dbt/parser/schemas.py
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,11 @@ def _generate_source_config(self, fqn: List[str], rendered: bool):
base=False,
)

def _get_relation_name(self, node: ParsedSourceDefinition):
adapter = get_adapter(self.root_project)
relation_cls = adapter.Relation
return str(relation_cls.create_from(self.root_project, node))

def parse_source(
self, target: UnpatchedSourceDefinition
) -> ParsedSourceDefinition:
Expand Down Expand Up @@ -302,7 +307,7 @@ def parse_source(

default_database = self.root_project.credentials.database

return ParsedSourceDefinition(
parsed_source = ParsedSourceDefinition(
package_name=target.package_name,
database=(source.database or default_database),
schema=(source.schema or source.name),
Expand Down Expand Up @@ -330,6 +335,11 @@ def parse_source(
unrendered_config=unrendered_config,
)

# relation name is added after instantiation because the adapter does
# not provide the relation name for a UnpatchedSourceDefinition object
parsed_source.relation_name = self._get_relation_name(parsed_source)
return parsed_source

def create_test_node(
self,
target: Union[UnpatchedSourceDefinition, UnparsedNodeUpdate],
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{% snapshot snapshot_seed %}
{{
config(
unique_key='id',
strategy='check',
check_cols='all',
target_schema=var('alternate_schema')
)
}}
select * from {{ ref('seed') }}
{% endsnapshot %}
Loading