Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add relation info to node_info in logging #7304

Merged
merged 4 commits into from
Apr 10, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changes/unreleased/Features-20230410-174824.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
kind: Features
body: Add relation info (database, schema, alias) to node_info dictionary in structured
logging
time: 2023-04-10T17:48:24.870944+02:00
custom:
Author: jtcohen6
Issue: "6724"
6 changes: 6 additions & 0 deletions core/dbt/contracts/graph/nodes.py
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,12 @@ def node_info(self):
"node_started_at": self._event_status.get("started_at"),
"node_finished_at": self._event_status.get("finished_at"),
"meta": getattr(self, "meta", {}),
"node_relation": {
"database": getattr(self, "database", None),
"schema": getattr(self, "schema", None),
"alias": getattr(self, "alias", None),
"relation_name": getattr(self, "relation_name", None),
},
}
return node_info

Expand Down
8 changes: 3 additions & 5 deletions core/dbt/events/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,6 @@ logger = AdapterLogger("<database name>")

## Compiling types.proto

After adding a new message in types.proto, execute Makefile target:

make proto_types in the repository root directory, or
`protoc -I=. --python_out=. types.proto`
in the core/dbt/events directory
After adding a new message in `types.proto`, either:
- In the repository root directory: `make proto_types`
- In the `core/dbt/events` directory: `protoc -I=. --python_out=. types.proto`
9 changes: 9 additions & 0 deletions core/dbt/events/types.proto
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,14 @@ message TimingInfoMsg {
google.protobuf.Timestamp completed_at = 3;
}

// NodeRelation
message NodeRelation {
string database = 10;
string schema = 11;
string alias = 12;
string relation_name = 13;
}

// NodeInfo
message NodeInfo {
string node_path = 1;
Expand All @@ -37,6 +45,7 @@ message NodeInfo {
string node_started_at = 7;
string node_finished_at = 8;
google.protobuf.Struct meta = 9;
NodeRelation node_relation = 10;
}

// RunResult
Expand Down
1,648 changes: 825 additions & 823 deletions core/dbt/events/types_pb2.py

Large diffs are not rendered by default.

17 changes: 17 additions & 0 deletions tests/unit/test_proto_events.py
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,12 @@ def test_node_info_events():
"node_started_at": "some_time",
"node_finished_at": "another_time",
"meta": meta_dict,
"node_relation": {
"database": "some_database",
"schema": "some_schema",
"alias": "some_alias",
"relation_name": "some.relation.name",
},
}
event = LogStartLine(
description="some description",
Expand All @@ -132,6 +138,17 @@ def test_node_info_events():
assert event.node_info.node_path == "some_path"
assert event.to_dict()["node_info"]["meta"] == meta_dict

node_info["node_relation"]["database"] = None
node_info["node_relation"]["relation_name"] = "some_schema.some_alias"
event = LogStartLine(
description="some description",
index=123,
total=111,
node_info=node_info,
)
assert event.node_info.node_relation.database == ""
assert event.node_info.node_relation.relation_name == "some_schema.some_alias"


def test_extra_dict_on_event(monkeypatch):

Expand Down