[CT-1635] Populate adapter_response for tests + sources

[jtcohen6]

Describe the feature

#2747 requested that dbt log bytes processed for all query types, including tests, source-freshness, and run-operations. This prompted the first part of a solution in #2961: adding an adapter_response dict to record database-specific information about the queries dbt has run. In v0.19.0, that will include bytes_processed for all materializations (model runs, seeds, snapshots).

We'd like to record bytes processed for tests and source snapshots, too. What does this really mean?

• dbt test should populate adapter_response in run_results.json
• dbt source snapshot-freshness should populate adapter_response in sources.json

(I'm less clear on how we'd manage this in run-operation, since it doesn't write to run_results.)

Eventually, this has the potential to dovetail nicely with:

• #2079: add a new log line to the "summary" stdout, with the total bytes processed from the invocation. In an ideal world, users could get a human-friendly rounded number in stdout and the raw numbers in run_results.json for analysis/aggregation.
• Configurable node status in log output #2580: the ability to define which result information is populated to message in stdout (maybe via Jinja macro??)

Describe alternatives you've considered

• Not doing this

Additional context

• Most immediately relevant to BigQuery (bytes processed)
• Useful for any database which sends back significant information in its response

Who will this benefit?

• Projects with larger datasets, where filtering is important to limit cost of test queries

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[jtcohen6]

I think this may be as simple as using result.adapter_response to populate the RunResult for tests + freshness results, where currently we're setting them to just {}:

https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/include/global_project/macros/materializations/models/table/create_table_as.sql

dbt-core/core/dbt/task/freshness.py

Line 123 in 75f3e8c

adapter_response={},

We can look to the run task for the reference implementation:

dbt-core/core/dbt/task/run.py

Lines 221 to 223 in 75f3e8c

	adapter_response = {}
	if isinstance(result.response, dbtClassMixin):
	adapter_response = result.response.to_dict(omit_none=True)

[jtcohen6]

One thing we'd need to change to get this working for dbt source freshness: Currently, the collect_freshness macro returns just its table result. We need it to return the response object as well, which we can then serialize into an adapter_response dictionary.

To illustrate the changes that I think would be needed: a066c1e

[sebas-bdr]

hey @jtcohen6, this feature would be great.
I noticed the help_wanted label was recently removed, not sure (from reading the docs) whether this means contributions won't be accepted. But if possible I'd like to contribute.

To illustrate the changes that I think would be needed: a066c1e

Regarding testing, would the following code change suffice (specifically for source freshness)? sebas-bdr@572f143

[jamwalla]

I would love to have this feature to see how much my tests are costing!

I was able to get it working pretty easily for tests by returning the adapter response alongside the TestResultData object in TestRunner.execute_test:

dbt-core/core/dbt/task/test.py

Line 94 in 89d111a

def execute_test(self, test: TestNode, manifest: Manifest) -> TestResultData:

Same way that the ModelRunner does it:

dbt-core/core/dbt/task/run.py

Line 219 in 89d111a

adapter_response = result.response.to_dict(omit_none=True)

Then the adapter response can get passed up through TestRunner.execute.

Maybe not the prettiest solution but wanted to advocate for this being very useful + straightforward! thx dbt devs <3