[CT-475] Fail-fast writes no run_results.json on error

[jtcohen6]

dbt run --fail-fast does not write run_results.json if it encounters any error, for any model node, during its run.

I know --fail-fast is common in some development + CI setups. Metadata in those environments is lower-stakes than metadata from prod, but I believe it still has value.

We need to decide on and document the expected behavior. Perhaps it's right to consider --fail-fast like a keyboard interrupt mid-run, such that the invocation exits without completing.

This will also be less pressing in a future where we've instrumented dbt with a streaming event system, because a single JSON file at the end of a batch process will no longer be our only principal source of metadata.

[jtcohen6]

It would be desirable to write run_results.json for runs that are cancelled partway through.

Where we write those artifacts them currently, within the run step:

dbt-core/core/dbt/task/runnable.py

Lines 466 to 468 in 3a3bedc

	if flags.WRITE_JSON:
	self.write_manifest()
	self.write_result(result)

That result is the returned value from execute_with_hooks, which calls (in turn) execute_nodes. That's the method that both collect results and handles cancellation due to --fail-fast/KeyboardInterrupt:

dbt-core/core/dbt/task/runnable.py

Lines 383 to 391 in 3a3bedc

	except KeyboardInterrupt:
	self._cancel_connections(pool)
	print_run_end_messages(self.node_results, keyboard_interrupt=True)
	raise
	pool.close()
	pool.join()
	return self.node_results

Is there a way we can replumb this to write partially complete results? I think the right next step might be to spike the difficulty of that replumbing.

[iknox-fa]

I think the right next step might be to spike the difficulty of that replumbing.

In BLG we all agreed. See: #5204

[iknox-fa]

No estimate 5/2 /22 BLG, awaiting spike

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.

[kieronellis]

Having errors included in the run_results.json when there's a fail fast would be super helpful for my team as we are using fail fast in our CI setup - Is there any chance of this being implemented? Thanks.

[jtcohen6]

Discussing with @ChenyuLInx, an answer that doesn't feel so terrible: write run_results.json after every node completes. That way, we don't need to gracefully handle graceless exits (whether due to --fail-fast, keyboard interrupt, process termination...)

[jtcohen6]

Closing in favor of #7302

Update: Reopening per #7302 (comment)

[iknox-fa]

Closed per #7302 (comment)