[CT-2317] invocation_id is cached between invocations in same Python process

[jtcohen6]

Originally reported by @jeremyyeo:

With dbt >= 1.4, we seem to be maintaining invocation_id across steps in a single run.

Starting in v1.4, invocation_id is now stored on EventManager:

dbt-core/core/dbt/events/eventmgr.py

Line 185 in 65fcbe7

self.invocation_id: str = str(uuid4())

I think we should clear or reset invocation_id as part of the steps in cleanup_event_logger:

dbt-core/core/dbt/events/functions.py

Lines 269 to 272 in 0ef9931

	def set_invocation_id() -> None:
	# This is primarily for setting the invocation_id for separate
	# commands in the dbt servers. It shouldn't be necessary for the CLI.
	EVENT_MANAGER.invocation_id = str(uuid.uuid4())

dbt-core/core/dbt/events/functions.py

Lines 160 to 165 in 0ef9931

	def cleanup_event_logger():
	# Reset to a no-op manager to release streams associated with logs. This is
	# especially important for tests, since pytest replaces the stdout stream
	# during test runs, and closes the stream after the test is over.
	EVENT_MANAGER.loggers.clear()
	EVENT_MANAGER.callbacks.clear()

Reproduction case

• Create an on-run-start hook with {{ log(invocation_id, info = true) }}
• Create a dbt Cloud job with multiple run steps
• See if the invocation IDs are the same or different

[jtcohen6]

We've confirmed that this is happening when multiple dbt-core commands are called from within the same Python process.

@peterallenwebb's recommendation:

I do think calling set_invocation_id() between in-process dbt calls will solve the issue.

Also, to fix this issue going forward, and guarantee that programmatic invocations (dbtRunner.invoke()) have distinct values of invocation_id, we should either:

• Add a call to set_invocation_id() within setup_event_logger()
• Call set_invocation_id() within the preflight decorator, right before/after calling setup_event_logger()
• (Think about whether EventManager is the right place for invocation_id() to exist long-term)

[peterallenwebb]

The approach I took was to call set_invocation_id() within preflight. The unit tests show this approach will work for programmatic invocations of dbt through the official API.