Feature: More granular on-start and on-end hooks

[djbelknapdbs]

Describe the feature

Each dbt command should have its own set of on-[command]-start and on-[command]-end hooks, rather than just on-run-start and on-run-end that aren't specific to the run command. For example, on-test-start would run at the start of dbt test, on-source-start would run at the start of dbt source snapshot-freshness, etc.

Describe alternatives you've considered

I can run a second command right after the first one

Additional context

I'm opening this now because I'm trying to incorporate the dbt_artifacts package, and it would be really handy to be able to trigger it in the on-[command]-end hook. Now that I'm typing this, I'm not positive the artifacts would be saved out during the hook so maybe it wouldn't work anyways, but I still think the feature is valuable. I had an earlier use case where I wanted to run a command on-test-start and on-test-end which I've forgotten, but I think it had to do with adjusting Snowflake warehouse usage.

Who will this benefit?

Developers and admins who want more ability to trigger events or processes related to specific dbt commands.

Are you interested in contributing this feature?

This is probably beyond me

[jtcohen6]

@djbelknapdbs Thanks for opening!

I think, rather than a different on-[command]-end hook for each command type, I'd rather just have:

• All tasks observe the on-run-start and on-run-end hooks. (It's really just dbt test that ignores them today, and I don't think this makes a lot of sense.)
• We expose which arguments were passed to the current task to the Jinja context, via a flags or args context variable, as proposed in Add a "full run" indicator to Jinja context #2253 (comment). Then, you have full control in your on-run-x hook (or the macro it calls) to add conditional checks based on whether the command is dbt run, dbt test, dbt source snapshot-freshness, etc.

What do you think?

Now that I'm typing this, I'm not positive the artifacts would be saved out during the hook so maybe it wouldn't work anyways, but I still think the feature is valuable.

I think that's right, so this change wouldn't work for the particular use case. As a general rule, though, I think the feature would be valuable! There are some context variables, such as schemas and results, that are only available in the on-run-end context.

[jtcohen6]

The actual change to the test task to enable on-run-start + on-run-end hooks is tremendously straightforward, it just looks like removing this method override:
https://github.com/dbt-labs/dbt/blob/650b34ae246b4c60cde395fcb123418c7807f9aa/core/dbt/task/test.py#L170-L174

The trickier change would be the complementary addition of a flags/args context variable, enabling users to switch on/off a particular project hook based on the current task type. I don't think that's a blocker to making this switch, it's just a good idea that will make on-run-x hooks even more powerful/configurable.

Worth saying that the dbt build command, new in v0.21, is going to execute tests and execute on-run-x hooks.