Add Robust Support for Callbacks at Task and TaskGroup Level

[jroach-astronomer]

Pull request to address #253, which aims to provide enhanced support for callback types at the DAG, default_args, Task and TaskGroup level. There are now four different ways that callbacks can be defined via dag-factory. Below, you'll also see an example of each.

1. By passing a string that points to a callable.
2. By passing a file path and file name that points to a callable, similar to above.
3. By providing a string that points to a callable, but with some parameters being provided at runtime.
4. Using callbacks from providers (like Slack).

Passing a string that points to a callable

...
  task_groups:
    task_group_1:
      default_args:
        on_success_callback: print_hello.print_hello_from_callback
      dependencies: [task_1, task_2]
...

Passing a file path and file name that points to a callable

...
    task_2:
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: "echo 2"
      on_success_callback_name: print_hello_from_callback
      on_success_callback_file: $CONFIG_ROOT_DIR/print_hello.py
      dependencies: [start]
...

By providing a string that points to a callable, but with some parameters being provided at runtime

...
  on_failure_callback:
    callback: customized.callbacks.custom_callbacks.output_message
    param1: param1
    param2: param2
...

Using callbacks from providers

...
    on_failure_callback:
      callback: airflow.providers.slack.notifications.slack.send_slack_notification
      slack_conn_id: slack_conn_id
      text: |
        :red_circle: Task Failed.
        This task has failed and needs to be addressed.
        Please remediate this issue ASAP.
      channel: "#channel"
...

Unit-tests have been added and updated appropriately.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 93.10345% with 2 lines in your changes missing coverage. Please review.

Project coverage is 93.69%. Comparing base (442c824) to head (317b35e).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
dagfactory/dagbuilder.py	93.10%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #322   +/-   ##
=======================================
  Coverage   93.68%   93.69%           
=======================================
  Files          10       10           
  Lines         792      777   -15     
=======================================
- Hits          742      728   -14     
+ Misses         50       49    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jroach-astronomer]

@pankajastro, do you mind reviewing this, open to feedback?

cc: @tatiana

[pankajkoti]

@jroach-astronomer please let me know once you'd like me to re-review/test the pull request. Could you please also update the PR description after the changes especially with the example for on_success_callback: $CONFIG_ROOT_DIR.print_hello.print_hello_from_callback

There seem to be a couple of conflicts, would be nice if you could resolve them too. We're planning on releasing DAG Factory on 10th Jan, so would be nice if we could get this change in by then.

[jroach-astronomer]

@pankajkoti, resolved MC's, updated PR docs.

[pankajkoti]

This looks fantastic! Thank you for all the refactoring and for addressing my earlier questions and comments. I’ve shared an in-line thought in #322 (comment) about potentially dropping support for the file+name approach of specifying callbacks. It would be great to revisit this idea later, and we can plan to work on it in the next release since it’s outside the scope of this PR.

[pankajkoti]

I’m wondering if we should continue supporting the {callback_type}_name + {callback_type}_file approach for callbacks. Are there scenarios where simply specifying {callback_type} might be insufficient or lack necessary support?

Unifying the approach to a single, straightforward method seems more maintainable and less confusing for users. Additionally, what happens if a user specifies both on_success_callback and on_success_callback_name + on_success_callback_file? Currently, it appears that on_success_callback_name + on_success_callback_file will override the on_success_callback, which isn’t transparent to the end user and could cause confusion.

While this isn’t part of your changes and has been a pre-existing issue, it might be worth discussing whether we could drop support for the name+file approach altogether or at least an error when users try to provide callbacks using both approaches. What do you think?