[FEAT] raise exception with main notebook error in DatabricksRunNowDeferrableOperator

[gaurav7261]

1. Implement async method for getting job run output
2. logic for getting main notebook exception via async api calling and raising the exception

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[gaurav7261]

@Lee-W changes are done, please review and let me know

[gaurav7261]

Hi @Lee-W , is there any issue with PR

[Lee-W]

LGTM

[gaurav7261]

Hi @pankajkoti , can you please review

[Lee-W]

I think we're good to merge this one. Just want to see whether @pankajkoti have some time to double check if I missed anything.

I'm going to merge this later today. Please let me know if anyone want to take a deeper look. 🙂

[pankajkoti]

Can you please update the PR title and description by taking hints from https://cbea.ms/git-commit/ or something similar to mention what and why we're doing.

Also the would be nice in general for PR title to be in imperative mood like mentioned in https://cbea.ms/git-commit/#imperative

[Lee-W]

Left one nitpick. But I'm good with this PR now 👍 @pankajkoti could you please take another look? Thanks!

[pankajkoti]

I added my concern here #39110 (comment). But leave the decision to you @Lee-W :)

[gaurav7261]

@Lee-W @pankajkoti please review, deferrable changes done as per review

[pankajkoti]

LGTM

[gaurav7261]

Hi @Lee-W @pankajkoti , raised PR for runnow operator separately as discussed, please review #39354

[Lee-W]

Hi @Lee-W @pankajkoti , raised PR for runnow operator separately as discussed, please review #39354

Thanks for your prompt implementation! Will look into it today