Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix DataprocSubmitJobOperator in deferrable mode=True when task is marked as failed. #39230

Merged
merged 6 commits into from
Apr 26, 2024

Conversation

sunank200
Copy link
Collaborator

When a user cancels a task for DataprocSubmitJobOperator in deferrable mode, the job should be cancelled in the same way as non-deferrable mode. This PR intends to fix this behaviour.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Apr 24, 2024
Copy link
Member

@pankajkoti pankajkoti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM mostly. Minor suggestions inline.

airflow/providers/google/cloud/triggers/dataproc.py Outdated Show resolved Hide resolved
airflow/providers/google/cloud/triggers/dataproc.py Outdated Show resolved Hide resolved
@sunank200 sunank200 force-pushed the dataproc_submit_job_on_kill branch from 54196de to d9b2f65 Compare April 24, 2024 15:36
@sunank200 sunank200 requested a review from pankajkoti April 24, 2024 15:40
@sunank200 sunank200 force-pushed the dataproc_submit_job_on_kill branch from d9b2f65 to 39d5f71 Compare April 25, 2024 06:37
@sunank200 sunank200 requested a review from pankajastro April 25, 2024 08:54
@sunank200 sunank200 force-pushed the dataproc_submit_job_on_kill branch from 39d5f71 to ab7e923 Compare April 25, 2024 08:59
@sunank200 sunank200 requested a review from Lee-W April 25, 2024 10:41
@sunank200 sunank200 force-pushed the dataproc_submit_job_on_kill branch from 6558fa7 to 59a1d73 Compare April 25, 2024 16:15
@sunank200 sunank200 force-pushed the dataproc_submit_job_on_kill branch from 59a1d73 to fc6f54d Compare April 26, 2024 07:01
Copy link
Member

@Lee-W Lee-W left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, it looks good to me. Left one minor discussion.

airflow/providers/google/cloud/triggers/dataproc.py Outdated Show resolved Hide resolved
@sunank200 sunank200 requested a review from Lee-W April 26, 2024 12:10
@pankajkoti pankajkoti merged commit 2a913b6 into apache:main Apr 26, 2024
40 checks passed
@pankajkoti pankajkoti deleted the dataproc_submit_job_on_kill branch April 26, 2024 12:34
@pankajkoti
Copy link
Member

We will need to improvise here based on #36090 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants