Improve DataprocCreateClusterOperator in Triggers for Enhanced Error Handling and Resource Cleanup #39130
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:providers
provider:google
sunank200
force-pushed
the
google_dataproc_create_cluster_fix
branch
5 times, most recently
from
75bff4f
to
51eef81
Compare
sunank200
force-pushed
the
google_dataproc_create_cluster_fix
branch
from
51eef81
to
c133c83
Compare
sunank200
requested review from
Lee-W,
pankajkoti,
phanikumv,
pankajastro and
vatsrahul1001
sunank200
force-pushed
the
google_dataproc_create_cluster_fix
branch
from
c133c83
to
443e9f8
Compare
pankajkoti
reviewed
Apr 22, 2024
Lee-W
reviewed
Apr 23, 2024
sunank200
force-pushed
the
google_dataproc_create_cluster_fix
branch
from
7b5e114
to
ee85f59
Compare
pankajkoti
reviewed
Apr 23, 2024
sunank200
force-pushed
the
google_dataproc_create_cluster_fix
branch
2 times, most recently
from
18598b3
to
55e50d3
Compare
pankajkoti
reviewed
Apr 23, 2024
|
|
||
| self.log.info("Sleeping for %s seconds.", self.polling_interval_seconds) | ||
| await asyncio.sleep(self.polling_interval_seconds) | ||
| except asyncio.CancelledError: |
There was a problem hiding this comment.
Would like to understand when exactly is the CancelledError raised.
There was a problem hiding this comment.
This is raised when the user marks the task as failed.
There was a problem hiding this comment.
Looks like there are other cases too when CancelledError is raised e.g. when the trigger restarts as mentioned in #36090 (comment)
We might need some additional measures here.
sunank200
force-pushed
the
google_dataproc_create_cluster_fix
branch
from
55e50d3
to
63bfab1
Compare
sunank200
force-pushed
the
google_dataproc_create_cluster_fix
branch
4 times, most recently
from
5fa0f7a
to
86a43db
Compare
pankajkoti
reviewed
Apr 24, 2024
sunank200
force-pushed
the
google_dataproc_create_cluster_fix
branch
from
7de3c9f
to
c47de1f
Compare
pankajkoti
approved these changes
Apr 24, 2024
sunank200
force-pushed
the
google_dataproc_create_cluster_fix
branch
from
c47de1f
to
b261f33
Compare
pankajastro
reviewed
Apr 25, 2024
Lee-W
reviewed
Apr 25, 2024
vatsrahul1001
approved these changes
Apr 25, 2024
sunank200
force-pushed
the
google_dataproc_create_cluster_fix
branch
2 times, most recently
from
e0cacb5
to
e28fff9
Compare
sunank200
force-pushed
the
google_dataproc_create_cluster_fix
branch
from
e28fff9
to
36aa2f8
Compare
2 tasks
66 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces improvements to the
DataprocCreateClusterOperator, specifically addressing deficiencies in handling clusters that are created in an ERROR state when operating in deferrable modePreviously, when the
DataprocCreateClusterOperatorwas operating in deferrable mode, it did not correctly handle scenarios where clusters were initially created in an ERROR state. The lack of appropriate error handling and resource cleanup led to subsequent retries encountering these errored clusters, attempting deletion, and resulting in pipeline failures.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.