-
Notifications
You must be signed in to change notification settings - Fork 14.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implemented MSGraphSensor as a deferrable sensor #39304
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…en JSON decode error occurs
boring-cyborg
bot
added
area:providers
provider:microsoft-azure
Azure-related issues
labels
Apr 29, 2024
…raphAsyncOperator
FYI: I've seen accross different PR that following test is sometimes randomly failing:
|
…yload could not have a response key if call isn't done
…he status key present
…he status key present
…alized by MSGraphTrigger
dabla
changed the title
Implement default response handler method and suppress JSONDecodeError
Implement default response handler and use deferable mechanism in MSGraphSensor
Apr 30, 2024
dabla
changed the title
Implement default response handler and use deferable mechanism in MSGraphSensor
Implemented MSGraphSensor as a deferrable sensor
Apr 30, 2024
eladkal
approved these changes
May 5, 2024
66 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implemented a default response handler which suppresses JSONDecodeError when response is empty or isn't of type json but the default response type json was specified, otherwise will return the contents and if that's empty the response headers. This can for example occur when triggering PowerBI dataset refreshes. Also added a test in TestResponseHandler and added example on how to use the operator to refresh a PowerBI dataset. This PR also removed the response_handler param as lambda expressions can't be serialized and triggers always get serialized to the database when deferred before getting executed. The most import change in thi PR is that the MSGraphSensor now also uses the deferable task mechanism like the MSGraphAsyncOperator, so it doesn't rely on the classic blocking poke method but defers the tasks in a 2 step. First it will execute the call in a deferred way and check the event, if the condition is met the sensor will stop. If not then the sensor will first defer a TimeDeltaTrigger with the specified retry_delay (default is 60 for MSGraphSensor ) and once the task completed it will redefer the poll task, that way the sensor doesn't block the workers.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.