Fix Cluster Activity historical metrics for ongoing dagrun if end_date is provided #33488
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We were selecting DagRuns/Tasks that were either:
Issue, if the end_date is chosen in the past, runs without end_date (ongoing run) will be counted in the metrics, which is wrong.
This handle such cases i.e when
dagrun.end_date is None
:before
An old range such as this one would be updated with data regarding ongoing runs (i.e end_date is None)
After
This will not be the case anymore, data will remain stable on old ranges, because ongoing run will be taken out. In this example there is no records on this period.
To still be able to get data from ongoing run in the query we need to provide an
end_date
that is greater than 'now' by default we look ahead 1 hour.