Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Cluster Activity historical metrics for ongoing dagrun if end_date is provided #33488

Conversation

pierrejeambrun
Copy link
Member

@pierrejeambrun pierrejeambrun commented Aug 18, 2023

We were selecting DagRuns/Tasks that were either:

  • without end_date (to have current runs)
  • with an end_date defined that is less than the requested date

Issue, if the end_date is chosen in the past, runs without end_date (ongoing run) will be counted in the metrics, which is wrong.

This handle such cases i.e when dagrun.end_date is None:

  • If the end_date is in the future then we want data for ongoing runs
  • If the end_date is in the past, we don't want data for ongoing runs

before

An old range such as this one would be updated with data regarding ongoing runs (i.e end_date is None)
image

After

This will not be the case anymore, data will remain stable on old ranges, because ongoing run will be taken out. In this example there is no records on this period.
image

To still be able to get data from ongoing run in the query we need to provide an end_date that is greater than 'now' by default we look ahead 1 hour.

Runs are considered out of range if the requested end_date <= now and that the run still has no end_date because it will settle in the future with an end_date >= now, therefore cannot meet the condition.

@boring-cyborg boring-cyborg bot added area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues labels Aug 18, 2023
@pierrejeambrun pierrejeambrun added the type:bug-fix Changelog: Bug Fixes label Aug 18, 2023
@potiuk potiuk added this to the Airflow 2.7.1 milestone Aug 18, 2023
@pierrejeambrun pierrejeambrun merged commit 7c51c87 into apache:main Aug 18, 2023
@pierrejeambrun pierrejeambrun deleted the fix-cluster-activity-historical-metric-end-date branch August 18, 2023 17:09
ephraimbuddy pushed a commit that referenced this pull request Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues type:bug-fix Changelog: Bug Fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants