Fix logic for purge of recorder runs

[emontnemery]

Proposed change

Fix logic for purge of recorder runs to purge recorder runs which end before the purge date instead of purging runs which start before the purge data

Rationale:
Until fairly recently we did not use the existence of recorder runs as a shortcut to avoid searching for history events, so when we purged the recorder runs was not very important.

However, we do now because of #104724 and I think @RudolfRendier is right, we need to change the purging of recorder runs.

This was noticed in #123449

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Deprecation (breaking change to happen in the future)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• I have followed the perfect PR recommendations
• The code has been formatted using Ruff (ruff format homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[home-assistant]

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

• @home-assistant close Closes the pull request.
• @home-assistant rename Awesome new title Renames the pull request.
• @home-assistant reopen Reopen the pull request.
• @home-assistant unassign recorder Removes the current integration label and assignees on the pull request, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

[RudolfRendier]

Can you safely compare to .end even if it's not always present in the database record? Will it be set if a recorder_run is closed incorrectly?

Answering my own question:
Looks like 'end_incomplete_runs' does exactly that, it sets closed_incorrect to true and updates 'end' with the start_time (=now):

core/homeassistant/components/recorder/util.py

Line 625 in b19c44b

def end_incomplete_runs(session: Session, start_time: datetime) -> None:

Thanks for the fix @emontnemery

[emontnemery]

@RudolfRendier That's a good point, I wonder if the order here matters:

def delete_recorder_runs_rows(
    purge_before: datetime, current_run_id: int
) -> StatementLambdaElement:
    """Delete recorder_runs rows."""
    return lambda_stmt(
        lambda: delete(RecorderRuns)
        .filter(RecorderRuns.end < purge_before)
        .filter(RecorderRuns.run_id != current_run_id)
        .execution_options(synchronize_session=False)
    )

I'm setting this to draft until I've had the time to take a closer look.

[frenck]

As we are releasing the milestone today, I've removed this draft PR for now.

[emontnemery]

Implementation updated to explicitly not consider unclosed recorder runs in purge.
Resulting SQLite query:

INFO:sqlalchemy.engine.Engine:DELETE FROM recorder_runs WHERE recorder_runs."end" IS NOT NULL AND recorder_runs."end" < ? AND recorder_runs.run_id != ?
INFO:sqlalchemy.engine.Engine:[generated in 0.00016s] ('2024-11-25 15:48:52.311384', 1)