Prevent endless loop in recorder when using a filter and there are no more states to purge

[davinkevin]

Proposed change

After investigation for #124994, I found a situation where the recorder is stuck in a loop. I suspect a misunderstanding in the code related to variables names and expectations from methods

To summarize, in purge.py, we have:

            if apply_filter and _purge_filtered_data(instance, session) is False:
                _LOGGER.debug("Cleanup filtered data hasn't fully completed yet")
                return False

With a _purge_filtered_data definition (simplified):

def _purge_filtered_data(instance: Recorder, session: Session) -> bool:
    """Remove filtered states and events that shouldn't be in the database."""
    _LOGGER.debug("Cleanup filtered data")
    …
    has_more_states_to_purge = False
    excluded_metadata_ids: list[str] = […]
    if excluded_metadata_ids:
        has_more_states_to_purge = _purge_filtered_states(
            instance, session, excluded_metadata_ids, database_engine, now_timestamp
        )

    # Check if excluded event_types are in database
    has_more_events_to_purge = False
    if (…) and (…):
        has_more_events_to_purge = _purge_filtered_events(
            instance, session, excluded_event_type_ids, now_timestamp
        )

    # Purge has completed if there are not more state or events to purge
    return not (has_more_states_to_purge or has_more_events_to_purge)

The problem in my case is the _purge_filtered_states always returns True, but when we look at its implementation/description, we can see the following:

def _purge_filtered_states(…) -> bool:
    """Remove filtered states and linked events.

    Return true if all states are purged
    """
    state_ids: tuple[int, ...]
    attributes_ids: tuple[int, ...]
    event_ids: tuple[int, ...]
    to_purge = list(…)
    if not to_purge:
        return True
    
    …
    return False

This function returns True if "all states are purged", however the calling function assign that value to has_more_states_to_purge value… which is literally the opposite.

The same applies to the _purge_filtered_events function later in that function.

I just introduced a not operator to invert the value. If the purge is complete, then has_more_*****_to_purge will be false; so the following code can stay like this.

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Deprecation (breaking change to happen in the future)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• I have followed the perfect PR recommendations
• The code has been formatted using Ruff (ruff format homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[home-assistant]

Hi @davinkevin

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

[home-assistant]

Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍

Learn more about our pull request process.

[home-assistant]

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

• @home-assistant close Closes the pull request.
• @home-assistant rename Awesome new title Renames the pull request.
• @home-assistant reopen Reopen the pull request.
• @home-assistant unassign recorder Removes the current integration label and assignees on the pull request, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

[davinkevin]

Local test execution:

[github-actions]

There hasn't been any activity on this pull request recently. This pull request has been automatically marked as stale because of that and will be closed if no further activity occurs within 7 days.
If you are the author of this PR, please leave a comment if you want to keep it open. Also, please rebase your PR onto the latest dev branch to ensure that it's up to date with the latest changes.
Thank you for your contribution!

[davinkevin]

Up?

Stale

[bdraco]

This change makes sense

_purge_legacy_format _purge_states_and_attributes_ids _purge_events_and_data_ids all return True if there is more to purge

_purge_filtered_states and _purge_filtered_events behave differently.

I think this is good to merge once we get a test for it

[davinkevin]

FYI, I'm not the best to write python code. This contribution is the result of an issue I ha[d|ve]. tldr; if someone can write the test, it would be welcome.