Improve recorder history queries #131702

emontnemery · 2024-11-27T10:15:51Z

Proposed change

Improve recorder history query logic which avoids unnecessary database queries

Background

We want to avoid unnecessary database queries for time periods where we can't possibly have any states.

Without this PR, the recorder runs table is consulted to determine if there may be any states during the requested interval. This doesn't always work well though:

There was a bug related to purging of recorder runs which was recently fixed by Fix logic for purge of recorder runs #130378, which meant we may have states older than the oldest recorder run
A recorder run may be much longer than the purge interval

There's more details in this PR #123449

This PR changes the logic to instead consider the timestamp of the oldest state in the database

Type of change

Dependency upgrade
Bugfix (non-breaking change which fixes an issue)
New integration (thank you!)
New feature (which adds functionality to an existing integration)
Deprecation (breaking change to happen in the future)
Breaking change (fix/feature causing existing functionality to break)
Code quality improvements to existing code or addition of tests

Additional information

This PR fixes or closes issue: fixes #
This PR is related to issue:
Link to documentation pull request:

Checklist

The code change is tested and works locally.
Local tests pass. Your PR cannot be merged unless tests pass
There is no commented out code in this PR.
I have followed the development checklist
I have followed the perfect PR recommendations
The code has been formatted using Ruff (ruff format homeassistant tests)
Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

The manifest file has all fields filled out correctly.
Updated and included derived files by running: python3 -m script.hassfest.
New or updated dependencies have been added to requirements_all.txt.
Updated by running python3 -m script.gen_requirements_all.
For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

I have reviewed two other open pull requests in this repository.

home-assistant · 2024-11-27T10:15:58Z

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

@home-assistant close Closes the pull request.
@home-assistant rename Awesome new title Renames the pull request.
@home-assistant reopen Reopen the pull request.
@home-assistant unassign recorder Removes the current integration label and assignees on the pull request, add the integration domain after the command.
@home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
@home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

home-assistant · 2024-11-27T10:15:58Z

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (history) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of history can trigger bot actions by commenting:

@home-assistant close Closes the pull request.
@home-assistant rename Awesome new title Renames the pull request.
@home-assistant reopen Reopen the pull request.
@home-assistant unassign history Removes the current integration label and assignees on the pull request, add the integration domain after the command.
@home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
@home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

emontnemery

Some comments

In a follow-up PR, we should simplify the RecorderRunsManager to remove functionality which is no longer needed when the history API doesn't use it.

homeassistant/components/history/helpers.py

homeassistant/components/recorder/history/legacy.py

homeassistant/components/recorder/history/modern.py

homeassistant/components/recorder/purge.py

emontnemery · 2024-11-27T10:27:59Z

homeassistant/components/recorder/table_managers/states.py

+        if not result:
+            ts = None
+        else:
+            ts = result[0].last_updated_ts


Should we allow a scalar option to or add a scalar version of execute_stmt_lambda_element to avoid these extra checks?

If there are other places it could be used, it sure would be nice to avoid the boilerplate checks

Yes, there are. For example here:

core/homeassistant/components/recorder/statistics.py

Lines 1279 to 1281 in f428958

if stats := cast(Sequence[Row], execute_stmt_lambda_element(session, stmt)):

return dt_util.utc_from_timestamp(stats[0].start_ts)

return None

and several other places in the same module

I think we should do this in a separate PR though

Separate PR for sure 👍

emontnemery

One more comment about the choice of timestamp for the oldest state, I wonder if this may be a dealbreaker for the approach in this PR?

On second thought, it's probably fine to use last_updated because:

States are inserted in the states table in last_updated order, not last_changed order
States are purged according to their last_updated timestamps
The history API selects states according to their last_updated timestamps, not according to their last_changed timestamps

homeassistant/components/recorder/table_managers/states.py

emontnemery · 2024-11-27T11:27:09Z

Set to draft until I've discussed the implementation with @bdraco

homeassistant/components/recorder/table_managers/states.py

homeassistant/components/recorder/purge.py

homeassistant/components/recorder/history/modern.py

bdraco

Tested on production, did manual purge. Everything appears to work as expected

* Improve recorder history queries * Remove some comments * Update StatesManager._oldest_ts when adding pending state * Update after review * Improve tests * Improve post-purge logic * Avoid calling dt_util.utc_to_timestamp in new code --------- Co-authored-by: J. Nick Koston <[email protected]>

Improve recorder history queries

fbec7a7

emontnemery requested a review from a team as a code owner November 27, 2024 10:15

home-assistant bot added bugfix cla-signed core integration: history integration: recorder by-code-owner Quality Scale: internal labels Nov 27, 2024

emontnemery marked this pull request as draft November 27, 2024 10:16

Remove some comments

6ec1340

emontnemery commented Nov 27, 2024

View reviewed changes

homeassistant/components/recorder/table_managers/states.py Show resolved Hide resolved

Update StatesManager._oldest_ts when adding pending state

f428958

emontnemery mentioned this pull request Nov 27, 2024

Fix history API not returning data when recorder runs are lost #123449

Closed

19 tasks

bdraco reviewed Nov 27, 2024

View reviewed changes

homeassistant/components/recorder/table_managers/states.py Outdated Show resolved Hide resolved

Update after review

bb09cda

emontnemery commented Nov 27, 2024

View reviewed changes

homeassistant/components/recorder/purge.py Outdated Show resolved Hide resolved

emontnemery added 2 commits November 27, 2024 16:59

Improve tests

68e1989

Improve post-purge logic

cb1682a

emontnemery marked this pull request as ready for review November 27, 2024 16:03

bdraco reviewed Nov 27, 2024

View reviewed changes

homeassistant/components/recorder/history/modern.py Outdated Show resolved Hide resolved

emontnemery and others added 2 commits November 27, 2024 17:26

Avoid calling dt_util.utc_to_timestamp in new code

2e61fab

Merge branch 'dev' into recorder_tweak_history_api

886162e

bdraco approved these changes Nov 27, 2024

View reviewed changes

emontnemery merged commit 381d545 into dev Nov 27, 2024
64 checks passed

emontnemery deleted the recorder_tweak_history_api branch November 27, 2024 20:12

emontnemery mentioned this pull request Nov 28, 2024

Simplify recorder RecorderRunsManager #131785

Merged

19 tasks

emontnemery added this to the 2024.12.0 milestone Nov 28, 2024

emontnemery mentioned this pull request Nov 28, 2024

History API not returning data from a specific time range when recorder runs are lost. Extending end time works #117793

Closed

frenck added the cherry-picked label Nov 28, 2024

github-actions bot locked and limited conversation to collaborators Nov 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve recorder history queries #131702

Improve recorder history queries #131702

emontnemery commented Nov 27, 2024

home-assistant bot commented Nov 27, 2024

home-assistant bot commented Nov 27, 2024

emontnemery left a comment

emontnemery Nov 27, 2024 •

edited

Loading

bdraco Nov 27, 2024

emontnemery Nov 27, 2024

bdraco Nov 27, 2024

emontnemery left a comment •

edited

Loading

emontnemery commented Nov 27, 2024

bdraco left a comment

	if stats := cast(Sequence[Row], execute_stmt_lambda_element(session, stmt)):
	return dt_util.utc_from_timestamp(stats[0].start_ts)
	return None

Improve recorder history queries #131702

Improve recorder history queries #131702

Conversation

emontnemery commented Nov 27, 2024

Proposed change

Background

Type of change

Additional information

Checklist

home-assistant bot commented Nov 27, 2024

home-assistant bot commented Nov 27, 2024

emontnemery left a comment

Choose a reason for hiding this comment

emontnemery Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

bdraco Nov 27, 2024

Choose a reason for hiding this comment

emontnemery Nov 27, 2024

Choose a reason for hiding this comment

bdraco Nov 27, 2024

Choose a reason for hiding this comment

emontnemery left a comment • edited Loading

Choose a reason for hiding this comment

emontnemery commented Nov 27, 2024

bdraco left a comment

Choose a reason for hiding this comment

emontnemery Nov 27, 2024 •

edited

Loading

emontnemery left a comment •

edited

Loading