Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove recorder PostSchemaMigrationTask #125076

Merged
merged 3 commits into from
Sep 3, 2024

Conversation

emontnemery
Copy link
Contributor

@emontnemery emontnemery commented Sep 2, 2024

Proposed change

Remove recorder PostSchemaMigrationTask which took care of additional cleanup after migration to schema versions 32 and 35.

Migration to schema versions 32 and 35 is done before Home Assistant starts, and by also doing the additional cleanup before Home Assistant starts the state is well defined once the non-live migration tasks are completed.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Ruff (ruff format homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

@emontnemery emontnemery requested a review from a team as a code owner September 2, 2024 11:57
@home-assistant
Copy link

home-assistant bot commented Sep 2, 2024

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

  • @home-assistant close Closes the pull request.
  • @home-assistant rename Awesome new title Renames the pull request.
  • @home-assistant reopen Reopen the pull request.
  • @home-assistant unassign recorder Removes the current integration label and assignees on the pull request, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

# columns to be timestamps. In version 32 we need to wipe the old columns
# since they are no longer used and take up a significant amount of space.
assert self.instance.engine is not None, "engine should never be None"
_wipe_old_string_time_columns(self.instance, self.instance.engine, session)
Copy link
Contributor Author

@emontnemery emontnemery Sep 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is limited to 100k rows for MySQL / MariaDB / PostgreSQL, additional rows will eventually be purged. Should we instead loop until there are no more rows?

Copy link
Member

@bdraco bdraco Sep 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figured it wasn't urgent to delete them and it was better to let them get cleaned up over time so the migration didn't take too long. At this point since most people have been migrated it might make more sense to do the full cleanup instead of waiting for purge.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, that makes sense 👍

@emontnemery
Copy link
Contributor Author

emontnemery commented Sep 2, 2024

I think there's a problem with test coverage, because when applying the patch below on top of the PR - which makes the post migrators not run - all recorder tests still pass.

diff --git a/homeassistant/components/recorder/migration.py b/homeassistant/components/recorder/migration.py
index 213462e3731..d0263ffbef3 100644
--- a/homeassistant/components/recorder/migration.py
+++ b/homeassistant/components/recorder/migration.py
@@ -1401,6 +1401,7 @@ class _SchemaVersion32Migrator(_SchemaVersionMigrator, target_version=32):
         _drop_index(self.session_maker, "events", "ix_events_event_type_time_fired")
         _drop_index(self.session_maker, "states", "ix_states_last_updated")
         _drop_index(self.session_maker, "events", "ix_events_time_fired")
+        return
         with session_scope(session=self.session_maker()) as session:
             # In version 31 we migrated all the time_fired, last_updated, and last_changed
             # columns to be timestamps. In version 32 we need to wipe the old columns
@@ -1488,6 +1489,7 @@ class _SchemaVersion35Migrator(_SchemaVersionMigrator, target_version=35):
         # In version 34 we migrated all the created, start, and last_reset
         # columns to be timestamps. In version 35 we need to wipe the old columns
         # since they are no longer used and take up a significant amount of space.
+        return
         while not cleanup_statistics_timestamp_migration(self.instance):
             pass

Edit: Tests improved in PR #125100 and PR #125091

@emontnemery emontnemery merged commit 7c223db into dev Sep 3, 2024
40 checks passed
@emontnemery emontnemery deleted the recorder_remove_post_migration_job branch September 3, 2024 05:51
@github-actions github-actions bot locked and limited conversation to collaborators Sep 4, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants