Remove legacy foreign key constraint from sqlite states table

[bdraco]

Do not tag this for backport. It should bake in dev for at least a week, and than beta before appearing in a release. Since we were unable to test in beta, we have instead enough people test the custom component in #117263. While a 12 step migration should be quite safe (and in most cases what sqlite actually does under the hood for complex alters), this is brand new for this code base and we cannot fully predict impact.

Proposed change

Some old databases have FOREIGN KEY(event_id) REFERENCES events (event_id) ON DELETE CASCADE, in the states table and the index is now empty which results in a full table scan per each event_id.

Unfortunately sqlite does not support dropping a foreign key constraint.

To fix this the whole states table needs to be recreated with a 12 step table rebuild to remove the legacy foreign key from the states table

Thankfully a table rebuild is very fast in sqlite. With 2GiB database taking about 20s (1GiB took < 10s). One iteration of the purge cycle that was happening in the issue would take longer than a table rebuild.

2024-06-28 09:39:57.325 WARNING (Recorder) [homeassistant.components.recorder.migration] Rebuilding SQLite table states; This will take a while; Please be patient!
2024-06-28 09:40:18.633 WARNING (Recorder) [homeassistant.components.recorder.migration] Rebuilding SQLite table states finished

A nice side effect is it will reduce the database size by ~4-8% on average after the next repack (2nd sunday of the month) and improve insert performance a tiny bit.

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Deprecation (breaking change to happen in the future)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes Purge causes recorder to stop writing to the DB until HA is restarted (Auto purge happens at 4:12am) #117263
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• I have followed the perfect PR recommendations
• The code has been formatted using Ruff (ruff format homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
• Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[home-assistant]

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

• @home-assistant close Closes the pull request.
• @home-assistant rename Awesome new title Renames the pull request.
• @home-assistant reopen Reopen the pull request.
• @home-assistant unassign recorder Removes the current integration label and assignees on the pull request, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

[bdraco]

The 9 step alter isn't going to work for this case, and quite frankly its too risky. We need to do the full 12 step procedure https://www.sqlite.org/lang_altertable.html which is slower, but overall sqlite table rebuilds are fast so this is fine.

[bdraco]

Tested on 2x production
and @dmulcahey did as well

Need to write some tests

[bdraco]

We already have an migration test for v32+. I've updated it now to verify the index can now be dropped for sqlite as well.

I want to write some direct coverage for rebuild_sqlite_table as well on a test database

[bdraco]

Thanks.

We need to do some more migrations for #120115 and #119167 so 2024.8.x would probably be a good target to fix those as well if I can find the bandwidth to dedicate to them