Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto repair incorrect collation on MySQL schema #92270

Merged
merged 4 commits into from
Apr 30, 2023
Merged

Conversation

bdraco
Copy link
Member

@bdraco bdraco commented Apr 29, 2023

Proposed change

As we do more union queries in 2023.5.x if there is a mismatch between collations on tables, they will fail with an error that is hard for the user to figure out how to fix

Error executing query: (MySQLdb.OperationalError) (1271, "Illegal mix of collations for operation UNION")

This was reported in the #beta channel and by PM from others so the problem is not isolated to a single user.

Since we never enforced this in earlier versions its likely some tables will have a mix of utf8mb4_unicode_ci and utf8_general_ci, and utf8mb4_general_ci as they were created with whatever the user's system default were. If they are not all utf8mb4_unicode_ci the unions data between tables with non-like collations will fail.

https://discord.com/channels/330944238910963714/427516175237382144/1100908739910963272

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Black (black --fast homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
  • Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

As we do more union queries in 2023.5.x if there is a mismatch
between collations on tables, they will fail with an error
that is hard for the user to figure out how to fix

`Error executing query: (MySQLdb.OperationalError) (1271, "Illegal mix of collations for operation UNION")`

This was reported in the #beta channel and by PM from others
so the problem is not isolated to a single user

https://discord.com/channels/330944238910963714/427516175237382144/1100908739910963272
@home-assistant
Copy link

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

  • @home-assistant close Closes the pull request.
  • @home-assistant rename Awesome new title Renames the pull request.
  • @home-assistant reopen Reopen the pull request.
  • @home-assistant unassign recorder Removes the current integration label and assignees on the pull request, add the integration domain after the command.

@bdraco
Copy link
Member Author

bdraco commented Apr 29, 2023

If that test doesn't work with 10.3/10.6 I'll have to do the longer CONVERT TO. It will make the test just a tiny bit slower

@bdraco
Copy link
Member Author

bdraco commented Apr 29, 2023

The problem with 10.6/10.3 was that older versions return '' for the collation if it matches the server default so we have to check the server default

@bdraco bdraco marked this pull request as ready for review April 30, 2023 00:11
@bdraco bdraco requested a review from a team as a code owner April 30, 2023 00:11
@balloob balloob merged commit 1a82b35 into dev Apr 30, 2023
@balloob balloob deleted the auto_repair_collation branch April 30, 2023 01:17
balloob pushed a commit that referenced this pull request Apr 30, 2023
* Auto repair incorrect collation on MySQL schema

As we do more union queries in 2023.5.x if there is a mismatch
between collations on tables, they will fail with an error
that is hard for the user to figure out how to fix

`Error executing query: (MySQLdb.OperationalError) (1271, "Illegal mix of collations for operation UNION")`

This was reported in the #beta channel and by PM from others
so the problem is not isolated to a single user

https://discord.com/channels/330944238910963714/427516175237382144/1100908739910963272

* test with ascii since older maraidb versions may not work otherwise

* Revert "test with ascii since older maraidb versions may not work otherwise"

This reverts commit 787fda1.t

* older version need to check collation_server because the collation is not reflected if its the default
raman325 added a commit to raman325/home-assistant that referenced this pull request Apr 30, 2023
* dev: (581 commits)
  Only store and pass around coordinator in Rituals Perfume Genie (home-assistant#92298)
  Avoid returning statistics columns that the metadata knows are impossible (home-assistant#92095)
  Add `date` platform (home-assistant#81948)
  Prevent pysnmp from being installed as it does not work with newer python (home-assistant#92292)
  Use SnapshotAssertion in Renault tests (home-assistant#90778)
  Home Connect add WasherDryer support (home-assistant#90673)
  Move lastfm constants to separate file (home-assistant#92289)
  Fix august lock state when API reports locking and locked with the same timestamp (home-assistant#92276)
  Ensure onvif webhook can be registered (home-assistant#92295)
  Bump beacontools to fix conflict with construct<2.10 and >=2.8.16 (home-assistant#92293)
  Bump pynina to 0.3.0 (home-assistant#92286)
  Fix mqtt not available when starting snips (home-assistant#92296)
  Extract Rituals Perfume Genie DataUpdateCoordinator into module (home-assistant#92284)
  Bump bleak to 0.20.2 (home-assistant#92294)
  Add missing fstrings in Local Calendar (home-assistant#92288)
  Remove myself from switcher_kis codeowners (home-assistant#92277)
  Handle AttributeError from wrong port in ONVIF config flow (home-assistant#92272)
  Auto repair incorrect collation on MySQL schema (home-assistant#92270)
  Update orjson to 3.8.11 (home-assistant#92228)
  Update coverage to 7.2.4 (home-assistant#92229)
  ...
@github-actions github-actions bot locked and limited conversation to collaborators May 1, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.