MariaDB is not starting after update to 2.7.0 #3566
Comments
|
I don't have all the log data to back me up, but after performing the latest MariaDB update my Home Assistant setup starts up, works for five minutes, and then stops responding. |
|
Similar issues on my end. After updating to 2.7.0 the Database wont start properly. [05:20:32] INFO: Starting MariaDB |
|
Hm, Alpine 3.16 (as used in add-on before the 2.7.0 upgrade) and Alpine 3.17 come with MariaDB 10.6.16, Alpine 3.18 and 3.19 come with 10.11.6. So there is no Alpine release with 10.7 🤔 |
|
Ok, reading this again, the problem seems to be the add-on shutdown before the upgrade was not graceful in your case: Maybe just downgrading and upgrading again helps? If not (or in any case), you can use |
in the HA terminal I cant scroll up enough to see the reason |
Note, I am talking about the reason the previous shutdown failed, not the startup... If you can see the reason of the shutdown failure, can you please share that part of the log as well? This way we can improve the add-on to correctly shutdown also in your case. FWIW, my guess is a timeout of some sort. From what I understand we wait up to 18s for MariaDB to shutdown. But maybe it took longer in your case? 🤔 |
|
now I rolled back to 2.6.1 |
|
Yeah a manual stop and share the logs to the end of the stop would be interesting. |
|
For me, a graceful shutdown looks like this: |
|
I have same trouble. 🙈 |
I haven't stopped the addon while restoring the backup. I'm using "Home Assistant Google Drive Backup" as an addon to backup and restore my HA :) |
|
Same issue after update from 2.6.1 to 2.7.0 |
1 similar comment
|
Same issue after update from 2.6.1 to 2.7.0 |
|
Can someone provide the logs from the shutdown before the upgrade (see #3566 (comment)). |
s6-rc: info: service legacy-services: stopping s6-rc: info: service mariadb-post successfully stopped [14:02:20] INFO: Service mariadb exited with code 0 (by signal 0) |
|
After having the problem described here, I had success by restoring 2.6.1, stopping the add-on, doing the update to 2.7, and then starting the add-on. |
did you tried to restart HA to check if MariaBD 2.7.0 start as it should ? |
Three scenarios:
|
|
I am running the Nextcloud add-on alongside MariaDB. After upgrading MariaDB I can see in the logs that Nextcloud starts without issue but there is not a word in the logs about any homeassistant tables. |
Just gave it a try on my test instance and do not see an issue for far on this machine (HAOS 12.2 on a Pi4) although there are some strange errors concerning incorrect definitions of table mqsql.column_stats. Full upgrade log including stop, install, start: |
|
@N3rdix I actually noticed this errors in my MariaDB too: I think the problem here is that just the upgrade step gets run afterwards: Reading MariaDB documentation this kinda seems expected (see https://mariadb.com/kb/en/upgrading-between-major-mariadb-versions/#step-by-step-instructions-for-upgrades). Do they appear for you when restarting the addon? |
your method works, thank you ! |
No, seems to have been a one-time error. Log looks clean at restart: |
|
@N3rdix ok thanks for confirming. So I think these errors are are really not problematic (and not really relevant for the issue here). @EvgenyBash so your shutdown log looks good. Is that really from the last shutdown before the upgrade? And you see the same error on startup as the original poster saw? |
|
I just realized, the Supervisor logs OP shared actually show the problem we run into: The MariaDB took longer than 30s to shutdown. It got killed by Docker, so that then caused problem for the new version: @paboman what is the size of your database? Is each stop of MariaDB taking that long for you? 🤔 |
|
@agners This is the log already after the 2.6.1 restore. Most likely MariaDB really took a long time to shut down. |
|
After update to 2.7.0 log: |
It seems this stop right after a restore just took not as long then. 🤔 I guess the stop on upgrade took much longer as the add-on has been started much longer, and had a lot of data in page cache and whatnot. So stop took longer that time around. I guess the fix here is to just increase the shutdown timeout (currently 18 seconds, plus some more until kill kicks in). |
After a major update, the mariadb-upgrade tool should be run (see https://mariadb.com/kb/en/mariadb-upgrade/). However, we don't have a specific post-upgrade script we can run. Luckily, MariaDB has a feature to run the upgrade on it's own when needed. Enable the feature using the MARIADB_AUTO_UPGRADE environment variable.
|
Anyone which runs into Please restore the 2.6.1 backup and upgrade again. This second upgrade should typically work as shutdown is much quicker after a fresh restore/start of the old add-on version. As this is a problem of 2.6.x not shutting down properly, we can't really fix it with an update. The fix #3570 will only address this problem for future upgrades. |
|
@cortado your case looks a bit different: Maybe still try restore the old version and upgrade again, it might be shutdown related to. If the problem persists, please open a new issue. |
|
Just ran into this problem, updated from the command line, so no backup. Lost all my historical data from the past 2 years 😢 Downgrading to 2.5.0 (the last backup I had from 2 years ago) works fine. Stopping the addon and then updating results in the exact same error as before. |
|
Same here. After 3 unsuccessful tries, I finally managed to get it working. The key to update from a previous (restored) version is waiting time. For all unsuccessful tries I either did not wait long enough or interrupted the process (restore/stop/update) at some point... |
* mariadb: Increase timeout for MariaDB shutdown Currently, when MariaDB takes longer than 20s to shutdown, Docker will simply kill it. This can be problematic on upgrade, as the new MariaDB version cannot handle a not cleanly shutdown MariaDB (see #3566). Increase the timeout of the container shutdown to 60s. Also define a per-service kill timeout for mariadb-core. This is largely optional, and essentially leads to the same outcome if MariaDB takes even longer (it will get killed, one way or another). But by defining this as part of the s6 service makes sure that the add-on log itself shows that MariaDB had to be killed, and makes sure the rest of the s6 service tree gets a chance to shutdown gracefully still: [09:29:37] INFO: Service mariadb exited with code 256 (by signal 9) s6-rc: info: service mariadb-core successfully stopped s6-rc: info: service mariadb-pre: stopping ... The 18s timeout of S6_SERVICES_GRACETIME seems to apply for legacy services only which are no longer used in this add-on. So drop this definition. Fixes #3566 * Increase shutdown timeout to 5 minutes
Ugh. Yeah unfortunately you can't downgrade without having a backup. What you could do in this case is take a backup of the upgraded/non-working state (as the data is still there, it is just the new version of MariaDB cannot recover it 😢 ), download that backup, open the tar (I think 7-Zip can do that on Windows) edit |
|
Hello, my HA is currently running MariaDB 2.6.1 |
The important thing here is that the stop happened cleanly. Check the add-on logs after the stop to learn if the stop worked cleanly. It should contain something this line: In any case, a backup is always a good idea 😄 |
|
Thank you for this quick answer! |
|
hi, I'm trying to update, but my container always shuts down with code 137
and obviously if I try to update this happens: I don't know what to do, fortunately restore version 2.6.1, it always works |
|
Can you restore 2.6.1 again, then try the following command to stop the add-on before updating in the Advanced SSH & Web Terminal (from the community store) |
yes, it works, it took 42 seconds maybe it should be added in the supervisor that it has to wait that time only when updating. I imagine that it cannot be set as a default for things such as that the system must be able to shut down in an emergency situation such as when the power goes out and you are on UPS. but I don't know how it works, I just wanted to say my opinion and thank you so much for your help Even if it works, errors have come up repeatedly, I don't know if it helps: |
If you restart the 2.7.1 version, these errors should disappear: during first startup after the upgrade, mariadb has issues with these statistics data (20:29:55 4 [ERROR]), but a few seconds later a script runs that updates these schema ([20:29:59] INFO: Ensuring internal database upgrades are performed). |
This comment was marked as outdated.
This comment was marked as outdated.
|
Today, after updating HomeAssistant, I ran into this issue. I thought I was running successfully on MariaDB 2.7.1 since the update and since 2.7.x. I did see something about needing to restart before updating to 2.7.1 - I verified I was on 2.7.0, so I updated to Anyway, after updating HomeAssistant (supervisord), MariaDB now will not start giving me the above error. And for some reason, MariaDB isn't in my backups (probably because it was crashing ....), I'm left without a way to downgrade. How do I manually tell Supervisor to use 2.6.1? Or how can I just download the right Docker image and run it myself? I just need the full docker cli, and I run it myself as a temporary solution. I can't seen to find that. I've tried this, but no go: |
|
@guice what you can do (but it is a hack) just download the other version and tag it as the latest: Then restart the add-on via UI. Then try stopping it, and check for successful stop in the logs. Don't forget to restore the original container image after 2.6.1 startup/stop succeeded. Then restart the container. Note this is a hack, and only really works as long as there is no incompatible config change (which is the case between these two versions). |
Yeah it is not all that easy, as we get the binaries from Alpine directly. Afaik, it isn't possible to deploy two mariadb packages (from different Alpine versions) into one container. In the end, it seems that only so many people are affected, and there are a couple of workarounds documented here. |
Well I've tried your backup -> modify the json -> upload backup method. I could be wrong, but in 7zip it appears like the json file is in a "." folder. If extracted under windows with 7zip the folder gets dropped and the json is plain in the root folder of the extraction. And you can't create a "." folder under windows. Just modifying the tar with 7zip does not work either, as the folder is deleted as soon as you delete the json file. Creating a dot folder in 7zip does not work. I've tried to just pack the json in the tar without a folder but either I did something wrong or it doesn't work that way: The backup does not show up in HA. In addition I discovered that trying to backing up mariadb in the broken state does not work: You end up with a 10 KB file which only contains the json. Only after stopping the add on the backup completes correctly. |
|
Hm yeah I did not try the modify json approach myself, and I can imagine that under Windows the outcome might not work 😢 But there are many ways to solve the issue. So in what state are you now exactly (what does the log show)?
This sounds like a different issue altogether, which probably warrants investigation on it's own. |
* mariadb: Increase timeout for MariaDB shutdown Currently, when MariaDB takes longer than 20s to shutdown, Docker will simply kill it. This can be problematic on upgrade, as the new MariaDB version cannot handle a not cleanly shutdown MariaDB (see home-assistant#3566). Increase the timeout of the container shutdown to 60s. Also define a per-service kill timeout for mariadb-core. This is largely optional, and essentially leads to the same outcome if MariaDB takes even longer (it will get killed, one way or another). But by defining this as part of the s6 service makes sure that the add-on log itself shows that MariaDB had to be killed, and makes sure the rest of the s6 service tree gets a chance to shutdown gracefully still: [09:29:37] INFO: Service mariadb exited with code 256 (by signal 9) s6-rc: info: service mariadb-core successfully stopped s6-rc: info: service mariadb-pre: stopping ... The 18s timeout of S6_SERVICES_GRACETIME seems to apply for legacy services only which are no longer used in this add-on. So drop this definition. Fixes home-assistant#3566 * Increase shutdown timeout to 5 minutes
|
I hope this is not considered spam, but I just have to say it: |
|
There is no way in HAos to go to the previous plugin version or choose one (would be ideal). I can somehow edit something to make it... Even if it means losing the data, it would be deleting the plugin and deleting something, right? |
There is a difference between plug-in and add-on: Plug-ins (e.g. DNS,Audio or CLI plug-in) extend Supervisor functionality and are tighly managed by Supervisor. For these, there are no backups. However, you can downgrade these from the console, e.g.: But you are probably referring to add-ons. Add-ons can only be downgraded through backups. This is currently a limitation of add-ons. Technically, the main problem is that our add-on repositories only carry the metadata of the add-ons of the latest version 😢 Backups do have the metadata of the older version included. That is why it is possible to go back to an older version by (partially) restoring an older backup. |
|
Thx. for all of the information here. After 6 hours of restore time for 2.6.1 I finally migrated to 2.7.1!
|
My backup was deleted... so I'll have to start again. Do I have to delete something after uninstalling to start over? (In case there is a mariadb folder or volume left) Note: It would be nice to be able to mark the backups with the last copy of the plugin or a number, so that not everything counts as backups (I guess that's why it was deleted). Note: From now on I will mark the backup of the mariadb and nginx addonas as backups not to be deleted |
|
I have the errors: I make regular backups using samba backup, but for unclear reasons it does not include mariadb. So I also seem stuck. How do I get back to having my home assistant working again? |
|
If you use HAos you have to recover the previous copy of mariadb. If you do not have the backup, you have to delete the plugin, restart and install again (configure). |
`s6-rc: info: service legacy-services successfully started s6-rc: info: service mariadb-post successfully stopped [10:04:50] INFO: Service mariadb exited with code 0 (by signal 0) Took a little longe for me and I had to add |
and others
Describe the issue you are experiencing
what should i do ?
What type of installation are you running?
Home Assistant OS
Which operating system are you running on?
Home Assistant Operating System
Which add-on are you reporting an issue with?
MariaDB
What is the version of the add-on?
2.7.0
Steps to reproduce the issue
1.update from 2.6.1 to 2.7.0
System Health information
System Information
Home Assistant Community Store
Home Assistant Cloud
Home Assistant Supervisor
Dashboards
Sonoff
Anything in the Supervisor logs that might be useful for us?
Anything in the add-on logs that might be useful for us?
Additional information
No response
The text was updated successfully, but these errors were encountered: