-
Notifications
You must be signed in to change notification settings - Fork 532
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: ABS Server becomes unreachable #3251
Comments
Was the server accessible/working after upgrading to 2.12.1 and only became inaccessible later? |
Yes the server was accessible and working for at least 8 hours after the upgrade (I skipped 2.12.0). In that time it had been used by 3 people, 1x iOS and 2x Android to listen to books, I had also accessed by the website but did not start a listening session that way. In the logs above you can see the last event that was captured before I restarted the server at 07:35 was someone listening to an audiobook using the android app. |
My guess is that it is related to the problem in #3241. Had the same problem after upgrading. First the db crashed with " SQLITE_CORRUPT: database disk image is malformed" or "ERROR: [Database] Failed to load extension /unicode.so" and after that the server was unreachable, because it couldnt start the container with corrupted db. Do you use docker perhaps take a look in those logs also? |
I thought that issue happened immediately upon updating and wouldn't boot at all (haven't experienced #3241). This issue shows the server becoming unresponsive after the server was working on 2.12.1 for several hours |
For me v2.12.1 worked sometime and just after re-scanning a library the bug in #3241 happend (I think also it happend after batch deleting stuff). Perhaps it can be triggered otherwise also. I think docker logs of this issue would be helpful. In there should be the reason why server wont start up. |
The server worked after restarting the container and I can't see any errors. The docker logs look the same as the ones from ABS (see below) and the crash_logs file has one entry from 5 months ago.
|
i experienced the same today. the server (in docker, nginx reverse proxy) became unresponsive, no further Logs were generated. after restarting (the docker container) everything is working again and no errors regarding the DB. This started to happen after the upgrade to |
Same behavior here. ABS seems to not reachable anymore after about 24h runtime for me. Only happened after upgrading to 2.12.1. No logs or anything whatsoever. Resource usage also seems to be fine here. Restarting the container fixes it for the day unless it happens again the other day :) |
I encountered a similar issue after updating to version 2.12.1 yesterday. Approximately 24 hours later, the instance became unreachable. There were no automatic or scheduled library scans, folder monitoring, or similar processes running, as all those features were deactivated. Until then, the server had been running continuously 24/7 for nearly a year, with restarts only occurring for updates. |
Sadly same issue here. I would like to provide something to solve it but nothing shown. Server was running on previous update for some weeks without any issue. Directly after update, server became unreachable. I restarted the server and was working for around 2 hours. Then was getting unreachable during tagging a new audiobook. Restarted it, didnt work. Restarted it again and worked for around 5 hours. Then became unreachable over night. Since then I can't reach it anymore at all. EDIT: Now I found this in log (nothing changed on the server except updating it, never crashed before in months):
Restarted it and now works again without complain about the db. |
I’m running with docker and experiencing the same issue daily after upgrading from 2.11.0 to 2.12.1. Can anyone tell me if it’s safe to simply downgrade back to 2.11.0 by editing my docker-compose, specifying the version and re-creating the container? That won’t cause database issues? |
I can confirm @pjv take a backup and roll it back, it worked for me. |
Seeing this exact behaviour in a Debian LXC install. Every few hours ABS becomes unresponsive and one core is pegged at 100% usage. Nothing in the logs. Restarting the ABS service fixes the issue. So my "solution" is to just set a cron job to restart the ABS service every 8 hours (time arbitrarily chosen are this stage). |
@migol Thanks - 2.11.0 running again and looks fine so far; nothing adverse in the logs. |
Exact same issues as others here. Running in an LXC container, ABS goes unresponsive after a bit and restarting seems to fix it. Checking my stats, the cpu spikes to about 50% when this issue ocuurs. And if I look back 90 days, across all my LXCs it's the only one to exhibit this behavoir since it occured after the update. |
It is only the CPU which is spiking for everyone, correct? RAM, disk, and network appear normal? |
It was only CPU for me. |
I have woken up the past two mornings to find my docker installation of ABS unresponsive in the same manner as everyone else here. No logs indicating a problem, a quick container restart brings it back online. No usage issues in between failures. Been rock solid for quite some time until this last update. (2.12.1) |
My RAM was around ~500mb, but I have no idea if that is a normal amount after a day of use with multiple listening sessions. It doesn't seem excessive for how often it gets used. (On a fresh restart it uses bout 50mb though). |
I can add that this behavior also happens on the windows build of ABS 2.12.1 |
Had the same crash happened on 2.12.1: I wonder if I have to reset audiobookshelf? Sounds like the DB is corrupted. Anyways, I just updated to 2.12.1 because I tried to access abs today and saw it crashed for no reason, when I saw there were updates available and seemed to target server crashing I installed it only to have the server crash again when I tried to "match Authors", server also hasn't picked up a new book I placed on the directory I'm watching. Strange. |
@mantas3 What's TOD? Me wants! |
@rpgdev time of death 😄 monitoring - uptime kuma https://github.com/louislam/uptime-kuma |
My Deb PPA install also crashed. I didnt collect any data because I figured it was an isolated incident. I will keep an eye on my install and collect data if it happens again. |
Time Of Death |
ESL here, didn't know that abbreviation. Guess eventually I'll have a tod too.. not to get too existential lol. I was after uptime kuma, looks like it sends you updates to whatsapp @mantas3 ? or is that telegram? |
I have had this happen over the last 48 hours, about 24 hours after the server was started. |
Many thanks to @advplyr for getting a fix out so quickly! |
I use telegram, but kuma supports A LOT of notification methods |
Any issue related to a crash should not go in this thread. This is specific to the issue with MemoryStore that was hanging the server with CPU spike. The other issue getting patched in v2.12.3 in #3241 is the sqlite corrupt crashes caused by the unicode sqlite extension. |
My Docker container was running for 24h without issues with Version 2.12.2, but was updated by Watchtower to 2.12.3 this morning, so I cannot say if it would have crashed later. |
This is fixed as of Thanks everyone for helping us track it down. |
What happened?
This morning (07:00 local time) my audiobookshelf instance was unreachable via the app and browser. I checked using the internal IP and external address.
There are no entries in the debug log or the server crash log and I don't know what time the issue occurred.
Additional Information:
-My automatic backup scheduled at 06:00 did not run.
-I changed the time and ran an automatic backup after this incident which was successful so crash unlikely to be related.
-I upgraded to 2.12.1 yesterday and this was the first night this version was running.
-No new books/files have been added for two days.
What did you expect to happen?
I expected to be able to reach the ABS install via website/app. This is the first crash I've noticed on this install in 6+ months.
Steps to reproduce the issue
Currently unknown, sorry.
Audiobookshelf version
2.12.1
How are you running audiobookshelf?
Docker
What OS is your Audiobookshelf server hosted from?
Linux
If the issue is being seen in the UI, what browsers are you seeing the problem on?
None
Logs
Additional Notes
I asked on discord and another user experienced the same issue.
The text was updated successfully, but these errors were encountered: