Sonarr | Resolve database corruption

[ezekini]

Creating a bug report/issue

Required Information

• DietPi version:
  G_DIETPI_VERSION_CORE=6
  G_DIETPI_VERSION_SUB=24
  G_DIETPI_VERSION_RC=1
  G_GITBRANCH='master'
  G_GITOWNER='MichaIng'
• Distro version: stretch 9.9
• Kernel version: Linux DietPi-rpi 4.19.42-v7+ General | Update VM images to Stretch #1219 SMP Tue May 14 21:20:58 BST 2019 armv7l GNU/Linux
• SBC device: RPi 3 Model B+ (armv7l)
• Power supply used: 5V 1A RAVpower
• SDcard used: SanDisk ultra

Additional Information (if applicable)

• Software title: Sonarr
• Was the software title installed freshly or updated/migrated?
• Can this issue be replicated on a fresh installation of DietPi? I don't know

Steps to reproduce

1. ...I don't know what triggered this. I didn't update or anything to my knowledge
2. ...

Expected behaviour

• ...Sonarr working flawless

Actual behaviour

root@DietPi-rpi:~# dietpi-services status
[ OK ] DietPi-Services | Root access verified.

DietPi-Services
─────────────────────────────────────────────────────
Mode: status

[ OK ] DietPi-Services | avahi-daemon active (running) since Wed 2019-07-03 19:07:53 -03; 1h 44min ago
[ OK ] DietPi-Services | proftpd active (running) since Wed 2019-07-03 19:07:53 -03; 1h 44min ago
[ OK ] DietPi-Services | nfs-kernel-server active (exited) since Wed 2019-07-03 19:07:53 -03; 1h 44min ago
[ OK ] DietPi-Services | php7.3-fpm active (running) since Wed 2019-07-03 19:07:54 -03; 1h 44min ago
[ OK ] DietPi-Services | lighttpd active (running) since Wed 2019-07-03 19:07:54 -03; 1h 44min ago
[ OK ] DietPi-Services | shairport-sync active (running) since Wed 2019-07-03 19:07:54 -03; 1h 44min ago
[ OK ] DietPi-Services | roonbridge active (running) since Wed 2019-07-03 19:07:54 -03; 1h 44min ago
[ OK ] DietPi-Services | raspotify active (running) since Wed 2019-07-03 19:07:54 -03; 1h 44min ago
[ OK ] DietPi-Services | jackett active (running) since Wed 2019-07-03 19:07:54 -03; 1h 44min ago
[FAILED] DietPi-Services | sonarr failed (Result: exit-code) since Wed 2019-07-03 19:08:24 -03; 1h 44min ago
● sonarr.service - Sonarr (NzbDrone) Daemon (DietPi)
Loaded: loaded (/etc/systemd/system/sonarr.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2019-07-03 19:08:24 -03; 1h 44min ago
Process: 13357 ExecStart=/usr/bin/mono -O=-aot /opt/NzbDrone/NzbDrone.exe -nobrowser -data=/mnt/dietpi_userdata/sonarr (code=exited, status=1/FAILURE)
Main PID: 13357 (code=exited, status=1/FAILURE)

Jul 03 19:08:22 DietPi-rpi mono[13357]: at NzbDrone.Core.Datastore.DbFactory.RegisterDatabase (NzbDrone.Common.Composition.IContainer container) [0x00006] in <96f52d5e5b95442a89ae3bbf62e59664>:0
Jul 03 19:08:22 DietPi-rpi mono[13357]: at NzbDrone.Host.NzbDroneServiceFactory.Start () [0x00037] in <83751627b1fd4bb1b4a2a3075c439f56>:0
Jul 03 19:08:22 DietPi-rpi mono[13357]: at NzbDrone.Host.Router.Route (NzbDrone.Host.ApplicationModes applicationModes) [0x0007b] in <83751627b1fd4bb1b4a2a3075c439f56>:0
Jul 03 19:08:22 DietPi-rpi mono[13357]: at NzbDrone.Host.Bootstrap.Start (NzbDrone.Host.ApplicationModes applicationModes, NzbDrone.Common.EnvironmentInfo.StartupContext startupContext) [0x00047] in <83751627b1fd4bb1b4a2a3075c439f56>:0
Jul 03 19:08:22 DietPi-rpi mono[13357]: at NzbDrone.Host.Bootstrap.Start (NzbDrone.Common.EnvironmentInfo.StartupContext startupContext, NzbDrone.Host.IUserAlert userAlert, System.Action`1[T] startCallback) [0x00075] in <83751627b1fd4bb1b4a2a3075c439f56>:0
Jul 03 19:08:22 DietPi-rpi mono[13357]: at NzbDrone.Console.ConsoleApp.Main (System.String[] args) [0x0002f] in :0
Jul 03 19:08:23 DietPi-rpi mono[13357]: Press enter to exit...
Jul 03 19:08:24 DietPi-rpi systemd[1]: sonarr.service: Main process exited, code=exited, status=1/FAILURE
Jul 03 19:08:24 DietPi-rpi systemd[1]: sonarr.service: Unit entered failed state.
Jul 03 19:08:24 DietPi-rpi systemd[1]: sonarr.service: Failed with result 'exit-code'.
[ OK ] DietPi-Services | radarr active (running) since Wed 2019-07-03 19:07:54 -03; 1h 44min ago
[ OK ] DietPi-Services | cron active (running) since Wed 2019-07-03 19:07:55 -03; 1h 44min ago
[ OK ] DietPi-Services | dropbear active (running) since Wed 2019-07-03 06:01:43 -03; 14h ago
[ OK ] DietPi-Services | pihole-FTL active (running) since Wed 2019-07-03 06:01:28 -03; 14h ago
[ OK ] DietPi-Services | dietpi-ramdisk active (exited) since Wed 2019-07-03 06:01:12 -03; 14h ago
[ OK ] DietPi-Services | dietpi-ramlog active (exited) since Wed 2019-07-03 06:01:12 -03; 14h ago
[ OK ] DietPi-Services | dietpi-preboot active (exited) since Wed 2019-07-03 06:01:16 -03; 14h ago
[ OK ] DietPi-Services | dietpi-boot active (exited) since Wed 2019-07-03 06:01:43 -03; 14h ago
[ OK ] DietPi-Services | dietpi-postboot active (exited) since Wed 2019-07-03 06:01:43 -03; 14h ago
[ INFO ] DietPi-Services | dietpi-wifi-monitor inactive (dead)
[ OK ] DietPi-Services | dietpi-arr_to_RAM active (exited) since Wed 2019-07-03 06:01:15 -03; 14h ago

Extra details

Tried reinstalling via dietpi-software reinstall XXX but the problem continued

[MichaIng]

@ezekini
Many thanks for your report.

Did this happen the first time after some reboot or system crash or do you remember any maintenance task that triggered a restart of all services? Or did indeed the service crash just out of a sudden overnight or something?

Can you please paste the full log: journalctl -u sonarr
Perhaps a database issue. Do you have a backup? I check our install script later to verify that a reinstall cleanly recreates the whole install dir, so that database stays the only possible cause of failure.
Mono itself should not be the issue since Radarr works fine.

[ezekini]

More something like the service crash just out of a sudden overnight. I don't remember any maintenance, installation or anything new or out of the ordinary.

Perhaps a database issue. Do you have a backup? I check our install script later to verify that a reinstall cleanly recreates the whole install dir, so that database stays the only possible cause of failure.

I will check, but I think I haven't backup up in a while.

Mono itself should not be the issue since Radarr works fine.

Can you please paste the full log: journalctl -u sonarr

root@DietPi-rpi:~# journalctl -u sonarr
-- Logs begin at Thu 2019-07-04 06:01:12 -03, end at Thu 2019-07-04 11:13:27 -03. --
jul 04 06:01:47 DietPi-rpi systemd[1]: Started Sonarr (NzbDrone) Daemon (DietPi).
jul 04 06:01:53 DietPi-rpi mono[1259]: [Info] Bootstrap: Starting Sonarr - /opt/NzbDrone/NzbDrone.exe - Version 2.0.0.5337
jul 04 06:01:57 DietPi-rpi mono[1259]: [Info] AppFolderInfo: Data directory is being overridden to [/mnt/dietpi_userdata/sonarr]
jul 04 06:01:58 DietPi-rpi mono[1259]: [Debug] ProcessProvider: Found 0 processes with the name: NzbDrone.Console
jul 04 06:01:58 DietPi-rpi mono[1259]: [Debug] ProcessProvider: Found 1 processes with the name: NzbDrone
jul 04 06:01:58 DietPi-rpi mono[1259]: [Debug] ProcessProvider:  - [1259] NzbDrone
jul 04 06:01:58 DietPi-rpi mono[1259]: [Info] Router: Application mode: Interactive
jul 04 06:01:58 DietPi-rpi mono[1259]: [Debug] Router: Tray selected
jul 04 06:02:00 DietPi-rpi mono[1259]: [Info] MigrationLogger: *** Migrating data source=/mnt/dietpi_userdata/sonarr/nzbdrone.db;cache size
=-10000;datetimekind=Utc;journal mode=Wal;pooling=True;version=3 ***
jul 04 06:02:10 DietPi-rpi mono[1259]: [Fatal] ConsoleApp: EPIC FAIL!
jul 04 06:02:10 DietPi-rpi mono[1259]: [v2.0.0.5337] NzbDrone.Core.Datastore.CorruptDatabaseException: Database file: /mnt/dietpi_userdata/
sonarr/nzbdrone.db is corrupt, restore from backup if available. See: https://github.com/Sonarr/Sonarr/wiki/FAQ#i-am-getting-an-error-datab
ase-disk-image-is-malformed ---> System.Data.SQLite.SQLiteException: database disk image is malformed
jul 04 06:02:10 DietPi-rpi mono[1259]: database disk image is malformed
jul 04 06:02:10 DietPi-rpi mono[1259]:   at System.Data.SQLite.SQLite3.Prepare (System.Data.SQLite.SQLiteConnection cnn, System.String strS
ql, System.Data.SQLite.SQLiteStatement previous, System.UInt32 timeoutMS, System.String& strRemain) [0x0033c] in <61a20cde294d4a3eb43b9d9f6
284613b>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at System.Data.SQLite.SQLiteCommand.BuildNextCommand () [0x000f6] in <61a20cde294d4a3eb43b9d9f6284
613b>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at System.Data.SQLite.SQLiteCommand.GetStatement (System.Int32 index) [0x00008] in <61a20cde294d4a
3eb43b9d9f6284613b>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at (wrapper remoting-invoke-with-check) System.Data.SQLite.SQLiteCommand.GetStatement(int)
jul 04 06:02:10 DietPi-rpi mono[1259]:   at System.Data.SQLite.SQLiteDataReader.NextResult () [0x0011e] in <61a20cde294d4a3eb43b9d9f6284613
b>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at System.Data.SQLite.SQLiteDataReader..ctor (System.Data.SQLite.SQLiteCommand cmd, System.Data.Co
mmandBehavior behave) [0x00090] in <61a20cde294d4a3eb43b9d9f6284613b>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at (wrapper remoting-invoke-with-check) System.Data.SQLite.SQLiteDataReader..ctor(System.Data.SQLi
te.SQLiteCommand,System.Data.CommandBehavior)
jul 04 06:02:10 DietPi-rpi mono[1259]:   at System.Data.SQLite.SQLiteCommand.ExecuteReader (System.Data.CommandBehavior behavior) [0x0000c]
 in <61a20cde294d4a3eb43b9d9f6284613b>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery (System.Data.CommandBehavior behavior) [0x0000
6] in <61a20cde294d4a3eb43b9d9f6284613b>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery () [0x00006] in <61a20cde294d4a3eb43b9d9f62846
13b>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at System.Data.SQLite.SQLiteConnection.Open () [0x00959] in <61a20cde294d4a3eb43b9d9f6284613b>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at FluentMigrator.Runner.Processors.GenericProcessorBase.EnsureConnectionIsOpen () [0x00014] in <1
37fb96feee44f379d6a8fba4e172d1c>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at FluentMigrator.Runner.Processors.SQLite.SQLiteProcessor.Exists (System.String template, System.
Object[] args) [0x00000] in <137fb96feee44f379d6a8fba4e172d1c>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at FluentMigrator.Runner.Processors.SQLite.SQLiteProcessor.TableExists (System.String schemaName,
System.String tableName) [0x00000] in <137fb96feee44f379d6a8fba4e172d1c>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at FluentMigrator.Runner.VersionLoader.get_AlreadyCreatedVersionTable () [0x0001c] in <137fb96feee
44f379d6a8fba4e172d1c>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at FluentMigrator.Runner.VersionLoader.LoadVersionInfo () [0x00028] in <137fb96feee44f379d6a8fba4e
172d1c>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at FluentMigrator.Runner.VersionLoader..ctor (FluentMigrator.Runner.IMigrationRunner runner, Fluen
tMigrator.Infrastructure.IAssemblyCollection assemblies, FluentMigrator.IMigrationConventions conventions) [0x00077] in <137fb96feee44f379d
6a8fba4e172d1c>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at FluentMigrator.Runner.MigrationRunner..ctor (FluentMigrator.Infrastructure.IAssemblyCollection
assemblies, FluentMigrator.Runner.Initialization.IRunnerContext runnerContext, FluentMigrator.IMigrationProcessor processor) [0x00175] in <
137fb96feee44f379d6a8fba4e172d1c>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at FluentMigrator.Runner.MigrationRunner..ctor (System.Reflection.Assembly assembly, FluentMigrato
r.Runner.Initialization.IRunnerContext runnerContext, FluentMigrator.IMigrationProcessor processor) [0x00007] in <137fb96feee44f379d6a8fba4
e172d1c>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at NzbDrone.Core.Datastore.Migration.Framework.MigrationController.Migrate (System.String connecti
onString, NzbDrone.Core.Datastore.Migration.Framework.MigrationContext migrationContext) [0x000b5] in <96f52d5e5b95442a89ae3bbf62e59664>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at NzbDrone.Core.Datastore.DbFactory.Create (NzbDrone.Core.Datastore.Migration.Framework.Migration
Context migrationContext) [0x00048] in <96f52d5e5b95442a89ae3bbf62e59664>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:    --- End of inner exception stack trace ---
jul 04 06:02:10 DietPi-rpi mono[1259]:   at NzbDrone.Core.Datastore.DbFactory.Create (NzbDrone.Core.Datastore.Migration.Framework.Migration
Context migrationContext) [0x00136] in <96f52d5e5b95442a89ae3bbf62e59664>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at NzbDrone.Core.Datastore.DbFactory.Create (NzbDrone.Core.Datastore.Migration.Framework.Migration
Type migrationType) [0x00010] in <96f52d5e5b95442a89ae3bbf62e59664>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at NzbDrone.Core.Datastore.DbFactory.RegisterDatabase (NzbDrone.Common.Composition.IContainer cont
ainer) [0x00006] in <96f52d5e5b95442a89ae3bbf62e59664>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at NzbDrone.Host.NzbDroneServiceFactory.Start () [0x00037] in <83751627b1fd4bb1b4a2a3075c439f56>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at NzbDrone.Host.Router.Route (NzbDrone.Host.ApplicationModes applicationModes) [0x0007b] in <8375
1627b1fd4bb1b4a2a3075c439f56>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at NzbDrone.Host.Bootstrap.Start (NzbDrone.Host.ApplicationModes applicationModes, NzbDrone.Common
.EnvironmentInfo.StartupContext startupContext) [0x00047] in <83751627b1fd4bb1b4a2a3075c439f56>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at NzbDrone.Host.Bootstrap.Start (NzbDrone.Common.EnvironmentInfo.StartupContext startupContext, N
zbDrone.Host.IUserAlert userAlert, System.Action`1[T] startCallback) [0x00075] in <83751627b1fd4bb1b4a2a3075c439f56>:0
jul 04 06:02:10 DietPi-rpi mono[1259]:   at NzbDrone.Console.ConsoleApp.Main (System.String[] args) [0x0002f] in <bd424479d84b43e2be7be110e
677f257>:0
jul 04 06:02:10 DietPi-rpi mono[1259]: Press enter to exit...
jul 04 06:02:12 DietPi-rpi systemd[1]: sonarr.service: Main process exited, code=exited, status=1/FAILURE
jul 04 06:02:12 DietPi-rpi systemd[1]: sonarr.service: Unit entered failed state.
jul 04 06:02:12 DietPi-rpi systemd[1]: sonarr.service: Failed with result 'exit-code'.

[MichaIng]

@ezekini

/mnt/dietpi_userdata/sonarr/nzbdrone.db is corrupt, restore from backup if available. See: https://github.com/Sonarr/Sonarr/wiki/FAQ#i-am-getting-an-error-database-disk-image-is-malformed

Damn, indeed a database curruption.
Following the link there is a guide about how to check and dump the database and re-import it: https://techblog.dorogin.com/sqliteexception-database-disk-image-is-malformed-77e59d547c50
It looks like in most cases even the corrupted entries can be dumped and imported again, if the issue is more on a meta level.

• Do a backup first: cp -a /mnt/dietpi_userdata/sonarr/nzbdrone.db /mnt/dietpi_userdata/sonarr/nzbdrone.db.bak

I never really worked with SQLite (MariaDB much more), but AFAIK to enter the SQLite console for the related database file you need to run: sqlite3 /mnt/dietpi_userdata/sonarr/nzbdrone.db ("SQLite shell" can be skipped then, the cmd line tool is installed already)

• From there you can try the steps from the above link.

---

Okay to put things together, IF all table data can be read:

cd /mnt/dietpi_userdata/sonarr
cp -a nzbdrone.db nzbdrone.db.bak
sqlite3 nzbdrone.db
---- Inside SQLite console
.mode insert
.output nzbdrone.sql
.dump
.exit
---- Exits SQLite console
sqlite3 nzbdrone_new.db
---- Inside new SQLite console
.read nzbdrone.sql
.exit
---- Exits SQLite console
rm nzbdrone.sql # cleanup
mv nzbdrone_new.db nzbdrone.db
chown sonarr:sonarr nzbdrone.db
systemctl start sonarr
---- If this works well, all Sonarr settings and status is as before, you could remove the backup as well:
rm nzbdrone.db.bak

[ezekini]

I get an error after

.read nzbdrone.sql:
Error: near line 24918: UNIQUE constraint failed: Commands.Id
Error: near line 24926: UNIQUE constraint failed: Commands.Id

If I can't solve that, I guess I will have to uninstall and reinstall.
Any idea on what could have caused this to avoid it in the future?
Regards

[MichaIng]

@ezekini
Possibly these can be ignored, two table creation commands seem to be doubled, but that might not be an issue.
Just to test another method, not sure if there is any difference thought:

rm nzbdrone_new.db
cat nzbdrone.sql | sqlite3 nzbdrone_new.db

Perhaps you face just the same two errors, but AFAIK all other parts are created, so you could simply try it out by going on with the other steps.

We could also check which table+key it actually is:

sed -n 24918p nzbdrone.sql
sed -n 24926p nzbdrone.sql

---

Any idea on what could have caused this to avoid it in the future?

Disk I/O error, process crash or something, but causes vary very much.
You could check dmesg for any kernel, power or disk I/O errors.
Also full memory could make the system emergency kill a process in a way the a database, that is just written to, gets corrupted. So also verify that there is always enough memory available, as well during some daily cron jobs or other peak situations.

[ezekini]

Ok, I managed to restore the database. Thank you for the info.