Sonarr 3.0.6.1342 - Certificate validation errors after updating to Mono 5.20.1.34-18

[mreid-tt]

Setup

Package Name: Sonarr
Package Version: 20210717-19

NAS Model: DS916+
NAS Architecture: INTEL Pentium N3710
DSM version: DSM 7.0.1-42218

Expected behavior

Following the update of Mono to version 5.20.1.34-18, Sonarr is restarted. Once restarted, I should be able to check for updates without error from System -> Updates

Actual behavior

On checking for updates I only get an error: Failed to fetch updates
I also note in the System -> Events a number of X509CertificateValidationService and SonarrErrorPipeline errors

Steps to reproduce

1. Update Mono to latest version 5.20.1.34-18
2. Sonarr reboots as a dependent package
3. Errors show in the System -> Events

Package log

Installer log: find logfile /var/log/packages/{package}.log, for DSM<6: /var/packages/{package}/target/var/{package}_installer.log and protocol in Package Center.
You find service log and other log files in /var/packages/{package}/target/var.

nzbdrone_install.log -- https://pastebin.com/5EzrSXca
nzbdrone.log -- https://pastebin.com/QjVGn1mu

Other logs

E.g. /var/log/messages or /var/log/synopkg.log

synopkg.log -- https://pastebin.com/75sn1W1y

Known workaround (see: #5051 (comment))

1. Within DSM, enable SSH service in Control Panel > Terminal & SNMP and click apply
2. Using Terminal (MacOS) connect to the NAS using ssh -l [admin username] [NAS address] or using Putty (Windows) connect to the network address of your NAS
3. Enter the required admin password and press enter
4. Enter the following command: sudo /var/packages/mono/target/bin/cert-sync /etc/ssl/certs/ca-certificates.crt and press enter
5. Enter the required admin password and press enter. When complete you should see the line Import process completed
6. Enter the following command: sudo chmod -R a+rX /usr/share/.mono and press enter
7. Disconnect the SSH session by typing exit and press enter
8. Within DSM, disable the SSH service in Control Panel > Terminal & SNMP and click apply

[bakerboy448]

Not a bug - you need to sync mono's certs

https://community.synology.com/enu/forum/1/post/148065

[hgy59]

a regular installation and update run the following command in service_postinst:

${SYNOPKG_PKGDEST}/bin/cert-sync /etc/ssl/certs/ca-certificates.crt

this is shown in the installer log file like this (upgrade with already uptodate list)

2022/01/04 21:14:28     ===> Step postinst. USER= GROUP= SHARE_PATH=
2022/01/04 21:14:28     Begin save_wizard_variables
2022/01/04 21:14:28     End save_wizard_variables
2022/01/04 21:14:28     Begin service_postinst
Mono Certificate Store Sync - version 5.8.0.108
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 150, your new list has 150
Import process completed.

Importing into BTLS system store:
I already trust 150, your new list has 150
Import process completed.
2022/01/04 21:14:31     End service_postinst
2022/01/04 21:14:31     upgrade mono 5.8.0.108-11 End postinst ret=[0]
2022/01/04 21:14:31     upgrade mono 5.8.0.108-11 Begin postupgrade
2022/01/04 21:14:31     Begin reload_inst_variables
2022/01/04 21:14:31     End reload_inst_variables
2022/01/04 21:14:31     Begin initialize_variables
2022/01/04 21:14:31     End initialize_variables
2022/01/04 21:14:31     ===> Step postupgrade. USER= GROUP= SHARE_PATH=
2022/01/04 21:14:31     upgrade mono 5.8.0.108-11 End postupgrade ret=[0]

If you want to execute this command manually use /var/packages/mono/target/ for SYNOPKG_PKGDEST, i.e. run
/var/packages/mono/target/bin/cert-sync /etc/ssl/certs/ca-certificates.crt

[bakerboy448]

mono 5.8 really isn't supported

mono 5.18 or 5.20 are needed

[hgy59]

@mreid-tt analyzing your log file above, i am missing the service_postinst logs for mono.
probably this is an issue with automatic mono update when installing another package (nzbdrone/sonarr).

Probably you can resolve this, by manually installing (updating) the downloaded mono spk with identical version (5.20.1.34-18).

[hgy59]

mono 5.8 really isn't supported

mono 5.18 or 5.20 are needed

I know, I know, this is only a sample installation log, that does not depend on mono version.

[hgy59]

It seems to be an issue on DSM 7

tried the same with mono 5.20.1.34-18 on DS218+ with DSM 7.0.1

2022/01/04 21:45:35 System.UnauthorizedAccessException: Access to the path "/usr/share/.mono" is denied.

here an extract of /var/log/packages/mono.log:

022/01/04 21:45:35     ===> Step postinst. USER= GROUP= SHARE_PATH=
2022/01/04 21:45:35     Begin save_wizard_variables
2022/01/04 21:45:35     End save_wizard_variables
2022/01/04 21:45:35     Begin syno_sync_var_folder
2022/01/04 21:45:35     End syno_sync_var_folder
2022/01/04 21:45:35     Begin service_postinst
2022/01/04 21:45:35     Mono Certificate Store Sync - version 5.20.1.34
2022/01/04 21:45:35     Populate Mono certificate store from a concatenated list of certificates.
2022/01/04 21:45:35     Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.
2022/01/04 21:45:35     Importing into legacy system store:
2022/01/04 21:45:35     I already trust 0, your new list has 138
2022/01/04 21:45:35     Warning: Could not import CN=ACCVRAIZ1, OU=PKIACCV, O=ACCV, C=ES
2022/01/04 21:45:35     System.UnauthorizedAccessException: Access to the path "/usr/share/.mono" is denied.
2022/01/04 21:45:35       at System.IO.Directory.CreateDirectoriesInternal (System.String path) [0x0005e] in <a7386717e97e435ea6394b4446fbe4ba>:0
2022/01/04 21:45:35       at System.IO.Directory.CreateDirectory (System.String path) [0x0008f] in <a7386717e97e435ea6394b4446fbe4ba>:0
2022/01/04 21:45:35       at System.IO.DirectoryInfo.Create () [0x00000] in <a7386717e97e435ea6394b4446fbe4ba>:0
2022/01/04 21:45:35       at (wrapper remoting-invoke-with-check) System.IO.DirectoryInfo.Create()
2022/01/04 21:45:35       at System.IO.Directory.CreateDirectoriesInternal (System.String path) [0x00036] in <a7386717e97e435ea6394b4446fbe4ba>:0
2022/01/04 21:45:35       at System.IO.Directory.CreateDirectory (System.String path) [0x0008f] in <a7386717e97e435ea6394b4446fbe4ba>:0
2022/01/04 21:45:35       at System.IO.DirectoryInfo.Create () [0x00000] in <a7386717e97e435ea6394b4446fbe4ba>:0
2022/01/04 21:45:35       at (wrapper remoting-invoke-with-check) System.IO.DirectoryInfo.Create()
2022/01/04 21:45:35       at System.IO.Directory.CreateDirectoriesInternal (System.String path) [0x00036] in <a7386717e97e435ea6394b4446fbe4ba>:0
2022/01/04 21:45:35       at System.IO.Directory.CreateDirectory (System.String path) [0x0008f] in <a7386717e97e435ea6394b4446fbe4ba>:0
2022/01/04 21:45:35       at Mono.Security.X509.X509Store.CheckStore (System.String path, System.Boolean throwException) [0x00020] in <0a5b7d3822b84f2a96c7f10697a24fe9>:0
2022/01/04 21:45:35       at Mono.Security.X509.X509Store.Import (Mono.Security.X509.X509Certificate certificate) [0x00000] in <0a5b7d3822b84f2a96c7f10697a24fe9>:0
2022/01/04 21:45:35       at Mono.Tools.CertSync.ImportToStore (Mono.Security.X509.X509CertificateCollection roots, Mono.Security.X509.X509Store store) [0x00050] in <a7d568c231944f48bd7618f2ee421b9       4>:0

PS: it is time to port sonarr to .net!

[mreid-tt]

Hey @hgy59, thanks so much for the analysis. This does seem to be a DSM7 issue since the suggestion below did not work:

Probably you can resolve this, by manually installing (updating) the downloaded mono spk with identical version (5.20.1.34-18).

I was able to manually import the certificates by putting a sudo before your command below:

If you want to execute this command manually use /var/packages/mono/target/ for SYNOPKG_PKGDEST, i.e. run /var/packages/mono/target/bin/cert-sync /etc/ssl/certs/ca-certificates.crt

The output looked like this -- https://pastebin.com/ZHgssSSn

Once this was completed, Sonarr is once again fully functional.

PS: it is time to port sonarr to .net!

I agree, but I think I recall one of the devs saying that Sonarr is unlikely to do that anytime soon.

EDIT

This is also evident in the source repository since they seem to have pushed this to v4 consideration based on Sonarr/Sonarr#948

[bakerboy448]

Sonarr will not be moving to .net anytime soon

Based on the time it took for v3 to replace v2 I would not expect Sonarr in .net for several years at best. Maybe I'll be wrong 🤷‍♂️

Sonarr on .Net also already exists for internal testing.
https://github.com/Sonarr/Sonarr/tree/widowmaker

For this SSL issue it seems that SynoCommunity's mono package has a bug and fails to sync certs upon install. This should be corrected as soon as it can given all the support issues it's causing.

[bakerboy448]

Any timeframe to fix this bug?

we're up to a lot of users on a daily basis posting about this bug both for Sonarr help and on the various Synology sites with the SynoCommunity mono package.

[mreid-tt]

I guess I could summarise the workaround for the community to follow in the meantime:

1. Within DSM, enable SSH service in Control Panel > Terminal & SNMP and click apply
2. Using Terminal (MacOS) connect to the NAS using ssh -l [admin username] [NAS address] or using Putty (Windows) connect to the network address of your NAS
3. Enter the required admin password and press enter
4. Enter the following command: sudo /var/packages/mono/target/bin/cert-sync /etc/ssl/certs/ca-certificates.crt and press enter
5. Enter the required admin password and press enter. When complete you should see the line Import process completed
6. Enter the following command: sudo chmod -R a+rX /usr/share/.mono and press enter
7. Disconnect the SSH session by typing exit and press enter
8. Within DSM, disable the SSH service in Control Panel > Terminal & SNMP and click apply

Once complete the errors in Sonarr should disappear on their own in a few minutes.

EDIT: Thanks to the contribution from @mmdriley below, step 6 was added to support DSM 7.1.

[Stanzilla]

sudo /volume1/@appstore/mono/bin/cert-sync /etc/ssl/certs/ca-certificates.crt for DSM 7, I think. Did not fix it for me though.

[bakerboy448]

DSM 7.1 for those this isn't fixing by chance?

it seems DSM 7.1 is causing a lot of package issues then.

[Stanzilla]

DSM 7.1 for those this isn't fixing by chance?

it seems DSM 7.1 is causing a lot of package issues then.

Yep, 7.1.

[Patrick010]

Another 7.1 victim here.

[kunude]

Mono 5.20.1.34-17

[Patrick010]

What about it

[Patrick010]

Mono 5.20.1.34-17

I downgraded to this version and it works now. Maybe you could elaborate a bit more on your comments ;)

[Stanzilla]

I can confirm this fixed it for me as well, you can download older versions from https://synocommunity.com/package/mono

[angelo-melis]

I have the same issue on DSM7.1. I did downgrade mono to Mono 5.20.1.34-17, and ran the cert-sync command. But still the same problem. Could someone explain what they did to get it to work on 7.1?

[bakerboy448]

It generally seems DSM7.1 is not supported for fresh installs and mono's certs cannot be synced.

[Patrick010]

I have the same issue on DSM7.1. I did downgrade mono to Mono 5.20.1.34-17, and ran the cert-sync command. But still the same problem. Could someone explain what they did to get it to work on 7.1?

I basically only uninstalled 5.20.1.34-18 and manually installed 17. Didnt run the cert-sync.

[angelo-melis]

Thanks for the response, bakerboy448 and Patrick010. I also did an uninstall and manually install, but still couldn't get it to work. I switched to Docker for now and got it working.

[mmdriley]

I fixed this on DSM 7.1 by updating permissions for Mono's machine certificate store to allow users like sc-nzbdrone to read:

$ sudo chmod -R a+rX /usr/share/.mono

[cesarfd]

I fixed this on DSM 7.1 by updating permissions for Mono's machine certificate store to allow users like sc-nzbdrone to read:

$ sudo chmod -R a+rX /usr/share/.mono

I don't even have this folder, the most similar would be something like /usr/local/mono/share/mono-2.0/mono?

[mmdriley]

If that folder doesn't exist, you may need to run the cert-sync commands listed above to create it.

The local machine store is under ${CommonApplicationData}/.mono, which on Linux is /usr/share. (ref)

[cesarfd]

Thank you! That did the trick.

[Agoris010]

I updated to DSM 7.1-42661 Update 2 just now and had the same problem!

After

sudo /volume1/@appstore/mono/bin/cert-sync /etc/ssl/certs/ca-certificates.crt

and

$ sudo chmod -R a+rX /usr/share/.mono

it worked fine again, thank you very much! :-)

Worked like a charme. Thx

[hanshendrix]

Thankx very much, this is the solution, now i'm very happy.

[NMe84]

How is this package still broken half a year later if the fix is this simple? I just got the same error again after having solved it months ago...

[hackgrid]

I guess #5070 has to be completed and merged first...?

[mmdriley]

Right, the workaround in my comments above is only "simple" because we can run it as root. Fixing this for real requires some nontrivial changes (keeping compatibility/upgrades in mind) to match DSM7's permission model.

[tonyellow]

chmod -R a+rX /usr/share/.mono worked for me too!

[Version3Synology]

Thanks a lot, both commands :

• /volume1/@appstore/mono/bin/cert-sync /etc/ssl/certs/ca-certificates.crt
• chmod -R a+rX /usr/share/.mono

did it for me.

Was wondering why Radarr was able to reach indexers but not Sonarr, this thread brought the answer.

[TheGianni]

I fixed this on DSM 7.1 by updating permissions for Mono's machine certificate store to allow users like sc-nzbdrone to read:

$ sudo chmod -R a+rX /usr/share/.mono

You're the man! I already thought I'd have to switch everything to docker... Thanks!

[nieroivan]

Worked for me as well! Thank you guys!

[NMe84]

Right, the workaround in my comments above is only "simple" because we can run it as root. Fixing this for real requires some nontrivial changes (keeping compatibility/upgrades in mind) to match DSM7's permission model.

Is anyone actually looking into those nontrivial changes? Is there anything the community can do to help? A DSM update just triggered the Sonarr package to get updated again, which once again broke certificates. It's of course simple to fix whenever it breaks but at this point it's getting annoying.

[we-are-borg]

For me chmod -R a+rX /usr/share/.mono is not working “no such file or directory” i’m on DSM 7.1.1-42962 Update 2 has the location changed.

[hackgrid]

You need to execute both commands mentioned further up!
After the latest update I also needed to do cert-sync and chmod again, but it worked fine immediately after that! :-)

[jocamero]

Update: this was an issue again today. The same fix worked (again).

[mreid-tt]

@mreid-tt regarding the mono certificate (and mono DSM 7) issues. I propose to create and install a script that does what you summarized in #5051 (comment).

This script could then be run (as root) with a task in DSM. The task could be executed on demand (or at DSM boot-up) and will avoid the need of an ssh login.

I prefer to add such a script to the mono package (there is still a pending PR #4669)

@hgy59, continuing this discussion here... how would you add a script to run as root exactly? In my reading online I get the impression that you cannot run scripts as root in DSM 7 unless your package is signed by Synology. Is there an approach you had in mind for me to try?

[bakerboy448]

This script could then be run (as root) with a task in DSM.

^ User would need to configure it

[mmdriley]

DSM7 made it hard to run things as root, and for good reasons. I don't think we're going to get anywhere good by building an experience that depends on updating files outside SynoCommunity packages.

I feel like two good options here are:

1. Sonarr manages its own cert store under a filesystem path to which its user has full access and convinces Mono to use it. Unfortunately doesn't seem to be easy on the Mono side, see Allow environment override of cert store mono/mono#6388.
2. The SynoCommunity Mono package takes more responsibility for maintaining the cert store Mono uses, then Sonarr and other packages benefit. Interestingly it looks like an attempt was made at this in mono: relocate special folders #3154 but... it's not obvious the Mono that Sonarr depends on is actually built with that patch? I'm sure there's a good explanation but I haven't dug in to find it.

Continuing with (2), it also looks like the Mono package already tries to update its certificate store, though only just after installation:

spksrc/spk/mono_58/src/service-setup.sh

Lines 1 to 5 in 12ebc6c

	service_postinst ()
	{
	# Sync ca certificates
	${SYNOPKG_PKGDEST}/bin/cert-sync /etc/ssl/certs/ca-certificates.crt
	}

If we go with the Mono-package-owns-its-cert-store path, we may want to change that command to run on postinstall and every service start.

[Stanzilla]

Obsolete with v4 and the move to .NET Core anyway though, no?

[bakerboy448]

No ETA as to when v4 will be officially stable.

V3 was in beta for several years

[mreid-tt]

Packaged Sonarr v4 beta (#5524) is ready to go. Just waiting on the admins to do a final review so it can be published for those who are willing to test it.

[jasonmp85]

Is this really the same issue going back a whole year?

[mreid-tt]

Greetings everyone, I am pleased to inform you that we have identified the issue with Mono and have created a new pull request (#5604) to fix it. We have tested the solution and it has been working well so far. As a result, the latest version of Mono (v5.20.1.34-19) is now available for download in the Package Center, or you can manually download it from the repository. Thank you for your patience and understanding.