Permissions on /var/lib/jenkins/plugins/* prevent newer Jenkins LTS releases starting

[moon127]

The charm installs plugins to /var/lib/jenkins/plugins and the .hp/.jpi and the unpacked plugins dirs are owned root:root

This seemed fine up to (tested) 2.277.1 but on upgrading to 2.303.1 (current upstream stable LTS) this prevents plugins installing and jenkins starting:

java.lang.IllegalArgumentException: File parameter 'file is not writable: '/var/lib/jenkins/plugins/bouncycastle-api.jpi'
at org.apache.commons.io.FileUtils.requireCanWrite(FileUtils.java:2619)
at org.apache.commons.io.FileUtils.openOutputStream(FileUtils.java:2447)
at org.apache.commons.io.FileUtils.openOutputStream(FileUtils.java:2415)
at org.apache.commons.io.FileUtils.copyToFile(FileUtils.java:1042)
at org.apache.commons.io.FileUtils.copyInputStreamToFile(FileUtils.java:952)
at org.apache.commons.io.FileUtils.copyURLToFile(FileUtils.java:1069)
at hudson.PluginManager.copyBundledPlugin(PluginManager.java:1059)
at hudson.PluginManager.loadPluginsFromWar(PluginManager.java:651)
at hudson.PluginManager.loadDetachedPlugins(PluginManager.java:749)
at hudson.LocalPluginManager.loadBundledPlugins(LocalPluginManager.java:79)
at hudson.PluginManager$1$1.run(PluginManager.java:407)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1160)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused: org.jvnet.hudson.reactor.ReactorException
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
at jenkins.InitReactorRunner.run(InitReactorRunner.java:50)
at jenkins.model.Jenkins.executeReactor(Jenkins.java:1193)
at jenkins.model.Jenkins.(Jenkins.java:991)
at hudson.model.Hudson.(Hudson.java:86)
at hudson.model.Hudson.(Hudson.java:82)
at hudson.WebAppMain$3.run(WebAppMain.java:298)
Caused: hudson.util.HudsonFailedToLoad
at hudson.WebAppMain$3.run(WebAppMain.java:315)

The charm should likely ensure jenkins:jenkins ownership for any plugin packages or unpacks on new installs or upgrades?!?

[mthaddon]

We do already have something that should be doing this https://github.com/jenkinsci/jenkins-charm/blob/master/lib/charms/layer/jenkins/plugins.py#L92 - can you provide a bit more detail on when this happened and ideally how to reproduce?

[moon127]

I observed on manually installing the 2.303.1 on top of the charm deployed 2.277.1 "bundle" as the trace was all the web UX displayed, this was also written out in /var/log/jenkins/jenkins.log.

Checking on disk everything indeed looked to be root:root at that point under /var/lib/jenkins/plugins, and the chown -R jenkins: * allowed it to restart successfully and resume operations.

It's possible I'd triggered a plugins change via the charm config at around the same time and tickled something, particularly as runnign with remove-unlisted-plugins=true which has issues and is known to cause a failure and revert of plugins to backup.