Lots of config errors on jenkins startup #2608

jfchevrette · 2018-03-15T00:08:58Z

Yesterday an update to OpenShift went in to the the latest CVEs [1]. That update made secrets and configmaps mounts go read-only.

Presumably on startup jenkins tries to update the files mounted from configmap before it copies it to /var/lib/jenkins. That doesn't work anymore. The file from the configmap mount should be copied elsewhere and updated there.

[1] kubernetes/kubernetes#60814

OPENSHIFT_JENKINS_JVM_ARCH is set to i686 so using 32 bit Java
mkdir: cannot create directory '/var/lib/jenkins/logs': File exists
No resources found.
No resources found.
Generating kubernetes-plugin configuration (/opt/openshift/configuration/config.xml.tpl) ...
/usr/libexec/s2i/run: line 111: /opt/openshift/configuration/config.xml: Read-only file system
Generating kubernetes-plugin credentials (/var/lib/jenkins/credentials.xml.tpl) ...
/usr/libexec/s2i/run: line 123: /opt/openshift/configuration/credentials.xml: Read-only file system
Copying Jenkins configuration to /var/lib/jenkins ...
rm: cannot remove '/opt/openshift/configuration/config.xml.tpl': Read-only file system
rm: cannot remove '/opt/openshift/configuration/credentials.xml.tpl': Read-only file system
rm: cannot remove '/opt/openshift/configuration/hudson.tasks.Maven.xml': Read-only file system
rm: cannot remove '/opt/openshift/configuration/io.fabric8.jenkins.openshiftsync.GlobalPluginConfiguration.xml': Read-only file system
rm: cannot remove '/opt/openshift/configuration/jenkins.plugins.nodejs.tools.NodeJSInstallation.xml': Read-only file system
rm: cannot remove '/opt/openshift/configuration/org.jenkinsci.main.modules.sshd.SSHD.xml': Read-only file system
rm: cannot remove '/opt/openshift/configuration/org.jenkinsci.plugins.updatebot.GlobalPluginConfiguration.xml': Read-only file system
rm: cannot remove '/opt/openshift/configuration/scriptApproval.xml': Read-only file system
Copying 143 Jenkins plugins to /var/lib/jenkins ...
Creating initial Jenkins 'admin' user ...
sed: can't read /var/lib/jenkins/users/admin/config.xml: No such file or directory
Running from: /usr/lib/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Mar 15, 2018 12:01:05 AM Main deleteWinstoneTempContents
WARNING: Failed to delete the temporary Winstone file /tmp/winstone/jenkins.war
Mar 15, 2018 12:01:05 AM org.eclipse.jetty.util.log.Log initialized
INFO: Logging initialized @1221ms to org.eclipse.jetty.util.log.JavaUtilLog
Mar 15, 2018 12:01:06 AM winstone.Logger logInternal
INFO: Beginning extraction from war file
Mar 15, 2018 12:01:06 AM org.eclipse.jetty.server.handler.ContextHandler setContextPath
WARNING: Empty contextPath
Mar 15, 2018 12:01:06 AM org.eclipse.jetty.server.Server doStart
INFO: jetty-9.4.z-SNAPSHOT
Mar 15, 2018 12:01:07 AM org.eclipse.jetty.webapp.StandardDescriptorProcessor visitServlet
INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
Mar 15, 2018 12:01:08 AM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: DefaultSessionIdManager workerName=node0
Mar 15, 2018 12:01:08 AM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: No SessionScavenger set, using defaults
Mar 15, 2018 12:01:08 AM org.eclipse.jetty.server.session.HouseKeeper startScavenging
INFO: Scavenging every 660000ms
Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
Mar 15, 2018 12:01:09 AM org.eclipse.jetty.server.handler.ContextHandler doStart
INFO: Started w.@112aa7c{/,file:///var/lib/jenkins/war/,AVAILABLE}{/var/lib/jenkins/war}
Mar 15, 2018 12:01:09 AM org.eclipse.jetty.server.AbstractConnector doStart
INFO: Started ServerConnector@1cb6996{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
Mar 15, 2018 12:01:09 AM org.eclipse.jetty.server.Server doStart
INFO: Started @5077ms
Mar 15, 2018 12:01:09 AM winstone.Logger logInternal
INFO: Winstone Servlet Engine v4.0 running: controlPort=disabled
Mar 15, 2018 12:01:11 AM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization

The text was updated successfully, but these errors were encountered:

kbsingh · 2018-03-15T02:21:01Z

we do need to get this done, but might not be a SEV1 - i think it needs a bit more of a deep dive from build team to scope up.

chmouel · 2018-03-15T10:31:18Z

added to our backlog, @pradeepto feel free to prioritize,

stevengutz · 2018-03-15T13:49:32Z

@chmouel Let's assume this is high priority (probably #3 in our current list of issues). It's not really breaking anything but it is polluting log files at a time when SD needs them to be concise to diagnose real issues.

pradeepto · 2018-03-20T02:44:59Z

Copy pasting from MM.

@lordofthejars mentioned :

@sthaha I have been investigating #2608 my bets is that we can remove the rm without any problem. The reason that now is failing is that kubernetes changed the permissions of mount path of secret key to just readable for this reason now it is failing.So why I think that we can remove it? Basically because notice thatthecp from /opt/openshift to JENKINS_HOME always occurs and it is with-f so any change on /opt/.... will be copied to JENKINS_HOME

sthaha · 2018-03-20T05:23:37Z

@jfchevrette

That update made secrets and configmaps mounts go read-only.

Is there a commit you can point me to that made this change? It will be quite helpful to the future maintainers if I can add the link to the commit in the fix that I am making.

sthaha · 2018-03-20T05:45:39Z

Status update

I have a PR#20 in progress that fixes this issue with rm -rf but I think the issue is much more than that since I see that the same mount point is being used to store expanded templates - see: run file which I hope to address in a separate commit but part of the same PR#20

sthaha · 2018-03-20T06:22:05Z

@jfchevrette did we onboard any users after this change to the mount point was made? I wonder if the build succeeds or even jenkins starts for them at all since IIUC credentials.xml and config.xml won't be generated based on line 111 and line 123 of run

sthaha · 2018-03-20T13:13:53Z

Status update

After reading the code a bit more I think I now understand how this is all supposed to work. I am not sure if this has been documented anywhere. I will add the doc to the PR.

Existing implementation

As Jenkins bootstraps one of the things it does is to sync the configuration information from the configmap,

expand the templates (*.tmpl) in the configuration/ dir; generates config.xml, credentials.xml
copy the files to the jenkins-home dir (PV)

To ensure that these steps aren't repeated in the next boot, it deletes all files from the configuration dir (rm -rf ${img_config_dir}). This ensures that configuration is regenerated only when there is a change to the configmap as the it won't be empty anymore.

ReadOnly mount breaks the above

By making configuration readonly, for new users, it not only breaks the optimisation but also the expansion of templates which is critical and thus resulting in missing config.xml and credentials.xml. I think jenkins might autogenerate a config.xml based on defaults but I am guessing it would be broken and the builds shouldn't run due to missing credentials.

Solution

A solution to this would be to figure out a way to detect if the configmap has been updated or not and I am thinking of ...

adding a flag in the form of a file with some content say timestamp at the time of generation of the configmap
Jenkins diffs its copy of the timestamp (initially empty; stored in jenkins-home) with the one in the configuration dir
if it detects a change, it generates all configurations and updates the timestamp

@aslakknutsen @jfchevrette any thoughts ? Is there a simpler/better way to figure out if the configmap has been updated?

jfchevrette · 2018-03-20T13:22:43Z

@sthaha kubernetes/kubernetes#58720

aslakknutsen · 2018-03-25T12:27:07Z

@sthaha A ConfigMap has a Revision id you could use to detect change.

chmouel · 2018-03-27T18:31:35Z

Working on this as well here #2749

sthaha · 2018-04-03T06:32:48Z

Reopening as the issue isn't closed. Somehow the issues get automatically closed when the PR is merged to master.

joshuawilson · 2018-04-03T18:01:13Z

@sthaha if the PR has fixes #[issue-number] then it closes it when it gets merged.

sthaha · 2018-04-03T23:49:09Z

@joshuawilson Thanks. The PR does fix the issue but I kept it open to indicate that it isn't released to prod. I guess I will track that elsewhere.

joshuawilson · 2018-04-04T02:27:41Z

@sthaha you should track it in https://github.com/openshiftio/openshift.io/projects/3
Go over about 15 columns to Implementation Review, that is where things go when they are completed but not "done".

jfchevrette added team/build-cd SEV1-urgent labels Mar 15, 2018

aslakknutsen added type/bug area/jenkins labels Mar 15, 2018

kbsingh added SEV3-medium and removed SEV1-urgent labels Mar 15, 2018

sthaha assigned sthaha and lordofthejars Mar 16, 2018

lordofthejars mentioned this issue Mar 19, 2018

Lots of config errors on jenkins startup fabric8-jenkins/jenkins-openshift-base#19

Closed

sthaha mentioned this issue Mar 20, 2018

Do not delete configs on a readonly mount fabric8-jenkins/jenkins-openshift-base#20

Merged

sthaha unassigned lordofthejars Mar 20, 2018

krishnapaparaju mentioned this issue Mar 28, 2018

Sprint plan for OSIO Build: # 147 #2779

Closed

33 tasks

sthaha closed this as completed in fabric8-jenkins/jenkins-openshift-base#20 Mar 30, 2018

sthaha mentioned this issue Apr 3, 2018

Update Jenkins base image to v4dd3216 fabric8io/openshift-jenkins-s2i-config#156

Merged

sthaha reopened this Apr 3, 2018

sthaha closed this as completed Apr 3, 2018

rupalibehera mentioned this issue Nov 26, 2018

Switch to using upstream Jenkins Master fabric8io/openshift-jenkins-s2i-config#215

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lots of config errors on jenkins startup #2608

Lots of config errors on jenkins startup #2608

jfchevrette commented Mar 15, 2018

kbsingh commented Mar 15, 2018

chmouel commented Mar 15, 2018

stevengutz commented Mar 15, 2018

pradeepto commented Mar 20, 2018

sthaha commented Mar 20, 2018

sthaha commented Mar 20, 2018

sthaha commented Mar 20, 2018 •

edited

Loading

sthaha commented Mar 20, 2018

jfchevrette commented Mar 20, 2018

aslakknutsen commented Mar 25, 2018

chmouel commented Mar 27, 2018

sthaha commented Apr 3, 2018

joshuawilson commented Apr 3, 2018

sthaha commented Apr 3, 2018

joshuawilson commented Apr 4, 2018

Lots of config errors on jenkins startup #2608

Lots of config errors on jenkins startup #2608

Comments

jfchevrette commented Mar 15, 2018

kbsingh commented Mar 15, 2018

chmouel commented Mar 15, 2018

stevengutz commented Mar 15, 2018

pradeepto commented Mar 20, 2018

sthaha commented Mar 20, 2018

sthaha commented Mar 20, 2018

Status update

sthaha commented Mar 20, 2018 • edited Loading

sthaha commented Mar 20, 2018

Status update

Existing implementation

ReadOnly mount breaks the above

Solution

jfchevrette commented Mar 20, 2018

aslakknutsen commented Mar 25, 2018

chmouel commented Mar 27, 2018

sthaha commented Apr 3, 2018

joshuawilson commented Apr 3, 2018

sthaha commented Apr 3, 2018

joshuawilson commented Apr 4, 2018

sthaha commented Mar 20, 2018 •

edited

Loading