Problem with jetty.sh start regression 10.0.6 -> 10.0.7 when using JETTY_USER

[pethers]

Jetty version(s)
10.0.7

Java version/vendor (use: java -version)
Openjdk 17

OS type/version
Ubuntu 21.10

Description
When using JETTY_USER, "jetty.sh start" fails to startup.

How to reproduce?
Setting JETTY_USER and configure as a service /etc/init.d/jetty
Worked with 10.0.6 (changes 2b0161e#diff-2b9f3399552ca99de76591c09421a925e0d63a4c7cea19757ae7a81551993d24 ) and stll works if replace with 10.0.6 version of jetty.sh only.

[joakime]

Setting JETTY_USER and configure as a service /etc/init.d/jetty

We need to know more details here.

You are using init.d, not system.d, right?
Are you using native init.? or the emulated one that system.d provides? or something custom?

Do these steps describe your configuration?

docker run -it ubuntu:21.10 bash

apt update
apt install -y vim
apt install -y curl
apt install -y openjdk-11-jdk-headless

cd /tmp
curl -O https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-home/10.0.7/jetty-home-10.0.7.tar.gz

useradd -m jetty

cd /opt
tar -zxf /tmp/jetty-home-10.0.7.tar.gz
mkdir jetty-base
cd /opt/jetty-base
java -jar /opt/jetty-home-10.0.7/start.jar --module=http,deploy
mkdir /opt/jetty-base/logs
chown -R jetty:jetty /opt/jetty-base/

ln -s /opt/jetty-home-10.0.7/bin/jetty.sh /etc/init.d/jetty

cd /etc/default
vim jetty
   JETTY_HOME=/opt/jetty-home-10.0.7
   JETTY_BASE=/opt/jetty-base
   JETTY_USER=jetty

service jetty start

This shows a result of ...

Starting Jetty: FAILED Mon Oct 18 08:43:12 CDT 2021

I need to investigate further.

[joakime]

📓 Note: if I use the above and simply swap out to using ubuntu:20.04 or ubuntu:18.04 then it works as intended.
There's something about ubuntu:21.10 that's aggravating the jetty.sh or start-stop-daemon in the wrong way.

Using the ubuntu:21.10 docker image and instructions in the last commit, the start-stop-daemon is used.

In Jetty 10.0.7, the RUN_ARGS is ...

-Djetty.home=/opt/jetty-home-10.0.7
-Djava.io.tmpdir=/tmp 
-Djetty.home=/opt/jetty-home-10.0.7 
-Djetty.base=/opt/jetty-base 
--class-path /opt/jetty-base/resources:/opt/jetty-home-10.0.7/lib/logging/slf4j-api-2.0.0-alpha5.jar:/opt/jetty-home-10.0.7/lib/logging/jetty-slf4j-impl-10.0.7.jar:/opt/jetty-home-10.0.7/lib/jetty-servlet-api-4.0.6.jar:/opt/jetty-home-10.0.7/lib/jetty-http-10.0.7.jar:/opt/jetty-home-10.0.7/lib/jetty-server-10.0.7.jar:/opt/jetty-home-10.0.7/lib/jetty-xml-10.0.7.jar:/opt/jetty-home-10.0.7/lib/jetty-util-10.0.7.jar:/opt/jetty-home-10.0.7/lib/jetty-io-10.0.7.jar:/opt/jetty-home-10.0.7/lib/jetty-security-10.0.7.jar:/opt/jetty-home-10.0.7/lib/jetty-servlet-10.0.7.jar:/opt/jetty-home-10.0.7/lib/jetty-webapp-10.0.7.jar:/opt/jetty-home-10.0.7/lib/jetty-deploy-10.0.7.jar 
org.eclipse.jetty.xml.XmlConfiguration 
java.version=11.0.12 
java.version.major=11 
java.version.micro=12 
java.version.minor=0 
java.version.platform=11 
jetty.base=/opt/jetty-base 
jetty.base.uri=file:///opt/jetty-base 
jetty.home=/opt/jetty-home-10.0.7
jetty.home.uri=file:///opt/jetty-home-10.0.7 
jetty.state=/opt/jetty-base/jetty.state 
jetty.webapp.addServerClasses=org.eclipse.jetty.logging.,${jetty.home.uri}/lib/logging/,org.slf4j. 
runtime.feature.alpn=true 
slf4j.version=2.0.0-alpha5 
/opt/jetty-home-10.0.7/etc/jetty-bytebufferpool.xml 
/opt/jetty-home-10.0.7/etc/jetty-threadpool.xml 
/opt/jetty-home-10.0.7/etc/jetty.xml 
/opt/jetty-home-10.0.7/etc/jetty-webapp.xml 
/opt/jetty-home-10.0.7/etc/jetty-deploy.xml 
/opt/jetty-home-10.0.7/etc/jetty-http.xml 
/opt/jetty-home-10.0.7/etc/jetty-started.xml

The RUN_ARGS using Jetty 10.0.6 is ...

-Djetty.home=/opt/jetty-home-10.0.7
-Djetty.base=/opt/jetty-base
-Djava.io.tmpdir=/tmp
-jar /opt/jetty-home-10.0.7/start.jar
jetty.state=/opt/jetty-base/jetty.state
jetty-started.xml

The biggest difference is that we now try hard to avoid forking the JVM in service mode.
Jetty 10.0.6 can fork the JVM depending on the configuration in your JETTY_BASE, Jetty 10.0.7 will never fork.

We want to be on "the no-fork ever" path, so lets see what is going on with this non-LTS ubuntu.

There are some quirky things in RUN_ARGS seen with 10.0.7 (why is -Djetty.home there twice?), but nothing that should prevent it starting, especially with start-stop-daemon.

The execution in 10.0.7 goes into start-stop-daemon line and then fails.

https://github.com/eclipse/jetty.project/blob/jetty-10.0.7/jetty-home/src/main/resources/bin/jetty.sh#L468

https://github.com/eclipse/jetty.project/blob/da8a4553af9dd84080931fa0f8c678cd2d60f3d9/jetty-home/src/main/resources/bin/jetty.sh#L468

Still need to dig deeper.

[pethers]

Have similar behavior, but building a deb package so slightly different setup.

Setting up variables https://github.com/Hack23/cia/blob/master/cia-dist-deb/src/main/config/etc/default/cia
and creating base dir and init.d setup https://github.com/Hack23/cia/blob/master/cia-dist-deb/src/main/unix/scripts/postinst

Strange "jetty.sh run" will actually start jetty.

All works with 10.0.6 version of jetty.sh.

[joakime]

I opened up PR #7009 with an updated jetty.sh that works for me.

Can you try it on your environments too?

Strange "jetty.sh run" will actually start jetty.

That's likely because jetty.sh run doesn't use start-stop-daemon, it just executes java directly.

[joakime]

Seems the start-stop-daemon is far more picky on it's command line options than before.

We chose the set of options we used in start-stop-daemon to work with the widest variety of versions of start-stop-daemon.
The fixes in PR #7009 is likely to break users of very old versions of start-stop-daemon (which I'm ok with doing for Jetty 10+, but not Jetty 9)

[pethers]

Tested the above https://github.com/eclipse/jetty.project/blob/c89866682c14ca1da0ce482eeca0953375b25da8/jetty-home/src/main/resources/bin/jetty.sh and it works as expected 👍