passenger-config fails with "too long unix socket path" on Ubuntu 18.04 LTS

[bpfh]

Issue report

When running Passenger on hosts upgraded from Ubuntu 16.04 LTS to 18.04 LTS, passenger-config fails with

Traceback (most recent call last):
	9: from /usr/bin/passenger-config:37:in `<main>'
	8: from /usr/lib/ruby/vendor_ruby/phusion_passenger/config/main.rb:79:in `run!'
	7: from /usr/lib/ruby/vendor_ruby/phusion_passenger/config/restart_app_command.rb:46:in `run'
	6: from /usr/lib/ruby/vendor_ruby/phusion_passenger/config/restart_app_command.rb:144:in `select_app_group_name'
	5: from /usr/lib/ruby/vendor_ruby/phusion_passenger/config/restart_app_command.rb:174:in `select_app_group_name_interactively'
	4: from /usr/lib/ruby/vendor_ruby/phusion_passenger/config/restart_app_command.rb:164:in `query_group_names'
	3: from /usr/lib/ruby/vendor_ruby/phusion_passenger/config/restart_app_command.rb:255:in `query_pool_xml'
	2: from /usr/lib/ruby/vendor_ruby/phusion_passenger/admin_tools/instance.rb:95:in `http_request'
	1: from /usr/lib/ruby/vendor_ruby/phusion_passenger/admin_tools/instance.rb:95:in `new'
/usr/lib/ruby/vendor_ruby/phusion_passenger/admin_tools/instance.rb:95:in `initialize': too long unix socket path (116bytes given but 108bytes max) (ArgumentError)

Question 1: What is the problem?

• What is the expected behavior?

Running passenger-config is possible.

• What is the actual behavior?

passenger-config fails with the above stack trace. Debugging shows that the UNIX socket path Passenger tries to use is something like

/tmp/systemd-private-673f3b327552487c9d5cc4d934decaf-apache2.service-GfIBRc/tmp/passenger.UqAQsAS/agents.s/core_api

which is over the system limit.

• How can we reproduce it?

I have not tried on a vanilla 18.04, but there seems to be additional evidence here, with no solution:

https://stackoverflow.com/questions/70012839/phusion-passenger-status-crashing-with-too-long-socket-name

Your answer:

Question 2: Passenger version and integration mode:

Your answer:

Ubuntu libapache2-mod-passenger 1:6.0.12-1~bionic1

Question 3: OS or Linux distro, platform (including version):

Your answer:

Ubuntu 18.04.6 LTS

Question 4: Passenger installation method:

Your answer:

4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[] RubyGems + Gemfile
[ ] RubyGems, no Gemfile
[x] Phusion APT repo
[ ] Phusion YUM repo
[ ] OS X Homebrew
[ ] source tarball
[ ] Other, please specify:

Question 5: Your app's programming language (including any version managers) and framework (including versions):

Your answer:

System Ruby is ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]

Question 6: Are you using a PaaS and/or containerization? If so which one?

Your answer:

No, plain VM

Question 7: Anything else about your setup that we should know?

Your answer:

As mentioned before, the system was upgraded from Ubuntu 16.04 LTS.

[CamJN]

Yes, this is because the path generated by the systemd private tmp feature is 80 chars long, which only leaves 27 chars for an application to use, and passenger uses 35 chars in its path to its socket.

Passenger's part of the path is: passenger.UqAQsAS/agents.s/core_api

The first part of the path identifies that passenger made the directory, and has a random segment to prevent symlink attacks, then the rest is just the path to the socket.

There are several ways to deal with this, such as:

• setting a different passenger instance directory path that doesn't involve /tmp
• turning off the systemd private tmp feature for apache
• shortening the apache2 service name by 8 chars so that the systemd generated path is only 72 chars long
• Passenger changes its path to be shorter, but this is a breaking change so we're a bit reluctant to do it.

[bpfh]

Thanks for the pointers @CamJN !

For posterity, I resolved this by turning off the systemd private tmp feature by commenting out the line

PrivateTmp=true

in /etc/systemd/system/multi-user.target.wants/apache2.service and then executing

$ sudo systemctl daemon-reload
$ sudo systemctl restart apache2

Now passenger-control has been restored to its former glory.

[coopeu]

How to fix this same error /usr/lib/ruby/vendor_ruby/phusion_passenger/admin_tools/instance.rb:
`initialize': too long unix socket path (116bytes given but 108bytes max) (ArgumentError)
of an Apache/2.4.51 (Debian 11 BullsEye) Phusion_Passenger/6.0.12) Ruby 3.1.0 and Rails 7.0.0
/etc/systemd/user/sockets.target.wants not including apache2.service file

[bpfh]

@coopeu, I suspect following the instructions in #2397 (comment) would work for Debian also.

[coopeu]

Maybe the passenger error origin is different:
The detailed diagnostic is: Initialize language runtime — 0.2s
The backtrace is: in 'bool Passenger::SpawningKit::HandshakePerform::checkCurrentState()' (Perform.h:238) ...
The environment variables are:
Environment variables
LANGUAGE=en_US
APACHE_RUN_DIR=/var/run/apache2
APACHE_PID_FILE=/var/run/apache2/apache2.pid
JOURNAL_STREAM=8:264760
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
INVOCATION_ID=3b7132069baf4f2f8fd3a2773ca024ea
APACHE_LOCK_DIR=/var/lock/apache2
LANG=C
APACHE_RUN_USER=www-data
APACHE_RUN_GROUP=www-data
APACHE_LOG_DIR=/var/log/apache2
PWD=/
PASSENGER_USE_FEEDBACK_FD=true
SERVER_SOFTWARE=Apache/2.4.51 (Debian) Phusion_Passenger/6.0.12

I hope this help to identify the error and to find the solution. Thanks in advance!

[SeanSith]

I'm confused why this got closed, honestly? This, to me, is a bug that needs to be fixed, not band-aided by adjusting apache2's systemd file. This package, including the one compatible with Ubuntu 20.04 is effectively broken straight out of the box.

@CamJN is there any chance that Phusion would take this back to the drawing board or raise an issue upstream with systemd?

[dlundgren]

For Ubuntu 18+ I'd recommend doing this instead of altering the systemd file

• Create /usr/lib/tmpfiles.d/passenger.conf with d /var/run/passenger-instreg root root
• Set PassengerInstanceRegistryDir /var/run/passenger-instreg or passenger_instance_registry_dir /var/run/passenger-instreg as appropriate for your apache/nginx configuration.

This should work ok as the passenger tooling looks there as one of it's default paths for the instance registry

[CamJN]

That sounds like a great approach, i'll see if i can add it to the .deb packages.

[dlundgren]

The correct settings for /usr/lib/tmpfiles.d/passenger.conf is d /var/run/passenger-instreg 0755 root root -

[CamJN]

6.0.16 packages are out now and have the mitigation for this.

[saiqulhaq-hh]

6.0.16 causing 4 hours down in my backend because I don't know that there was issue on passenger
passenger can't create /var/run/passenger-instreg directory

[dlundgren]

Due to the way that the tmpfiles works, the system needs to be rebooted in order for the folder to be created automatically.

The package installation should probably try to create that folder if it doesn't already exist. This should help prevent the need to reboot.

[saiqulhaq-hh]

I use Kubernetes
there is no Dockerfile changes
have rebuilt the image few times already before I found out that the issue is on /var/run/

[CamJN]

You can run systemd-tmpfile --create to apply the change without rebooting, in fact, in my testing that is run by dpkg automatically when the new passenger is installed.

[CamJN]

In a docker/kubernetes type setup systemd may be missing, so it probably won't create the dir for you.

[CamJN]

I had to deal with that over in phusion/passenger-docker#352 by creating the dir manually in the passenger-docker images.

[saiqulhaq-hh]

I think 6.0.16 should be a new major version
i did the same on my Dockerfile
had to add RUN mkdir /var/run/passenger-instreg to make it run

[bashcoder]

Just wanted to document that this exact same issue is present with nginx running on Amazon Linux 2023 and Passenger v 6.0.18. The same remedy works in this environment. Add the directory and add this directive to /usr/conf/nginx.conf:

passenger_instance_registry_dir /var/run/passenger-instreg;