Run multiple processes in a container

[xr09]

Description

Due to the implementation details of Wazuh (running multiple services) and in order to simplify the deployment it is required to define a methodology to start multiple processes on the same container and still behave as expected in a container-enabled environment.

We're currently using phusion/baseimage as base image which ships my_init as a process manager, but with the current Docker rework we need to take a step back and rethink this model.

Behavior requirements:

• Able to run multiple processes without a full init system (systemd)
• No auto-restart, no self-healing
• Forward signals to subprocesses
• Container should die if any of the child processes crashes
• Able to consolidate subprocesses logs to stdout (the Docker way)

Tasks

• Research existing implementations to achieve a multi-process container
• Experiment how wazuh-manager and wazuh-api behave under these solutions

Approaches to try:

• Supervisord
• Bash script
• tini + bash
• S6 Overlay
• runit

[xr09]

Supervisor

One popular solution to the multi-process problem is using Supervisor as the container entry point.

The container process tree would be roughly like this:

supervisord
├── wazuh-api
└── wazuh-manager

It is even possible to listen for STOPPED, EXITED and FATAL events which makes possible to hook up a shutdown script and comply with the requirement of crashing the whole container in case one of the processes crashes.

[program:wazuh-manager]
command=service wazuh-manager start
process_name=wazuh-manager

[program:wazuh-api]
command=service wazuh-api start
process_name=wazuh-api

[eventlistener:processes]
command=stop-supervisor.sh
events=PROCESS_STATE_STOPPED, PROCESS_STATE_EXITED, PROCESS_STATE_FATAL 

The one drawback with this approach is the fact that the Wazuh API performs a restart of the manager on rule reloads, which could be interpreted as a crash event from supervisor POV, leading to a complete container shutdown.

Requirements compliance:

• Able to run multiple processes without a full init system
• No auto-restart, no self-healing (can be configured either way)
• Forward signals to subprocesses
• Container should die if any of the child processes crashes
• Able to consolidate subprocesses logs to stdout

Cons:

• Requires a Python interpreter on the image but the official CentOS 7 docker image already ship Python because of the yum dependency.

This is a process tree of Wazuh Manager, API and Filebeat running under Supervisor.

Summary

Supervisor works as expected but due to the nature of Wazuh services (no single process, API will restart the manager), we can't take advantage of all its benefits. Still, I'd put it on hold until we research some of the other options, but not discarding it yet.

[xr09]

Bash

This is a minimal but highly effective approach.

Using a bash entrypoint to launch every process as a background job and then only waiting for API and Filebeat to die (%2 %3 are their respective job numbers in that context), if both crash then the container will exit. Also the manager logs are printed with a tail since the ossec-control script doesn't allow "foreground" execution.

/var/ossec/bin/ossec-control start &
/bin/node /var/ossec/api/app.js &
/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml ....   &

tail -f /var/ossec/logs/ossec.log &

wait %2 %3

Requirements compliance:

• Able to run multiple processes without a full init system
• No auto-restart, no self-healing
• Forward signals to subprocesses
• Container should die if any of the child processes crashes (will die if all the waited processes die)
• Able to consolidate subprocesses logs to stdout

Cons

• It doesn't forward external signals to child processes (docker stop will hang for 10s until a kill signal is triggered)
• It will die only after all the waited children die

Tini + Bash

Another approach which improves on the previous is using tini as the PID 1 and leaving the rest untouched.

Requirements compliance:

• Able to run multiple processes without a full init system
• No auto-restart, no self-healing
• Forward signals to subprocesses
• Container should die if any of the child processes crashes (will die if all the waited processes die)
• Able to consolidate subprocesses logs to stdout

Cons

• It will die only after all the waited children die

Summary

The Bash + Tini method looks efficient and pragmatic, with great requirement compliance and a low memory footprint. There's a branch with this method available for testing.

[xr09]

S6 Overlay

S6 is a process supervision suite, s6-overlay is a rework to make it more usable inside containers.

Highlights

• PID 1 functionality
• Execute tasks like initialization (cont-init.d), finalization (cont-finish.d) as well as fixing ownership permissions (fix-attrs.d).
• Allows to execute a finish script when a process crashes, which allows for a complete container termination
• Logging helpers
• Works on any image (Debian, Ubuntu, CentOS..)

S6 defines a directory structure on /etc/services.d where each service is represented by a run and an optional finish script. The con-init.d allows for init script execution.

├── cont-init.d
│   ├── 0-wazuh-init
│   └── 1-config-filebeat
└── services.d
    ├── api
    │   ├── finish
    │   └── run
    ├── filebeat
    │   ├── finish
    │   └── run
    └── manager
        ├── finish
        └── run

On the finish script it's possible to shutdown the container completely, allowing for a a full shutdown in case one of the subprocesses crashes.

#!/usr/bin/env sh
echo >&2 "Filebeat exited. code=${1}"

# terminate other services to exit from the container
exec s6-svscanctl -t /var/run/s6/services

Requirements compliance:

• Able to run multiple processes without a full init system
• No auto-restart, no self-healing (can be configured either way)
• Forward signals to subprocesses
• Container should die if any of the child processes crashes
• Able to consolidate subprocesses logs to stdout

The process tree looks like this, with s6-svscan as PID 1 in charge of reaping zombie processes and forwarding signals.

Summary

Being prepared specifically for containers gives this approach a great advantage. Written in C, S6 is fast and nimble with great feature support.

There's a testing branch with this approach on the repo, working great so far.

[xr09]

Runit

Runit is very similar to S6, as both are based on daemontools.

Highlights

• PID 1 functionality
• Allows to execute a finish script when a process crashes, which allows for a complete container termination
• Logging helpers (svlogd)
• Works on any image (Debian, Ubuntu, CentOS..)

Runit also defines a directory structure on /etc/services where each service is represented by a run and an optional finish script.

service/
├── api
│   ├── finish
│   └── run
└── filebeat
    ├── finish
    └── run

It's also possible for a complete server shutdown in the event of a process failure.

Requirements compliance:

• Able to run multiple processes without a full init system
• No auto-restart, no self-healing (can be configured either way)
• Forward signals to subprocesses
• Container should die if any of the child processes crashes
• Able to consolidate subprocesses logs to stdout

Cons:

• I couldn't figure out the recommended way to emulate S6's init scripts behavior, one option is to hack it into one of the run scripts (since these will run only once after all) if the process dies the whole container will shutdown.
• There's no obvious way of sending a shutdown signal, although a kill -TERM 1 will work.

Runit's process tree:

Summary

Runit doesn't look bad but I don't like the fact that the code repository (and hence it's development process) is nowhere to be found, the author only publishes the sources on the website when there's a new release. Also, last release was 4 years ago.

There's a testing branch with runit here.

[xr09]

Overall comparison

Here's a quick comparison table of the previous methods:

Parameter	Supervisord	Bash	Bash + Tini	S6 Overlay	Runit
PID 1 behavior	✔️	❌	✔️	✔️	✔️
Execute multiple processes	✔️	✔️	✔️	✔️	✔️
Optional auto restart behavior	✔️	✔️ **	✔️	✔️	✔️
Signal forwarding	✔️	❌	✔️ **	✔️	✔️ *
Shutdown if a subprocess dies	✔️ *	✔️ **	✔️ **	✔️	✔️ **
Consolidate logs	✔️	✔️ **	✔️ **	✔️	✔️ *
Additional dependencies	Python interpreter		tini (static build)	S6 (static build)	RPM ⚠️
Init process memory usage	~16MB	~3MB	~4m	<0.5MB	~1MB
Latest release	19-10-2019		tini 21-04-2018	21-03-2019	10-08-2014

Table legend:

* Partially supported
** workarounds required
Downloaded from official repo release on Github
⚠️ External, non-official package repository

Conclusion

From a purely technical POV it's possible to provide a solution with virtually all these methods (applying workarounds in some cases) but S6 Overlay stands out as the most container friendly, feature rich and straightforward to use.