Start workers explicitly

[subpop]

This PR refactors the way processes are started, watched, killed, and restarted. The routines in exec.go were a mixture of higher-level worker process management (including restart delays, PID file management, etc.) and lower-level process start and kill routines. This PR moves the higher-level logic out of exec.go and into a file called worker.go. The routines in exec.go are now purely lower-level routines that start and stop processes; they have no awareness of worker configuration or process restart management (startProcess, stopProcess, waitProcess). The functions in worker.go operate on a workerConfig data structure to start, stop and manage workers.

A workerConfig structure is created by parsing a configuration file that must be defined in /etc/yggdrasil/workers. The existence of this file determines whether yggd is aware of, and will manage, a worker (previously this was handled by the existence of an executable program in /usr/libexec/yggdrasil). While a binary may still exist there, it will not be automatically started unless a valid configuration file points to that file as its executable. The name of the configuration file, minus the .toml suffix, is assumed to be the worker's handle (this ensures that no two workers will attempt to claim the same handle value).

For example, the echo worker now includes a configuration file:

exec="/usr/local/libexec/ygg-echo-worker"
protocol="grpc"
env=[]

The exec field must be a valid executable program. The protocol field is used to define which RPC protocol the worker is using. Currently the only supported value is "grpc". Values in the env array must be strings in the form of "VAR=VALUE" and are inserted into a worker's environment before starting it. Any value in the array that begins with either "PATH" or "YGG_" is ignored (in other words, a worker may not override its own PATH or any variable beginning with YGG_ environment variables through its configuration file).

A worker's environment is automatically configured with a reasonable base PATH value (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin) and inherits values from its parent for: http_proxy, HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY.

A worker can be excluded from activation either permanently (by adding a value to the configuration file) or temporarily (by using the --exclude-worker command line flag). If a worker's name is found in this list during worker startup, it is skipped. It will not be started by yggd until the name is removed from the configuration value and yggd is restarted.

Little functionality should be changed by this PR. It mainly lays some groundwork to enable more functionality down the road. Nonetheless, this is a breaking change for workers, so it will not ship without advanced notice.

Fixes: ESSNTL-302, ESSNTL-298, ESSNTL-279

[subpop]

FYI @strider

[strider]

Ok, I've tested it again, but correctly this time. yggd detects new worker(s), starts them and stops them properly.

[strider]

Besides my comment about the go.mod file, it looks good to me and works pretty well.

[subpop]

This one is starting to get out of control. I'm going to look into rebasing these commits down to a more atomic change.

[subpop]

All wrapped up and ready to go. I'm pretty sure this is ready to merge now that the commits are more manageable.

[strider]

Last force-push works as expected. Yes, I agree with your statement about merging
this big one as soon as possible. Nice work on that one, Link!

[strider]

LGTM! 👍