Support SIGUSR1 log level incrementing

[echlebek]

What is this change?

This commit adds support to sensu-backend and sensu-agent for
incrementing the log level with SIGUSR1. When the process receives
this signal, the log level will be incremented in terms of
verbosity. When the level is at TRACE, the most verbose log level,
and the log level is incremented, it will wrap around to error
level.

Why is this change necessary?

Closes #1742

Does your change need a Changelog entry?

Yes

Were there any complications while making this change?

It's unclear at this time how to support this for Windows.

Have you reviewed and updated the documentation for this change? Is new documentation required?

@hillaryfraley, we will need to document that users can now increase the log level at runtime, with the following commands for backends and agents:

kill -s SIGUSR1 $(pidof sensu-backend)

kill -s SIGUSR1 $(pidof sensu-agent)

How did you verify this change?

I ran the above commands for the backend and agent.

Is this change a patch?

No.

[palourde]

@echlebek looks like it broke the Windows builds 😱

[echlebek]

@palourde I moved the signal handling stuff into some files that won't get built on Windows, which seems to have un-broken the build.

[palourde]

I figured this is expected, but I'm wondering if there's a way to by-pass the following behavior:

When the log level is incremented from trace --> error, the following entry is logged, which is expected:

{"level":"warning","msg":"set log level to error","time":"2020-04-14T18:38:58-04:00"}

However, when we go from error --> warning, the same entry never gets logged. I'm wondering if we could add a condition so when the current log level is error, the entry gets logged after the log level is changed. e.g.

logrus.Warnf("set log level to %s", newLevel)
logrus.SetLevel(newLevel)
// Log back the entry if the level passed from error to warning
if logrus.GetLevel() == logrus.WarningLevel {
	logrus.Warnf("set log level to %s", newLevel)
}

WDYT?

[echlebek]

It's a bit of an oddity, but I like that it makes the message appear in all cases. I'll add it!

[palourde]

🚀