log receiver service

[Ulexus]

Sidero should offer a logging service which accepts logs from managed machines and stores them to either (configurable) RAM-based ring buffer or disk storage.

These logs should be grouped by Cluster and by Machine.

The design of Loki seems like it fits well with the needs here, and Loki offers some useful storage backends, while being substantially lighter weight than other log aggregators like ElasticSearch. So perhaps the options should be ring buffer, disk file, or Loki.

In any case, we need both client and server sides of this feature: Sidero should act as a log server to broker and direct logs. It should configure Environments and Talos installations to set netconsole= kernel commandline flags to its log service. Thus, no client-side service is required (though we may wish to add in other logs later), just the kernel config.

[smira]

Also additional questions:

• how do we query these logs? do we need separate CLI tool?
• can we somehow make these logs go into Kubernetes? can we e.g. schedule a pod which dumps server logs to stderr, so that kubectl logs can be used to view the logs (and it solved the storage issue as well)

[smira]

https://www.kernel.org/doc/Documentation/networking/netconsole.txt

[smira]

• receiver for log messages
• receiver matches source IP address with SideoLink IPs and figures out Server, enhances JSON log message with:
  • Server UUID
  • Server hostname (if available)
  • Machine name (if available)
  • Cluster name (if available)
• receiver outputs logs to its stderr, so that logs can be accessed with kubectl logs on that container and forwarded to log system for archival