Centralised logging collection

[Oloremo]

Hi,

Currently, Nomad doesn't provide any way to do a centralized login by itself and relies on an external log collector or allocation drivers like docker logging.

I understand that reasoning but all current solutions are flawed in some way or another.

• Docker engine is kinda deprecated and will be less and less present in the following years.
• Side-car log collectors add complexity to job definitions and consume a lot of resources on the scale
• Not all application logging drivers could send logs via network and write logs to the ephemeral disk of the container comes with complications.
• Some logs are generated before logging system init - Java is a good example.
• System Job log collection application has to be run with raw exec to be able to collect stdout/stderr logs from Nomad data alloc dirs.
• OS system log collection app has to run from the root user to be able to collect Nomad data alloc stderr/stdout files.

Seems like a fluentbit is a CNCF preferred solution for logs collecting it would make sense if Nomad could support streaming its allocation logs to a fluentbit agent.

UPD.
We tried Vector and we decided to drop fluentbit in favor of it. :-D

[mr-karan]

#10219 could be merged.

[tgross]

Hi @Oloremo! Thanks for opening this issue!

As @mr-karan has noted, there's been a few different requests for this kind of thing. (See here). It's definitely something we want to do and we want to make sure we're tackling the problem holistically. I'll mark this for roadmapping.

[alexdulin]

Elastic added support for Nomad allocation discovery to their log collecting agent filebeat recently if that is of any help.

[Oloremo]

@alexdulin Oh, I didn't know that, interesting!

UPD: Hm, well not sure how it's better than just glob. Seems like it's useful if you need to collect not all but specific jobs logs.

[sofixa]

Personally i'm more in favour of prioritising doing everything to support logging agents ( e.g. adding support for discovery in alternatives to filebeat like Promtail, Vector, fluent*, etc. and making metadata from Nomad available) than adding functionalities inside Nomad to stream logs, mostly because:

• pull is more reliable than push ( if the logging agent restarts, or crashes or w/e, it can start from where it left over; Nomad would have to do retries and backoffs, and a whole lot of logic to achieve the same level of resilience)
• pull is more flexible than push ( any logging agent is capable of reading files, if Nomad sends it will have to manage all the different formats ( or at least a bunch of them)