Promtail not following symlinks as expected

[fchiorascu]

Is your feature request related to a problem? Please describe.
Possibility to collect (tail) logs from stdout/stderr, port?

The current static_config uses path (log files) https://grafana.com/docs/loki/latest/clients/promtail/configuration/#static_configs that for symlinks is not working as expected.

Using: PLG Stack
Grafana: 7.4.0
Loki: 2.1.0
Promtail: 2.1.0

Describe the solution you'd like

• Promtail being able to tail logs from: path file logs - symlinks, port, stdout, stderr of a pod/container in RHEL OpenSHift Container > 3.9 from a namespace.

Describe alternatives you've considered

• At least being able to tail path file logs - symlinks.

Additional context

• https://docs.fluentbit.io/manual/pipeline/filters/standard-output
• https://docs.fluentbit.io/manual/pipeline/inputs/tcp

[bramevo]

+1 For this request. I need to be able parse compressed log files (gzip) and this would be possible using the above I assume.
Now I need a dedicated container just for decompressing the logfiles before using the promtail container to parse them.

[proffalken]

Isn't this what Fluentd/fluent-bit are for?

Promtail seems like a good starting place to slurp logs, but if you want to do processing on the data as it comes in then there are plenty of existing tools for that and we should be careful not to re-invent the wheel.

Admittedly neither of them are able to parse compressed files, but it feels we should add the features to something like that rather than adding it to promtail?

[fchiorascu]

@proffalken thank you for your point of view.

I understand that is not a need of reinventing the wheel or covering the same aspects with multiple options/similar tools, this is clear and no doubt there. However I don't need promtail for log processing or to use in complex ecosystems, it was a good fit for us due to the fact that is lightweight and we could integrate in a small RHEL OpenShift Container Platform (namespace) to collect logs from 1 application (backend/frontend) but he is not behaving well with symlinks and doesn't have the maturity or alternatives to file log path such as stdout, stderr, port. Why to use fluentbit/fluentd instead of using entire PLG Stack, reducing the complexity and being sure that the pieces integrates seamlessly.

[proffalken]

@fchiorascu that's fair enough, objection removed ;)

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[proffalken]

Commenting to keep this open, I think it's a really useful addition to Loki/Promtail.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[proffalken]

No thanks @stalebot - let's keep this one open...

[majuansari]

Please keep this one open. Much needed feature

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[MurzNN]

Very useful feature, reopen pls!

[fchiorascu]

Reopen please.

[slim-bean]

Trying to understand this issue and it feels like it covers multiple topics.

In regards to the symlinks not working, this sounds like a bug, we use symlinks all the time without issue, are you able to provide more context here?

In regards to logs from stdout and stderr, promtail supports the --stdin flag now which lets you pipe logs into promtail.

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[MurzNN]

/keepalive

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[MurzNN]

Bump

[DylanGuedes]

Apparently, grafana/tail#12 solves this 👀