chore: switch promtails base image from debian to ubuntu

[cstyan]

This PR changes promtails base image from debian:12.8-slim to ubuntu:noble-20241015, noble being ubunutu's most recent lts version. The current debian base image has a lot of security CVEs that won't be updated, while ubuntu updates packages much more regularly.

Just as a quick example, the result of a trivy scan on the promtail image with debian base gives Total: 79 (UNKNOWN: 0, LOW: 59, MEDIUM: 14, HIGH: 5, CRITICAL: 1) while the trivy scan for the promtail image with ubunutu base results in Total: 15 (UNKNOWN: 0, LOW: 7, MEDIUM: 8, HIGH: 0, CRITICAL: 0)

EDIT: Moved to draft until I've confirmed the new image can still grab systemd/journal logs

EDIT: Couldn't trace through the makefile/Dockerfile what needs to be set for the image to build with promtail and cgo locally, but given we publish images that can get logs from journal then the automation should be set up properly. If I enforce CGO_ENABLED=1 for the make promtail-image target locally, I can use that image to get logs via the journal scrape job.

I ran promtail in a docker container with a scrape config like this:

scrape_configs:
- job_name: journal
  journal:
    path: /var/log/journal
    max_age: 12h
    labels:
      job: systemd-journal

I also mounted my machines /var/log/journal directory to the same path on the container, and set --network="host" so that:

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

would be able to send to loki running locally on my machine.

Then once both promtail in docker and loki locally are running, I added loki as a datasource to my local grafana as well.

In my journalctl logs I see:

and we can see the same in the logs that make it to loki

from promtail

[cstyan]

as for why the image from ecr instead of the official ubuntu docker hub image, see grafana/agent#6612

[chaudum]

👍

[loki-gh-app]

Hello @cstyan!
Backport pull requests need to be either:

• Pull requests which address bugs,
• Urgent fixes which need product approval, in order to get merged,
• Docs changes.

Please, if the current pull request addresses a bug fix, label it with the type/bug label.
If it already has the product approval, please add the product-approved label. For docs changes, please add the type/docs label.
If the pull request modifies CI behaviour, please add the type/ci label.
If none of the above applies, please consider removing the backport label and target the next major/minor release.
Thanks!

[loki-gh-app]

Hello @cstyan!
Backport pull requests need to be either:

• Pull requests which address bugs,
• Urgent fixes which need product approval, in order to get merged,
• Docs changes.

Please, if the current pull request addresses a bug fix, label it with the type/bug label.
If it already has the product approval, please add the product-approved label. For docs changes, please add the type/docs label.
If the pull request modifies CI behaviour, please add the type/ci label.
If none of the above applies, please consider removing the backport label and target the next major/minor release.
Thanks!

[loki-gh-app]

Hello @cstyan!
Backport pull requests need to be either:

• Pull requests which address bugs,
• Urgent fixes which need product approval, in order to get merged,
• Docs changes.

Please, if the current pull request addresses a bug fix, label it with the type/bug label.
If it already has the product approval, please add the product-approved label. For docs changes, please add the type/docs label.
If the pull request modifies CI behaviour, please add the type/ci label.
If none of the above applies, please consider removing the backport label and target the next major/minor release.
Thanks!