Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker Compose(logs): Log Ingestion via Labels #47

Merged
merged 3 commits into from
Mar 12, 2024

Conversation

qclaogui
Copy link
Owner

@qclaogui qclaogui commented Mar 11, 2024

Added support for ingesting logs and controlling the behavior of their ingestion through labels.

The following docker compose service labels are supported:

Label Description
logs.agent.grafana.com/scrape Allow a service to declare it's logs should be dropped.
logs.agent.grafana.com/tenant Allow a service to override the tenant for its logs.
logs.agent.grafana.com/log-format If specified additional processing is performed to extract details based on the specified format. This value can be a comma-delimited list, in the instances a pod may have multiple containers. The following formats are currently supported:
  • common-log
  • donet
  • istio
  • json
  • klog
  • log4j-json
  • logfmt
  • otel
  • postgres
  • python
  • spring-boot
  • syslog
  • zerolog
logs.agent.grafana.com/scrub-level Boolean whether or not the level should be dropped from the log message (as it is a label).
logs.agent.grafana.com/scrub-timestamp Boolean whether or not the timestamp should be dropped from the log message (as it is metadata).
logs.agent.grafana.com/scrub-nulls Boolean whether or not keys with null values should be dropped from json, reducing the size of the log message.
logs.agent.grafana.com/scrub-empties Boolean whether or not keys with empty values ("", [], {}) should be dropped from json, reducing the size of the log message.
logs.agent.grafana.com/drop-info Boolean whether or not info messages should be dropped (default is false), but a pod can override this temporarily or permanently.
logs.agent.grafana.com/drop-debug Boolean whether or not debug messages should be dropped (default is true), but a pod can override this temporarily or permanently.
logs.agent.grafana.com/drop-trace Boolean whether or not trace messages should be dropped (default is true), but a pod can override this temporarily or permanently.
logs.agent.grafana.com/mask-ssn Boolean whether or not to mask SSNs in the log line, if true the data will be masked as *SSN*salt*
logs.agent.grafana.com/mask-credit-card Boolean whether or not to mask credit cards in the log line, if true the data will be masked as *credit-card*salt*
logs.agent.grafana.com/mask-email Boolean whether or not to mask emails in the log line, if true the data will be masked as*email*salt*
logs.agent.grafana.com/mask-ipv4 Boolean whether or not to mask IPv4 addresses in the log line, if true the data will be masked as*ipv4*salt*
logs.agent.grafana.com/mask-ipv6 Boolean whether or not to mask IPv6 addresses in the log line, if true the data will be masked as *ipv6*salt*
logs.agent.grafana.com/mask-phone Boolean whether or not to mask phone numbers in the log line, if true the data will be masked as *phone*salt*

Signed-off-by: Weifeng Wang <[email protected]>

update loki log_format

Signed-off-by: Weifeng Wang <[email protected]>

update loki log_format

Signed-off-by: Weifeng Wang <[email protected]>
@qclaogui qclaogui marked this pull request as ready for review March 12, 2024 03:45
@qclaogui qclaogui merged commit 578112e into main Mar 12, 2024
1 check passed
@qclaogui qclaogui deleted the log-ingestion-via-labels branch March 12, 2024 04:08
This was referenced Mar 26, 2024
This was referenced Mar 29, 2024
This was referenced Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant