Skip to content
Permalink

Comparing changes

This is a direct comparison between two commits made in this repository or its related repositories. View the default comparison for this range or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: grafana/loki
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: c9fc02d853c6b7fc6a07cf156975013d7aefc847
Choose a base ref
..
head repository: grafana/loki
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 97c1dca1ea5921387835c588643bee031b2a97d3
Choose a head ref
Showing with 3,712 additions and 458 deletions.
  1. +3 −0 .gitignore
  2. +64 −0 docs/sources/send-data/alloy/_index.md
  3. +405 −0 docs/sources/send-data/alloy/examples/alloy-kafka-logs.md
  4. +292 −0 docs/sources/send-data/alloy/examples/alloy-otel-logs.md
  5. +1 −1 docs/sources/setup/install/helm/reference.md
  6. +1 −1 docs/sources/setup/upgrade/_index.md
  7. +25 −0 docs/sources/shared/configuration.md
  8. +5 −5 pkg/ingester-rf1/flush.go
  9. +1 −1 pkg/ingester-rf1/ingester.go
  10. +1 −3 pkg/ingester-rf1/metastore/client/client.go
  11. +79 −0 pkg/ingester-rf1/metastore/metastore_state_list_blocks_for_query.go
  12. +128 −0 pkg/ingester-rf1/metastore/metastore_state_list_blocks_for_query_test.go
  13. +607 −31 pkg/ingester-rf1/metastore/metastorepb/metastore.pb.go
  14. +11 −0 pkg/ingester-rf1/metastore/metastorepb/metastore.proto
  15. +25 −31 pkg/ingester-rf1/metrics.go
  16. +8 −0 pkg/ingester-rf1/objstore/storage.go
  17. +0 −1 pkg/logql/downstream.go
  18. +59 −1 pkg/logql/downstream_test.go
  19. +44 −24 pkg/logql/engine.go
  20. +5 −2 pkg/logql/first_last_over_time.go
  21. +0 −1 pkg/logql/test_utils.go
  22. +4 −1 pkg/loki/loki.go
  23. +22 −7 pkg/loki/modules.go
  24. +149 −0 pkg/querier-rf1/handler.go
  25. +370 −0 pkg/querier-rf1/http.go
  26. +930 −0 pkg/querier-rf1/querier.go
  27. +1 −0 pkg/querier/querier.go
  28. +2 −0 pkg/querier/querier_mock_test.go
  29. +2 −14 pkg/querier/queryrange/roundtrip.go
  30. +20 −0 pkg/storage/chunk/client/alibaba/oss_object_client.go
  31. +34 −0 pkg/storage/chunk/client/aws/s3_storage_client.go
  32. +32 −3 pkg/storage/chunk/client/azure/blob_storage_client.go
  33. +13 −0 pkg/storage/chunk/client/baidubce/bos_storage_client.go
  34. +7 −0 pkg/storage/chunk/client/congestion/controller.go
  35. +3 −0 pkg/storage/chunk/client/congestion/controller_test.go
  36. +18 −0 pkg/storage/chunk/client/gcp/gcs_object_client.go
  37. +31 −0 pkg/storage/chunk/client/ibmcloud/cos_object_client.go
  38. +24 −0 pkg/storage/chunk/client/local/fs_object_client.go
  39. +1 −0 pkg/storage/chunk/client/object_client.go
  40. +14 −0 pkg/storage/chunk/client/openstack/swift_object_client.go
  41. +4 −0 pkg/storage/chunk/client/prefixed_object_client.go
  42. +20 −0 pkg/storage/chunk/client/testutils/inmemory_storage_client.go
  43. +1 −1 pkg/storage/stores/shipper/bloomshipper/client.go
  44. +30 −49 pkg/storage/wal/manager.go
  45. +99 −35 pkg/storage/wal/manager_test.go
  46. +22 −27 pkg/storage/wal/metrics.go
  47. +73 −43 pkg/storage/wal/segment.go
  48. +14 −172 pkg/storage/wal/segment_test.go
  49. +5 −1 production/helm/loki/CHANGELOG.md
  50. +1 −1 production/helm/loki/Chart.yaml
  51. +1 −1 production/helm/loki/README.md
  52. BIN production/helm/loki/src/helm-test/helm-test
  53. +1 −1 production/helm/loki/values.yaml
3 changes: 3 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -56,3 +56,6 @@ nix/result

# snyk
.dccache

# helm test
production/helm/loki/src/helm-test/helm-test
64 changes: 64 additions & 0 deletions docs/sources/send-data/alloy/_index.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
---
title: Ingesting logs to Loki using Alloy
menuTitle: Grafana Alloy
description: Configuring Grafana Alloy to send logs to Loki.
weight: 250
---


# Ingesting logs to Loki using Alloy

Grafana Alloy is a versatile observability collector that can ingest logs in various formats and send them to Loki. We recommend Alloy as the primary method for sending logs to Loki, as it provides a more robust and feature-rich solution for building a highly scalable and reliable observability pipeline.

{{< figure src="/media/docs/alloy/flow-diagram-small-alloy.png" alt="Alloy flow diagram" >}}

## Installing Alloy

To get started with Grafana Alloy and send logs to Loki, you need to install and configure Alloy. You can follow the [Alloy documentation](https://grafana.com/docs/alloy/latest/get-started/install/) to install Alloy on your preferred platform.

## Components of Alloy for logs

Alloy pipelines are built using components that perform specific functions. For logs these can be broken down into three categories:

- **Collector:** These components collect/receive logs from various sources. This can be scraping logs from a file, receiving logs over HTTP, gRPC or ingesting logs from a message queue.
- **Transformer:** These components can be used to manipulate logs before they are sent to a writer. This can be used to add additional metadata, filter logs, or batch logs before sending them to a writer.
- **Writer:** These components send logs to the desired destination. Our documentation will focus on sending logs to Loki, but Alloy supports sending logs to various destinations.

### Log components in Alloy

Here is a non-exhaustive list of components that can be used to build a log pipeline in Alloy. For a complete list of components, refer to the [components list](https://grafana.com/docs/alloy/latest/reference/components/).

| Type | Component |
|------------|-----------------------------------------------------------------------------------------------------|
| Collector | [loki.source.api](https://grafana.com/docs/alloy/latest/reference/components/loki.source.api/) |
| Collector | [loki.source.awsfirehose](https://grafana.com/docs/alloy/latest/reference/components/loki.source.awsfirehose/) |
| Collector | [loki.source.azure_event_hubs](https://grafana.com/docs/alloy/latest/reference/components/loki.source.azure_event_hubs/) |
| Collector | [loki.source.cloudflare](https://grafana.com/docs/alloy/latest/reference/components/loki.source.cloudflare/) |
| Collector | [loki.source.docker](https://grafana.com/docs/alloy/latest/reference/components/loki.source.docker/) |
| Collector | [loki.source.file](https://grafana.com/docs/alloy/latest/reference/components/loki.source.file/) |
| Collector | [loki.source.gcplog](https://grafana.com/docs/alloy/latest/reference/components/loki.source.gcplog/) |
| Collector | [loki.source.gelf](https://grafana.com/docs/alloy/latest/reference/components/loki.source.gelf/) |
| Collector | [loki.source.heroku](https://grafana.com/docs/alloy/latest/reference/components/loki.source.heroku/) |
| Collector | [loki.source.journal](https://grafana.com/docs/alloy/latest/reference/components/loki.source.journal/) |
| Collector | [loki.source.kafka](https://grafana.com/docs/alloy/latest/reference/components/loki.source.kafka/) |
| Collector | [loki.source.kubernetes](https://grafana.com/docs/alloy/latest/reference/components/loki.source.kubernetes/) |
| Collector | [loki.source.kubernetes_events](https://grafana.com/docs/alloy/latest/reference/components/loki.source.kubernetes_events/) |
| Collector | [loki.source.podlogs](https://grafana.com/docs/alloy/latest/reference/components/loki.source.podlogs/) |
| Collector | [loki.source.syslog](https://grafana.com/docs/alloy/latest/reference/components/loki.source.syslog/) |
| Collector | [loki.source.windowsevent](https://grafana.com/docs/alloy/latest/reference/components/loki.source.windowsevent/) |
| Collector | [otelcol.receiver.loki](https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.loki/) |
| Transformer| [loki.relabel](https://grafana.com/docs/alloy/latest/reference/components/loki.relabel/) |
| Transformer| [loki.process](https://grafana.com/docs/alloy/latest/reference/components/loki.process/) |
| Writer | [loki.write](https://grafana.com/docs/alloy/latest/reference/components/loki.write/) |
| Writer | [otelcol.exporter.loki](https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.loki/) |
| Writer | [otelcol.exporter.logging](https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.logging/) |


## Interactive Tutorials

To learn more about how to configure Alloy to send logs to Loki within different scenarios, follow these interactive tutorials:

- [Sending OpenTelemetry logs to Loki using Alloy]({{< relref "./examples/alloy-otel-logs" >}})
- [Sending logs over Kafka to Loki using Alloy]({{< relref "./examples/alloy-kafka-logs" >}})


Loading