diff --git a/docs/sources/setup/install/helm/install-microservices/_index.md b/docs/sources/setup/install/helm/install-microservices/_index.md new file mode 100644 index 0000000000000..71f94673fe53c --- /dev/null +++ b/docs/sources/setup/install/helm/install-microservices/_index.md @@ -0,0 +1,346 @@ +--- +title: Install the microservice Helm chart +menuTitle: Install microservice Loki +description: Installing Loki in microservice (distributed) mode using the Helm chart. +weight: 300 +keywords: +--- + +# Install the microservice Helm chart + +This Helm Chart deploys Grafana Loki on Kubernetes. + +This chart configures Loki to run Loki in [microservice / distributed mode]({{< relref "../../../../get-started/deployment-modes#microservices-mode" >}}). The microservices deployment mode runs components of Loki as distinct processes. + +The default Helm chart deploys the following components: +- **Compactor component** (1 replica): Compacts and processes stored data. +- **Distributor component** (3 replicas, maxUnavailable: 2): Distributes incoming requests. Up to 2 replicas can be unavailable during updates. +- **IndexGateway component** (2 replicas, maxUnavailable: 1): Handles indexing. Up to 1 replica can be unavailable during updates. +- **Ingester component** (3 replicas): Handles ingestion of data. +- **Querier component** (3 replicas, maxUnavailable: 2): Processes queries. Up to 2 replicas can be unavailable during updates. +- **QueryFrontend component** (2 replicas, maxUnavailable: 1): Manages frontend queries. Up to 1 replica can be unavailable during updates. +- **QueryScheduler component** (2 replicas): Schedules queries. + +It is not recommended to run scalable mode with `filesystem` storage. For the purpose of this guide, we will use MinIO as the object storage to provide a complete example. + +**Prerequisites** + +- Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). +- A running Kubernetes cluster. +- (Optional) A Memcached deployment for better query performance. For information on configuring Memcached, refer to the [caching section](https://grafana.com/docs/loki//operations/caching/). + + +**To deploy Loki in microservice mode (with MinIO):** + + +1. Add [Grafana's chart repository](https://github.com/grafana/helm-charts) to Helm: + + ```bash + helm repo add grafana https://grafana.github.io/helm-charts + ``` + +2. Update the chart repository: + + ```bash + helm repo update + ``` + +3. Create the configuration file `values.yaml`. The example below illustrates how to deploy Loki in test mode using MinIO as storage: + + ```yaml + loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: s3 + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing + max_concurrent: 4 + + #gateway: + # ingress: + # enabled: true + # hosts: + # - host: FIXME + # paths: + # - path: / + # pathType: Prefix + + deploymentMode: Distributed + + ingester: + replicas: 3 + querier: + replicas: 3 + maxUnavailable: 2 + queryFrontend: + replicas: 2 + maxUnavailable: 1 + queryScheduler: + replicas: 2 + distributor: + replicas: 3 + maxUnavailable: 2 + compactor: + replicas: 1 + indexGateway: + replicas: 2 + maxUnavailable: 1 + + bloomCompactor: + replicas: 0 + bloomGateway: + replicas: 0 + + # Enable minio for storage + minio: + enabled: true + + # Zero out replica counts of other deployment modes + backend: + replicas: 0 + read: + replicas: 0 + write: + replicas: 0 + + singleBinary: + replicas: 0 + ``` + +4. Install or upgrade the Loki deployment. + - To install: + ```bash + helm install --values values.yaml loki grafana/loki + ``` + - To upgrade: + ```bash + helm upgrade --values values.yaml loki grafana/loki + ``` + + +1. Verify that Loki is running: + ```bash + kubectl get pods -n loki + ``` + The output should an output similar to the following: + + ```bash + loki-canary-8thrx 1/1 Running 0 167m + loki-canary-h965l 1/1 Running 0 167m + loki-canary-th8kb 1/1 Running 0 167m + loki-chunks-cache-0 0/2 Pending 0 167m + loki-compactor-0 1/1 Running 0 167m + loki-compactor-1 1/1 Running 0 167m + loki-distributor-7c9bb8f4dd-bcwc5 1/1 Running 0 167m + loki-distributor-7c9bb8f4dd-jh9h8 1/1 Running 0 167m + loki-distributor-7c9bb8f4dd-np5dw 1/1 Running 0 167m + loki-gateway-77bc447887-qgc56 1/1 Running 0 167m + loki-index-gateway-0 1/1 Running 0 167m + loki-index-gateway-1 1/1 Running 0 166m + loki-ingester-zone-a-0 1/1 Running 0 167m + loki-ingester-zone-b-0 1/1 Running 0 167m + loki-ingester-zone-c-0 1/1 Running 0 167m + loki-minio-0 1/1 Running 0 167m + loki-querier-bb8695c6d-bv9x2 1/1 Running 0 167m + loki-querier-bb8695c6d-bz2rw 1/1 Running 0 167m + loki-querier-bb8695c6d-z9qf8 1/1 Running 0 167m + loki-query-frontend-6659566b49-528j5 1/1 Running 0 167m + loki-query-frontend-6659566b49-84jtx 1/1 Running 0 167m + loki-query-frontend-6659566b49-9wfr7 1/1 Running 0 167m + loki-query-scheduler-f6dc4b949-fknfk 1/1 Running 0 167m + loki-query-scheduler-f6dc4b949-h4nwh 1/1 Running 0 167m + loki-query-scheduler-f6dc4b949-scfwp 1/1 Running 0 167m + loki-results-cache-0 2/2 Running 0 167m + ``` + +## Object Storage Configuration + +After testing Loki with MinIO, it is recommended to configure Loki with an object storage provider. The following examples shows how to configure Loki with different object storage providers: + +{{< code >}} + +```s3 +# Example configuration for Loki with S3 storage + + loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: s3 + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + max_concurrent: 4 + + storage: + type: s3 + bucketNames: + chunks: "chunks" + ruler: "ruler" + admin: "admin" + s3: + # s3 URL can be used to specify the endpoint, access key, secret key, and bucket name + s3: s3://access_key:secret_access_key@custom_endpoint/bucket_name + # AWS endpoint URL + endpoint: + # AWS region where the S3 bucket is located + region: + # AWS secret access key + secretAccessKey: + # AWS access key ID + accessKeyId: + # AWS signature version (e.g., v2 or v4) + signatureVersion: + # Forces the path style for S3 (true/false) + s3ForcePathStyle: false + # Allows insecure (HTTP) connections (true/false) + insecure: false + # HTTP configuration settings + http_config: {} + + deploymentMode: Distributed + + # Disable minio storage + minio: + enabled: false + + ingester: + replicas: 3 + querier: + replicas: 3 + maxUnavailable: 2 + queryFrontend: + replicas: 2 + maxUnavailable: 1 + queryScheduler: + replicas: 2 + distributor: + replicas: 3 + maxUnavailable: 2 + compactor: + replicas: 1 + indexGateway: + replicas: 2 + maxUnavailable: 1 + + bloomCompactor: + replicas: 0 + bloomGateway: + replicas: 0 + + backend: + replicas: 0 + read: + replicas: 0 + write: + replicas: 0 + + singleBinary: + replicas: 0 + +``` + +```azure +# Example configuration for Loki with Azure Blob Storage + +loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: azure + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + max_concurrent: 4 + + storage: + type: azure + azure: + # Name of the Azure Blob Storage account + accountName: + # Key associated with the Azure Blob Storage account + accountKey: + # Comprehensive connection string for Azure Blob Storage account (Can be used to replace endpoint, accountName, and accountKey) + connectionString: + # Flag indicating whether to use Azure Managed Identity for authentication + useManagedIdentity: false + # Flag indicating whether to use a federated token for authentication + useFederatedToken: false + # Client ID of the user-assigned managed identity (if applicable) + userAssignedId: + # Timeout duration for requests made to the Azure Blob Storage account (in seconds) + requestTimeout: + # Domain suffix of the Azure Blob Storage service endpoint (e.g., core.windows.net) + endpointSuffix: + bucketNames: + chunks: "chunks" + ruler: "ruler" + admin: "admin" +deploymentMode: Distributed + +ingester: + replicas: 3 +querier: + replicas: 3 + maxUnavailable: 2 +queryFrontend: + replicas: 2 + maxUnavailable: 1 +queryScheduler: + replicas: 2 +distributor: + replicas: 3 + maxUnavailable: 2 +compactor: + replicas: 1 +indexGateway: + replicas: 2 + maxUnavailable: 1 + +bloomCompactor: + replicas: 0 +bloomGateway: + replicas: 0 + +backend: + replicas: 0 +read: + replicas: 0 +write: + replicas: 0 + +singleBinary: + replicas: 0 + +``` +{{< /code >}} + +To configure other storage providers, refer to the [Helm Chart Reference]({{< relref "../reference" >}}). + +## Next Steps +* Configure an agent to [send log data to Loki](/docs/loki//send-data/). +* Monitor the Loki deployment using the [Meta Monitoring Healm chart](/docs/loki//setup/install/helm/monitor-and-alert/) diff --git a/docs/sources/setup/install/helm/install-scalable/_index.md b/docs/sources/setup/install/helm/install-scalable/_index.md index 69e54ec1c3196..e27f544b28f0c 100644 --- a/docs/sources/setup/install/helm/install-scalable/_index.md +++ b/docs/sources/setup/install/helm/install-scalable/_index.md @@ -10,12 +10,10 @@ keywords: --- # Install the simple scalable Helm chart - - This Helm Chart deploys Grafana Loki on Kubernetes. -This chart configures Loki to run `read`, `write`, and `backend` targets in a [scalable mode]({{< relref "../../../../get-started/deployment-modes#simple-scalable" >}}), highly available architecture designed to work with AWS S3 object storage. The chart also supports self-monitoring or meta-monitoring by deploying Grafana Agent to monitor Loki itself, by scraping its metrics and logs. +This chart configures Loki to run `read`, `write`, and `backend` targets in a [scalable mode]({{< relref "../../../../get-started/deployment-modes#simple-scalable" >}}). Loki’s simple scalable deployment mode separates execution paths into read, write, and backend targets. The default Helm chart deploys the following components: - Read component (3 replicas) @@ -24,11 +22,8 @@ The default Helm chart deploys the following components: - Loki Canary (1 DaemonSet) - Gateway (1 NGINX replica) - Minio (optional, if `minio.enabled=true`) -- Grafana Agent Operator + Grafana Agent (1 DaemonSet) - configured to monitor the Loki application. - - -It is not recommended to run scalable mode with `filesystem` storage. +It is not recommended to run scalable mode with `filesystem` storage. For the purpose of this guide, we will use MinIO as the object storage to provide a complete example. **Prerequisites** @@ -46,42 +41,80 @@ It is not recommended to run scalable mode with `filesystem` storage. helm repo add grafana https://grafana.github.io/helm-charts ``` -1. Update the chart repository: +2. Update the chart repository: ```bash helm repo update ``` -1. Configure the object storage: - - - Create the configuration file `values.yaml`. The example below illustrates a s3 configuration: +3. Create the configuration file `values.yaml`. The example below illustrates how to deploy Loki in test mode using MinIO as storage: ```yaml - loki: - storage: - bucketNames: - chunks: chunks - ruler: ruler - admin: admin - type: s3 - s3: - endpoint: - region: - secretAccessKey: - accessKeyId: - s3ForcePathStyle: false - insecure: false - ``` + loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: s3 + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing + max_concurrent: 4 - To configure other storage providers, refer to the [Helm Chart Reference]({{< relref "../reference" >}}). + #gateway: + # ingress: + # enabled: true + # hosts: + # - host: FIXME + # paths: + # - path: / + # pathType: Prefix - - If you're just trying things, you can use the following configuration, that sets MinIO as storage: - ```yaml - minio: - enabled: true + deploymentMode: SimpleScalable + + backend: + replicas: 3 + read: + replicas: 3 + write: + replicas: 3 + + # Enable minio for storage + minio: + enabled: true + + # Zero out replica counts of other deployment modes + singleBinary: + replicas: 0 + + ingester: + replicas: 0 + querier: + replicas: 0 + queryFrontend: + replicas: 0 + queryScheduler: + replicas: 0 + distributor: + replicas: 0 + compactor: + replicas: 0 + indexGateway: + replicas: 0 + bloomCompactor: + replicas: 0 + bloomGateway: + replicas: 0 ``` -1. Install or upgrade the Loki deployment. +4. Install or upgrade the Loki deployment. - To install: ```bash helm install --values values.yaml loki grafana/loki @@ -91,5 +124,175 @@ It is not recommended to run scalable mode with `filesystem` storage. helm upgrade --values values.yaml loki grafana/loki ``` +## Object Storage Configuration + +After testing Loki with MinIO, it is recommended to configure Loki with an object storage provider. The following examples shows how to configure Loki with different object storage providers: + +{{< code >}} + +```s3 + loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: s3 + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + max_concurrent: 4 + + storage: + type: s3 + bucketNames: + chunks: "chunks" + ruler: "ruler" + admin: "admin" + s3: + # s3 URL can be used to specify the endpoint, access key, secret key, and bucket name + s3: s3://access_key:secret_access_key@custom_endpoint/bucket_name + # AWS endpoint URL + endpoint: + # AWS region where the S3 bucket is located + region: + # AWS secret access key + secretAccessKey: + # AWS access key ID + accessKeyId: + # AWS signature version (e.g., v2 or v4) + signatureVersion: + # Forces the path style for S3 (true/false) + s3ForcePathStyle: false + # Allows insecure (HTTP) connections (true/false) + insecure: false + # HTTP configuration settings + http_config: {} + + deploymentMode: SimpleScalable + + backend: + replicas: 3 + read: + replicas: 3 + write: + replicas: 3 + + # Disable minio storage + minio: + enabled: false + + # Zero out replica counts of other deployment modes + singleBinary: + replicas: 0 + + ingester: + replicas: 0 + querier: + replicas: 0 + queryFrontend: + replicas: 0 + queryScheduler: + replicas: 0 + distributor: + replicas: 0 + compactor: + replicas: 0 + indexGateway: + replicas: 0 + bloomCompactor: + replicas: 0 + bloomGateway: + replicas: 0 +``` +```azure + loki: + schemaConfig: + configs: + - from: 2024-04-01 + store: tsdb + object_store: azure + schema: v13 + index: + prefix: loki_index_ + period: 24h + ingester: + chunk_encoding: snappy + tracing: + enabled: true + querier: + max_concurrent: 4 + + storage: + type: azure + azure: + # Name of the Azure Blob Storage account + accountName: + # Key associated with the Azure Blob Storage account + accountKey: + # Comprehensive connection string for Azure Blob Storage account (Can be used to replace endpoint, accountName, and accountKey) + connectionString: + # Flag indicating whether to use Azure Managed Identity for authentication + useManagedIdentity: false + # Flag indicating whether to use a federated token for authentication + useFederatedToken: false + # Client ID of the user-assigned managed identity (if applicable) + userAssignedId: + # Timeout duration for requests made to the Azure Blob Storage account (in seconds) + requestTimeout: + # Domain suffix of the Azure Blob Storage service endpoint (e.g., core.windows.net) + endpointSuffix: + bucketNames: + chunks: "chunks" + ruler: "ruler" + admin: "admin" + + deploymentMode: SimpleScalable + + backend: + replicas: 3 + read: + replicas: 3 + write: + replicas: 3 + + # Disable minio storage + minio: + enabled: false + + # Zero out replica counts of other deployment modes + singleBinary: + replicas: 0 + + ingester: + replicas: 0 + querier: + replicas: 0 + queryFrontend: + replicas: 0 + queryScheduler: + replicas: 0 + distributor: + replicas: 0 + compactor: + replicas: 0 + indexGateway: + replicas: 0 + bloomCompactor: + replicas: 0 + bloomGateway: + replicas: 0 +``` + +{{< /code >}} + +To configure other storage providers, refer to the [Helm Chart Reference]({{< relref "../reference" >}}). + ## Next Steps -Configure an agent to [send log data to Loki](/docs/loki//send-data/). +* Configure an agent to [send log data to Loki](/docs/loki//send-data/). +* Monitor the Loki deployment using the [Meta Monitoring Healm chart](/docs/loki//setup/install/helm/monitor-and-alert/) \ No newline at end of file