diff --git a/docs/clients/README.md b/docs/clients/README.md index bb54fd657b4e4..900586c137f2c 100644 --- a/docs/clients/README.md +++ b/docs/clients/README.md @@ -1,13 +1,13 @@ -# Loki Clients +# Loki clients Loki supports the following official clients for sending logs: -1. [Promtail](./promtail/README.md) -2. [Docker Driver](./docker-driver/README.md) -3. [Fluentd](./fluentd/README.md) -4. [Fluent Bit](../../cmd/fluent-bit/README.md) +- [Promtail](./promtail/README.md) +- [Docker Driver](./docker-driver/README.md) +- [Fluentd](./fluentd/README.md) +- [Fluent Bit](../../cmd/fluent-bit/README.md) -## Picking a Client +## Picking a client While all clients can be used simultaneously to cover multiple use cases, which client is initially picked to send logs depends on your use case. @@ -41,9 +41,9 @@ and you already have configured `Parser` and `Filter` plugins. Fluentd also works well for extracting metrics from logs when using its Prometheus plugin. -# Unofficial Clients +# Unofficial clients -Please note that the Loki API is not stable yet and breaking changes may occur +Please note that the Loki API is not stable yet, so breaking changes might occur when using or writing a third-party client. - [promtail-client](https://github.com/afiskon/promtail-client) (Go) diff --git a/docs/clients/promtail/README.md b/docs/clients/promtail/README.md index 51e5fed0716e9..f503928d41f98 100644 --- a/docs/clients/promtail/README.md +++ b/docs/clients/promtail/README.md @@ -6,14 +6,14 @@ deployed to every machine that has applications needed to be monitored. It primarily: -1. Discovers targets -2. Attaches labels to log streams -3. Pushes them to the Loki instance. +- Discovers targets +- Attaches labels to log streams +- Pushes them to the Loki instance. Currently, Promtail can tail logs from two sources: local log files and the systemd journal (on AMD64 machines only). -## Log File Discovery +## Log file discovery Before Promtail can ship any data from log files to Loki, it needs to find out information about its environment. Specifically, this means discovering @@ -32,12 +32,12 @@ Just like Prometheus, `promtail` is configured using a `scrape_configs` stanza. drop, and the final metadata to attach to the log line. Refer to the docs for [configuring Promtail](configuration.md) for more details. -## Receiving Logs From Syslog +## Receiving logs From Syslog When the [Syslog Target](./scraping.md#syslog-target) is being used, logs can be written with the syslog protocol to the configured port. -## Labeling and Parsing +## Labeling and parsing During service discovery, metadata is determined (pod name, filename, etc.) that may be attached to the log line as a label for easier identification when diff --git a/docs/clients/promtail/installation.md b/docs/clients/promtail/installation.md index c32d16e359729..fdf2dc3fc6f88 100644 --- a/docs/clients/promtail/installation.md +++ b/docs/clients/promtail/installation.md @@ -1,4 +1,4 @@ -# Installing Promtail +# Install Promtail Promtail is distributed as a [binary](#binary), [Docker container](#docker), and [Helm chart](#helm). @@ -12,7 +12,7 @@ Every release includes binaries for Promtail which can be found on the ```bash # modify tag to most recent version -$ docker pull grafana/promtail:1.5.0 +docker pull grafana/promtail:1.5.0 ``` ## Helm @@ -23,13 +23,13 @@ Make sure that Helm is Then you can add Loki's chart repository to Helm: ```bash -$ helm repo add loki https://grafana.github.io/loki/charts +helm repo add loki https://grafana.github.io/loki/charts ``` And the chart repository can be updated by running: ```bash -$ helm repo update +helm repo update ``` Finally, Promtail can be deployed with: @@ -40,7 +40,7 @@ $ helm upgrade --install promtail loki/promtail --set "loki.serviceName=loki" ## Kubernetes -### DaemonSet (Recommended) +### DaemonSet (recommended) A `DaemonSet` will deploy `promtail` on every node within a Kubernetes cluster. diff --git a/docs/getting-started/get-logs-into-loki.md b/docs/getting-started/get-logs-into-loki.md new file mode 100644 index 0000000000000..572fadd69a7f8 --- /dev/null +++ b/docs/getting-started/get-logs-into-loki.md @@ -0,0 +1,74 @@ +# Get logs into Loki + +After you [install and run Loki](./installation/local.md), you probably want to get logs from other applications into it. + +To get application logs into Loki, you need to edit the [Promtail](./clients/promtail/README.md) config file. + +Detailed information about configuring Promtail is available in [Promtail configuration](./clients/promtail/configuration.md). + +The following instructions should help you get started. + +1. If you haven't already, download a Promtail configuration file. Keep track of where it is, because you will need to cite it when you run the binary. + +``` +wget https://github.com/grafana/loki/blob/master/cmd/promtail/promtail-local-config.yaml +``` + +2. Open the config file in the text editor of your choice. It should look similar to this: + +``` +server: + http_listen_port: 9080 + grpc_listen_port: 0 + +positions: + filename: /tmp/positions.yaml + +clients: + - url: http://loki:3100/loki/api/v1/push + +scrape_configs: +- job_name: system + static_configs: + - targets: + - localhost + labels: + job: varlogs + __path__: /var/log/*log +``` + + The seven lines under `scrape_configs` are what send the logs that Loki generates to Loki, which then outputs them in the command line and http://localhost:3100/metrics. + +3. Copy the seven lines under `scrape_configs`, and then paste them under the original job (you can also just edit the original seven lines). + + Below is an example that sends logs from a default Grafana installation to Loki. We updated the following fields: + - job_name - This differentiates the logs collected from other log groups. + - targets - Optional for static_configs, however is often defined because in older versions of Promtail it was not optional. This was an artifact from directly using the Prometheus service discovery code which required this entry. + - labels - Static label to apply to every log line scraped by this definition. Good examples would be environment name, job name, or app name. + - __path__ - The path to where the logs are stored that I want Loki to consume. + +``` +- job_name: grafana + static_configs: + - targets: + - grafana + labels: + job: grafana + __path__: "C:/Program Files/GrafanaLabs/grafana/data/log/grafana.log" +``` + +4. Enter the following command to run Promtail. Examples below assume you have put the config file in the same directory as the binary. + +**Windows** + +``` +`.\promtail-windows-amd64.exe --config.file=promtail-local-config.yaml` +``` + +**Linux** + +``` +./promtail-linux-amd64 -config.file=promtail-local-config.yaml +``` + +You should now see your application logs. If you are using Grafana, you might need to refresh your instance in order to see the logs. diff --git a/docs/installation/README.md b/docs/installation/README.md index c6655b7cede11..9ba4e0b52fcda 100644 --- a/docs/installation/README.md +++ b/docs/installation/README.md @@ -1,6 +1,21 @@ -# Installing Loki +# Install Loki -1. [Installing using Tanka (recommended)](./tanka.md) -2. [Installing through Helm](./helm.md) -3. [Installing through Docker or Docker Compose](./docker.md) -4. [Installing locally](./local.md) +## Installation methods + +Instructions for different methods of installing Loki and Promtail. + +- [Install using Tanka (recommended)](./tanka.md) +- [Install through Helm](./helm.md) +- [Install through Docker or Docker Compose](./docker.md) +- [Install and run locally](./local.md) +- [Install from source](./install-from-source.md) + +## General process + +In order to run Loki, you must: + +1. Download and install both Loki and Promtail. +1. Download config files for both programs. +1. Start Loki. +1. Update the Promtail config file to get your logs into Loki. +1. Start Promtail. diff --git a/docs/installation/docker.md b/docs/installation/docker.md index 16fde9d577b15..3c8e077149e3e 100644 --- a/docs/installation/docker.md +++ b/docs/installation/docker.md @@ -1,7 +1,7 @@ -# Installing Loki with Docker or Docker Compose +# Install Loki with Docker or Docker Compose -You can install Loki with Docker or Docker Compose for evaluating, testing, or developing Loki. -For production, we recommend Tanka or Helm. +You can install Loki and Promtail with Docker or Docker Compose if you are evaluating, testing, or developing Loki. +For production, we recommend installing with Tanka or Helm. ## Prerequisites @@ -25,8 +25,7 @@ When finished, `loki-config.yaml` and `promtail-config.yaml` are downloaded in t Navigate to http://localhost:3100/metrics to view the metrics and http://localhost:3100/ready for readiness. -As of v1.5.0, image is configured to run by default as user loki with UID `10001` and GID `10001`. You can use a different user, specially if you are using bind mounts, by specifying uid with docker run command -by specifying `--user=UID` with numeric UID suited to your needs. +As of v1.5.0, image is configured to run by default as user loki with UID `10001` and GID `10001`. You can use a different user, specially if you are using bind mounts, by specifying the UID with a `docker run` command and using `--user=UID` with numeric UID suited to your needs. **Windows** @@ -46,7 +45,9 @@ Navigate to http://localhost:3100/metrics to view the output. ## Install with Docker Compose +Run the following commands in your command line. They work for Windows or Linux systems. + ```bash -$ wget https://raw.githubusercontent.com/grafana/loki/v1.5.0/production/docker-compose.yaml -O docker-compose.yaml -$ docker-compose -f docker-compose.yaml up +wget https://raw.githubusercontent.com/grafana/loki/v1.5.0/production/docker-compose.yaml -O docker-compose.yaml +docker-compose -f docker-compose.yaml up ``` diff --git a/docs/installation/helm.md b/docs/installation/helm.md index b082228301579..31d9f70bf4cbd 100644 --- a/docs/installation/helm.md +++ b/docs/installation/helm.md @@ -1,19 +1,20 @@ -# Installing Loki with Helm +# Install Loki with Helm ## Prerequisites Make sure you have Helm [installed](https://helm.sh/docs/using_helm/#installing-helm) and -[deployed](https://helm.sh/docs/using_helm/#installing-tiller) to your cluster. Then add -[Loki's chart repository](https://github.com/grafana/loki/tree/master/production/helm/loki) to Helm: +[deployed](https://helm.sh/docs/using_helm/#installing-tiller) to your cluster. + +Add [Loki's chart repository](https://github.com/grafana/loki/tree/master/production/helm/loki) to Helm: ```bash -$ helm repo add loki https://grafana.github.io/loki/charts +helm repo add loki https://grafana.github.io/loki/charts ``` -You can update the chart repository by running: +To update the chart repository, run: ```bash -$ helm repo update +helm repo update ``` ## Deploy Loki to your cluster @@ -21,32 +22,32 @@ $ helm repo update ### Deploy with default config ```bash -$ helm upgrade --install loki loki/loki-stack +helm upgrade --install loki loki/loki-stack ``` ### Deploy in a custom namespace ```bash -$ helm upgrade --install loki --namespace=loki loki/loki +helm upgrade --install loki --namespace=loki loki/loki ``` ### Deploy with custom config ```bash -$ helm upgrade --install loki loki/loki --set "key1=val1,key2=val2,..." +helm upgrade --install loki loki/loki --set "key1=val1,key2=val2,..." ``` ### Deploy Loki Stack (Loki, Promtail, Grafana, Prometheus) ```bash -$ helm upgrade --install loki loki/loki-stack --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false +helm upgrade --install loki loki/loki-stack --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false ``` ### Deploy Loki Stack (Loki, Fluent Bit, Grafana, Prometheus) ```bash -$ helm upgrade --install loki loki/loki-stack \ - --set fluent-bit.enabled=true,promtail.enabled=false,grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false +helm upgrade --install loki loki/loki-stack \ + --set fluent-bit.enabled=true,promtail.enabled=false,grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false ``` ## Deploy Grafana to your cluster @@ -54,19 +55,19 @@ $ helm upgrade --install loki loki/loki-stack \ To install Grafana on your cluster with Helm, use the following command: ```bash -$ helm install stable/grafana -n loki-grafana +helm install stable/grafana -n loki-grafana ``` To get the admin password for the Grafana pod, run the following command: ```bash -$ kubectl get secret --namespace loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo +kubectl get secret --namespace loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` To access the Grafana UI, run the following command: ```bash -$ kubectl port-forward --namespace service/loki-grafana 3000:80 +kubectl port-forward --namespace service/loki-grafana 3000:80 ``` Navigate to `http://localhost:3000` and login with `admin` and the password @@ -79,8 +80,7 @@ If Loki and Promtail are deployed on different clusters you can add an Ingress in front of Loki. By adding a certificate you create an HTTPS endpoint. For extra security you can also enable Basic Authentication on the Ingress. -In Promtail, set the following values to communicate using HTTPS and basic -authentication: +In Promtail, set the following values to communicate using HTTPS and basic authentication: ```yaml loki: diff --git a/docs/installation/install-from-source.md b/docs/installation/install-from-source.md new file mode 100644 index 0000000000000..53584369705e4 --- /dev/null +++ b/docs/installation/install-from-source.md @@ -0,0 +1,26 @@ +# Build from source + +In order to build Loki manually, you need to clone the GitHub repo and then `make Loki`. + +## Prerequisites + +- Go 1.13 or later +- Make +- Docker (for updating protobuf files and yacc files) + +## Build manually on your local system + +Clone Loki to `$GOPATH/src/github.com/grafana/loki`: + +```bash +git clone https://github.com/grafana/loki $GOPATH/src/github.com/grafana/loki +``` + +Then change into that directory and run `make loki`: + +```bash +cd $GOPATH/src/github.com/grafana/loki +make loki +``` + +A file at ./cmd/loki/loki will be created and is the final built binary. diff --git a/docs/installation/local.md b/docs/installation/local.md index c75c6f8878d5e..2a65b5a0716e0 100644 --- a/docs/installation/local.md +++ b/docs/installation/local.md @@ -1,11 +1,43 @@ -# Installing Loki Locally +# Install and run Loki locally -## Release Binaries +In order to log events with Loki, you must download and install both Promtail and Loki. +- Loki is the logging engine. +- Promtail sends logs to Loki. + +## Install and run + +1. Navigate to the [release page](https://github.com/grafana/loki/releases/). +2. Scroll down to the Assets section under the version that you want to install. +3. Download the Loki and Promtail .zip files that correspond to your system. + **Note:** Do not download LogCLI or Loki Canary at this time. [LogCLI](./getting-started/logcli.md) allows you to run Loki queries in a command line interface. [Loki Canary](./operations/loki-canary.md) is a tool to audit Loki performance. +4. Unzip the package contents into the same directory. This is where the two programs will run. +5. In the command line, change directory (`cd` on most systems) to the directory with Loki and Promtail. Copy and paste the commands below into your command line to download generic configuration files: +``` +wget https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml +wget https://raw.githubusercontent.com/grafana/loki/master/cmd/promtail/promtail-local-config.yaml +``` +6. Enter the following command to start Loki: + +**Windows** + +``` +.\loki-windows-amd64.exe --config.file=loki-local-config.yaml +``` + +**Linux** +``` +./promtail-linux-amd64 -config.file=promtail-local-config.yaml +``` + +Loki runs and displays Loki logs in your command line and on http://localhost:3100/metrics. + +Congratulations, Loki is installed and running! Next, you might want edit the Promtail config file to [get logs into Loki](./getting-started/get-logs-into-loki.md). + +## Release binaries - openSUSE Linux only Every release includes binaries for Loki which can be found on the [Releases page](https://github.com/grafana/loki/releases). - ## Community openSUSE Linux packages The community provides packages of Loki for openSUSE Linux. To install: @@ -13,35 +45,9 @@ The community provides packages of Loki for openSUSE Linux. To install: 1. Add the repository `https://download.opensuse.org/repositories/security:/logging/` to your system. For example, if you are using Leap 15.1, run `sudo zypper ar https://download.opensuse.org/repositories/security:/logging/openSUSE_Leap_15.1/security:logging.repo ; sudo zypper ref` -2. Install the Loki package with `zypper in loki` -3. Enable the Loki and Promtail services: +1. Install the Loki package with `zypper in loki` +1. Enable the Loki and Promtail services: - `systemd start loki && systemd enable loki` - `systemd start promtail && systemd enable promtail` -4. Modify the configuration files as needed: `/etc/loki/promtail.yaml` and +1. Modify the configuration files as needed: `/etc/loki/promtail.yaml` and `/etc/loki/loki.yaml`. - -## Manual Build - -### Prerequisites - -- Go 1.13 or later -- Make -- Docker (for updating protobuf files and yacc files) - -### Building - -Clone Loki to `$GOPATH/src/github.com/grafana/loki`: - -```bash -$ git clone https://github.com/grafana/loki $GOPATH/src/github.com/grafana/loki -``` - -Then change into that directory and run `make loki`: - -```bash -$ cd $GOPATH/src/github.com/grafana/loki -$ make loki - -# A file at ./cmd/loki/loki will be created and is the -# final built binary. -``` diff --git a/docs/installation/tanka.md b/docs/installation/tanka.md index 79b1be4e576f9..3a8564e5452a4 100644 --- a/docs/installation/tanka.md +++ b/docs/installation/tanka.md @@ -1,4 +1,4 @@ -# Installing Loki with Tanka +# Install Loki with Tanka [Tanka](https://tanka.dev) is a reimplementation of [Ksonnet](https://ksonnet.io) that Grafana Labs created after Ksonnet was @@ -6,7 +6,7 @@ deprecated. Tanka is used by Grafana Labs to run Loki in production. ## Prerequisites -Grab the latest version of Tanka (at least version v0.5.0) for the `tk env` +Install the latest version of Tanka (at least version v0.5.0) for the `tk env` commands. Prebuilt binaries for Tanka can be found at the [Tanka releases URL](https://github.com/grafana/tanka/releases). @@ -23,7 +23,7 @@ tk env add environments/loki --namespace=loki --server= ## Deploying -Grab the Loki & Promtail module using `jb`: +Download and install the Loki and Promtail module using `jb`: ```bash go get -u github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb @@ -75,8 +75,7 @@ loki + promtail + gateway { } ``` -Notice that `container_root_path` is your own data root for the Docker Daemon, -run `docker info | grep "Root Dir"` to get it. +Notice that `container_root_path` is your own data root for the Docker Daemon. +Run `docker info | grep "Root Dir"` to get the root path. -Run `tk show environments/loki` to see the manifests that will be deployed to the cluster and -finally run `tk apply environments/loki` to deploy it. +Run `tk show environments/loki` to see the manifests that will be deployed to the cluster. Run `tk apply environments/loki` to deploy the manifests.