diff --git a/content/en/docs/collector/_index.md b/content/en/docs/collector/_index.md index 88d8eca5bd20..e6137759dea4 100644 --- a/content/en/docs/collector/_index.md +++ b/content/en/docs/collector/_index.md @@ -11,7 +11,7 @@ receive, process and export telemetry data. It removes the need to run, operate, and maintain multiple agents/collectors in order to support open-source observability data formats (e.g. Jaeger, Prometheus, etc.) sending to one or more open-source or commercial back-ends. The Collector is the -default location instrumentation libraries export telemetry data. +default location instrumentation libraries export their telemetry data. Objectives: diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 778063848094..b4be28f6fc1a 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -3,6 +3,13 @@ title: "Configuration" weight: 20 --- +Please be sure to review the following documentation: + +- [Data Collection concepts](../../concepts/data-collection) in order to + understand the repositories applicable to the OpenTelemetry Collector. +- [Security + guidance](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security.md) + ## Basics The Collector consists of three components that access telemetry data: @@ -14,10 +21,10 @@ The Collector consists of three components that access telemetry data: These components once configured must be enabled via pipelines within the [service](#service) section. -Secondarily, there are extensions, which provide capabilities that can be added -to the Collector, but which do not require direct access to telemetry data and -are not part of pipelines. They are also enabled within the [service](#service) -section. +Secondarily, there are [extensions](#extensions), which provide capabilities +that can be added to the Collector, but which do not require direct access to +telemetry data and are not part of pipelines. They are also enabled within the +[service](#service) section. An example configuration would look like: @@ -110,9 +117,23 @@ service: ## Receivers A receiver, which can be push or pull based, is how data gets into the -Collector. One or more receivers must be configured. By default, no receivers +Collector. Receivers may support one or more [data +sources](../../concepts/data-sources). + +The `receivers:` section is how receivers are configured. Many receivers come +with default settings so simply specifying the name of the receiver is enough +to configure it (for example, `zipkin:`). If configuration is required or a +user wants to change the default configuration then such configuration must be +defined in this section. Configuration parameters specified for which the +receiver provides a default configuration are overridden. + +> Configuring a receiver does not enable it. Receivers are enabled via +> pipelines within the [service](#service) section. + +One or more receivers must be configured. By default, no receivers are configured. A basic example of all available receivers is provided below. -For detailed receiver configuration, please see the [receiver + +> For detailed receiver configuration, please see the [receiver README.md](https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/README.md). ```yaml @@ -171,9 +192,20 @@ receivers: ## Processors Processors are run on data between being received and being exported. -Processors are optional though some are recommended. A basic example of all -available processors is provided below. For detailed processor configuration, -please see the [processor +Processors are optional though [some are +recommended](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor#recommended-processors). + +The `processors:` section is how processors are configured. Processors may come +with default settings, but many require configuration. Any configuration for a +processor must be done in this section. Configuration parameters specified for +which the processor provides a default configuration are overridden. + +> Configuring a processor does not enable it. Processors are enabled via +> pipelines within the [service](#service) section. + +A basic example of all available processors is provided below. + +> For detailed processor configuration, please see the [processor README.md](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/README.md). ```yaml @@ -238,10 +270,22 @@ processors: ## Exporters An exporter, which can be push or pull based, is how you send data to one or -more backends/destinations. One or more exporters must be configured. By -default, no exporters are configured. A basic example of all available -exporters is provided below. For detailed exporter configuration, please see -the [exporter +more backends/destinations. Exporters may support one or more [data +sources](../../concepts/data-sources). + +The `exporters:` section is how exporters are configured. Exporters may come +with default settings, but many require configuration to specify at least the +destination and security settings. Any configuration for an exporter must be +done in this section. Configuration parameters specified for which the exporter +provides a default configuration are overridden. + +> Configuring an exporter does not enable it. Exporters are enabled via +> pipelines within the [service](#service) section. + +One or more exporters must be configured. By default, no exporters +are configured. A basic example of all available exporters is provided below. + +> For detailed exporter configuration, please see the [exporter README.md](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/README.md). ```yaml @@ -292,11 +336,25 @@ exporters: ## Extensions -Extensions are available for tasks that do not involve processing telemetry -data. Examples of extensions include health monitors. Extensions are optional. +Extensions are available primarily for tasks that do not involve processing telemetry +data. Examples of extensions include health monitoring, service discovery, and +data forwarding. Extensions are optional. + +The `extensions:` section is how extensions are configured. Many extensions +come with default settings so simply specifying the name of the extension is +enough to configure it (for example, `health_check:`). If configuration is +required or a user wants to change the default configuration then such +configuration must be defined in this section. Configuration parameters +specified for which the extension provides a default configuration are +overridden. + +> Configuring an extension does not enable it. Extensions are enabled within +> the [service](#service) section. + By default, no extensions are configured. A basic example of all available -extensions is provided below. For detailed extension configuration, please see -the [extension +extensions is provided below. + +> For detailed extension configuration, please see the [extension README.md](https://github.com/open-telemetry/opentelemetry-collector/blob/main/extension/README.md). ```yaml @@ -309,10 +367,10 @@ extensions: ## Service The service section is used to configure what components are enabled in the -Collector based on the configuration found in the receivers, -processors, exporters, and extensions sections. If a component is configured, -but not defined within the service section then it is not enabled. The service -section consists of two sub-sections: +Collector based on the configuration found in the receivers, processors, +exporters, and extensions sections. If a component is configured, but not +defined within the service section then it is not enabled. The service section +consists of two sub-sections: - extensions - pipelines @@ -326,9 +384,9 @@ Extensions consist of a list of all extensions to enable. For example: Pipelines can be of the following types: -- traces (stable): collects and processes trace data. -- metrics (alpha): collects and processes metric data. -- logs (alpha): collects and processes log data. +- traces: collects and processes trace data. +- metrics: collects and processes metric data. +- logs: collects and processes log data. A pipeline consists of a set of receivers, processors and exporters. Each receiver/processor/exporter must be defined in the configuration outside of the @@ -359,8 +417,8 @@ service: ### Configuration Environment Variables -The use of environment variables is supported in the Collector configuration. -For example: +The use and expansion of environment variables is supported in the Collector +configuration. For example: ```yaml processors: diff --git a/content/en/docs/collector/getting-started.md b/content/en/docs/collector/getting-started.md index 8804ee92b389..e39b08c86cc7 100644 --- a/content/en/docs/collector/getting-started.md +++ b/content/en/docs/collector/getting-started.md @@ -3,24 +3,19 @@ title: "Getting Started" weight: 1 --- -The OpenTelemetry Collector consists of the following projects: - -- [opentelemetry-collector](https://github.com/open-telemetry/opentelemetry-collector): - The core repository with primary capabilities and support for major - open-source receivers and exporters. -- [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib): - The community repository which is a superset of opentelemetry-collector plus - non-primary capabilities and support for other open-source and commercial - receivers and exporters. +Please be sure to review the [Data Collection +documentation](../../concepts/data-collection) in order to understand the +deployment models, components, and repositories applicable to the OpenTelemetry +Collector. ## Deployment -The OpenTelemetry Collector consists of a single binary and two deployment methods: +The OpenTelemetry Collector consists of a single binary and two primary deployment methods: -1. An agent running with the application or on the same host as the application -(e.g. binary, sidecar, or daemonset). -2. A gateway running as a standalone service (e.g. container or deployment) -typically per cluster, datacenter or region. +- **Agent:** A Collector instance running with the application or on the same + host as the application (e.g. binary, sidecar, or daemonset). +- **Gateway:** One or more Collector instances running as a standalone service + (e.g. container or deployment) typically per cluster, datacenter or region. ### Agent @@ -29,16 +24,21 @@ doing so, the Agent is capable of receiving telemetry data (push and pull based) as well as enhancing telemetry data with metadata such as custom tags or infrastructure information. In addition, the Agent can offload responsibilities that client instrumentation would otherwise need to handle including batching, -retry, encryption, compression and more. +retry, encryption, compression and more. OpenTelemetry instrumentation +libraries by default export their data assuming a locally running Collector is +available. ### Gateway -Additionally, a Gateway can be deployed in every cluster, datacenter, or region. -The Gateway runs as a standalone service and can offer advanced capabilities -over the Agent including tail-based sampling. In addition, the Gateway can -limit the number of egress points required to send data as well as consolidate -API token management. Each Collector instance operates independently so it is -easy to scale the architecture based on performance needs. +Additionally, a Gateway cluster can be deployed in every cluster, datacenter, +or region. A Gateway cluster runs as a standalone service and can offer +advanced capabilities over the Agent including tail-based sampling. In +addition, a Gateway cluster can limit the number of egress points required to +send data as well as consolidate API token management. Each Collector instance +in a Gateway cluster operates independently so it is easy to scale the +architecture based on performance needs with a simple load balancer. If a +gateway cluster is deployed, it usually receives data from Agents deployed +within an environment. ## Getting Started @@ -56,19 +56,26 @@ $ git clone git@github.com:open-telemetry/opentelemetry-collector.git; \ ### Docker -Starts a Docker container of the -[core](https://github.com/open-telemetry/opentelemetry-collector) -version of the Collector with all receivers enabled and exports all the data it -receives locally to a file. Data is sent to the container and the container -scrapes its own Prometheus metrics. +Every release of the Collector is published to Docker Hub and comes with a +default configuration file. + +```bash +$ docker run otel/opentelemetry-collector +``` + +In addition, you can use the local example provided. This example starts a +Docker container of the +[core](https://github.com/open-telemetry/opentelemetry-collector) version of +the Collector with all receivers enabled and exports all the data it receives +locally to a file. Data is sent to the container and the container scrapes its +own Prometheus metrics. ```bash $ git clone git@github.com:open-telemetry/opentelemetry-collector.git; \ cd opentelemetry-collector/examples; \ go build main.go; ./main & pid1="$!"; docker run --rm -p 13133:13133 -p 14250:14250 -p 14268:14268 \ - -p 55678-55680:55678-55680 -p 6060:6060 -p 7276:7276 -p 8888:8888 \ - -p 9411:9411 -p 9943:9943 \ + -p 55678-55680:55678-55680 -p 8888:8888 -p 9411:9411 \ -v "${PWD}/otel-local-config.yaml":/otel-local-config.yaml \ --name otelcol otel/opentelemetry-collector \ --config otel-local-config.yaml; \ @@ -93,6 +100,47 @@ features such as automatic upgrade handling, `Service` configuration based on the OpenTelemetry configuration, automatic sidecar injection into deployments, among others. +### Linux Packaging + +Every Collector release includes DEB and RPM packaging for Linux amd64/arm64 +systems. The packaging includes a default configuration that can be found at +`/etc/otel-collector/config.yaml` post-installation. + +> Please note that systemd is require for automatic service configuration + +To get started on Debian systems run the following replacing `v0.20.0` with the +version of the Collector you wish to run and `amd64` with the appropriate +architecture. + +```bash +$ sudo apt-get update +$ sudo apt-get -y install wget systemctl +$ wget https://github.com/open-telemetry/opentelemetry-collector/releases/download/v0.20.0/otel-collector_0.20.0_amd64.deb +$ dpkg -i otel-collector_0.20.0_amd64.deb +``` + +To get started on Red Hat systems run the following replacing `v0.20.0` with the +version of the Collector you wish to run and `x86_64` with the appropriate +architecture. + +```bash +$ sudo yum update +$ sudo yum -y install wget systemctl +$ wget https://github.com/open-telemetry/opentelemetry-collector/releases/download/v0.20.0/otel-collector_0.20.0-1_x86_64.rpm +$ rpm -ivh otel-collector_0.20.0-1_x86_64.rpm +``` + +### Windows Packaging + +Every Collector release includes EXE and MSI packaging for Linux amd64 systems. +The MSI packaging includes a default configuration that can be found at +`\Program Files\OpenTelemetry Collector\config.yaml`. + +> Please note the Collector service is not automatically started + +The easiest way to get started is to double-click the MSI package and follow +the wizard. Silent installation is also available. + ### Local Builds the latest version of the collector based on the local operating system, @@ -102,23 +150,7 @@ Prometheus metrics. ```bash $ git clone git@github.com:open-telemetry/opentelemetry-collector.git; \ - cd opentelemetry-collector; make otelcol; \ + cd opentelemetry-collector; make install-tools; make otelcol; \ go build examples/demo/app/main.go; ./main & pid1="$!"; \ ./bin/otelcol_$(go env GOOS)_$(go env GOARCH) --config ./examples/local/otel-config.yaml; kill $pid1 ``` - -## Other - -The OpenTelemetry collector can be extended or embedded into other applications. - -The list of applications extending the collector: - -- [opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) -- [jaeger](https://github.com/jaegertracing/jaeger/tree/master/cmd/opentelemetry) - -A guide on how to create your own distribution is available in this blog post: -["Building your own OpenTelemetry Collector distribution"](https://medium.com/p/42337e994b63) - -If you are building your own distribution, the [OpenTelemetry Collector -Builder](https://github.com/observatorium/opentelemetry-collector-builder) -might be a good starting point.