From ec668fdcba2aae0b5151c3bddfe5a754e26a904a Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Tue, 25 Apr 2023 15:57:32 -0600 Subject: [PATCH] Add OTel logs source doc to doc website. (#3608) * Add otel_logs_source doc. Signed-off-by: carolxob * Content updates mainly to formatting. Signed-off-by: carolxob * Converted list entries into tables and fixed formatting. Signed-off-by: carolxob * Minor updates. Signed-off-by: carolxob * Minor edits. Signed-off-by: carolxob * Minor edits. Signed-off-by: carolxob * Minor edits. Signed-off-by: carolxob * Adjusted nav_order in metadata. Signed-off-by: carolxob * Content edits from tech review. Signed-off-by: carolxob * Edit made based on technical review feedback. Signed-off-by: carolxob * Major revisions to table formatting and headers. Signed-off-by: carolxob * Minor adjustements. Signed-off-by: carolxob * Adjusted default values for items in the SSL table. Signed-off-by: carolxob * Reworded two options in the SSL section. Signed-off-by: carolxob * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Minor changes. Signed-off-by: carolxob * Minor changes. Signed-off-by: carolxob * Minor updates. Signed-off-by: carolxob * Minor updates. Signed-off-by: carolxob * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Content updates from feedback. Signed-off-by: carolxob * Minor update. Signed-off-by: carolxob * Minor change. Signed-off-by: carolxob * Minor updates. Signed-off-by: carolxob * Minor updates. Signed-off-by: carolxob * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Minor updates. Signed-off-by: carolxob * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Update _data-prepper/pipelines/configuration/sources/otel-logs-source.md Co-authored-by: Nathan Bower Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> * Minor updates from editorial feedback. Signed-off-by: carolxob --------- Signed-off-by: carolxob Signed-off-by: Caroline <113052567+carolxob@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower --- .../configuration/sources/otel-logs-source.md | 67 +++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 _data-prepper/pipelines/configuration/sources/otel-logs-source.md diff --git a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md new file mode 100644 index 0000000000..58d8a2b059 --- /dev/null +++ b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md @@ -0,0 +1,67 @@ +--- +layout: default +title: otel_logs_source +parent: Sources +grand_parent: Pipelines +nav_order: 25 +--- + +# otel_logs_source + + +The `otel_logs_source` source is an OpenTelemetry source that follows the [OpenTelemetry Protocol Specification](https://github.com/open-telemetry/oteps/blob/master/text/0035-opentelemetry-protocol.md) and receives logs from the OTel Collector in the form of `ExportLogsServiceRequest` records. + +This source supports the `OTLP/gRPC` protocol. +{: .note} + +## Configuration + +You can configure the `otel_logs_source` source with the following options. + +| Option | Type | Description | +| :--- | :--- | :--- | +| port | int | Represents the port that the `otel_logs_source` source is running on. Default value is `21892`. | +| path | string | Represents the path for sending unframed HTTP requests. You can use this option to support an unframed gRPC request with an HTTP idiomatic path to a configurable path. The path should start with `/`, and its length should be at least 1. The `/opentelemetry.proto.collector.logs.v1.LogsService/Export` endpoint is disabled for both gRPC and HTTP requests if the path is configured. The path can contain a `${pipelineName}` placeholder, which is replaced with the pipeline name. If the value is empty and `unframed_requests` is `true`, then the path that the source provides is `/opentelemetry.proto.collector.logs.v1.LogsService/Export`. | +| request_timeout | int | Represents the request timeout duration in milliseconds. Default value is `10000`. | +| health_check_service | Boolean | Enables the gRPC health check service under `grpc.health.v1/Health/Check`. Default value is `false`. | +| proto_reflection_service | Boolean | Enables a reflection service for Protobuf services (see [ProtoReflectionService](https://grpc.github.io/grpc-java/javadoc/io/grpc/protobuf/services/ProtoReflectionService.html) and [gRPC reflection](https://github.com/grpc/grpc-java/blob/master/documentation/server-reflection-tutorial.md)). Default value is `false`. | +| unframed_requests | Boolean | Enables requests that are not framed using the gRPC wire protocol. Default value is `false`. | +| thread_count | int | The number of threads to keep in the `ScheduledThreadPool`. Default value is `500`. | +| max_connection_count | int | The maximum number of open connections allowed. Default value is `500`. | + +### SSL + +You can configure SSL in the `otel_logs_source` source with the following options. + +| Option | Type | Description | +| :--- | :--- | :--- | +| ssl | Boolean | Enables TLS/SSL. Default value is `true`. | +| sslKeyCertChainFile | string | Represents the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | +| sslKeyFile | string | Represents the SSL key file path or Amazon S3 path. For example, see the Amazon S3 path `s3:///`. Required if `ssl` is set to `true`. | +| useAcmCertForSSL | Boolean | Enables TLS/SSL using a certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | +| acmCertificateArn | string | Represents the ACM certificate Amazon Resource Name (ARN). ACM certificates take precedence over Amazon S3 or local file system certificates. Required if `useAcmCertForSSL` is set to `true`. | +| awsRegion | string | Represents the AWS Region used by ACM or Amazon S3. Required if `useAcmCertForSSL` is set to `true` or `sslKeyCertChainFile` or `sslKeyFile` is the Amazon S3 path. | + +## Usage + +To get started, create a `pipeline.yaml` file and add `otel_logs_source` as the source: + +``` +source: + - otel_logs_source: +``` + +## Metrics + +You can use the following metrics with the `otel_logs_source` source. + +| Option | Type | Description | +| :--- | :--- | :--- | +| `requestTimeouts` | Counter | Measures the total number of requests that time out. | +| `requestsReceived` | Counter | Measures the total number of requests received by the `otel_logs_source` source. | +| `badRequests` | Counter | Measures the total number of requests that could not be parsed. | +| `requestsTooLarge` | Counter | Measures the total number of requests that exceed the maximum allowed size. Indicates that the size of the data being written into the buffer is beyond the buffer's maximum capacity. | +| `internalServerError` | Counter | Measures the total number of requests that are erroneous due to errors other than `requestTimeouts` or `requestsTooLarge`. | +| `successRequests` | Counter | Measures the total number of requests successfully written to the buffer. | +| `payloadSize` | Distribution summary | Measures the distribution of all incoming payload sizes. | +| `requestProcessDuration` | Timer | Measures the duration of request processing. | \ No newline at end of file