Port servicegraphprocessor to servicegraphconnector (#18389)

.chloggen/servicegraphconnector.yaml

@@ -0,0 +1,16 @@
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: new_component
+
+# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
+component: servicegraphconnector
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Add preliminary implementation as a connector.
+
+# One or more tracking issues related to the change
+issues: [18389]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:

.github/CODEOWNERS

@@ -29,6 +29,7 @@ cmd/telemetrygen/                                        @open-telemetry/collect
 confmap/provider/s3provider/                             @open-telemetry/collector-contrib-approvers @Aneurysm9
 
 connector/countconnector/                                @open-telemetry/collector-contrib-approvers @djaglowski @jpkrohling
+connector/servicegraphconnector/                         @open-telemetry/collector-contrib-approvers @jpkrohling @mapno
 
 examples/demo/                                           @open-telemetry/collector-contrib-approvers @open-telemetry/collector-approvers
 

.github/ISSUE_TEMPLATE/bug_report.yaml

@@ -25,6 +25,7 @@ body:
       - cmd/telemetrygen
       - confmap/provider/s3provider
       - connector/count
+      - connector/servicegraph
       - examples/demo
       - exporter/alibabacloudlogservice
       - exporter/awscloudwatchlogs

.github/ISSUE_TEMPLATE/feature_request.yaml

@@ -19,6 +19,7 @@ body:
       - cmd/telemetrygen
       - confmap/provider/s3provider
       - connector/count
+      - connector/servicegraph
       - examples/demo
       - exporter/alibabacloudlogservice
       - exporter/awscloudwatchlogs

.github/ISSUE_TEMPLATE/other.yaml

@@ -19,6 +19,7 @@ body:
       - cmd/telemetrygen
       - confmap/provider/s3provider
       - connector/count
+      - connector/servicegraph
       - examples/demo
       - exporter/alibabacloudlogservice
       - exporter/awscloudwatchlogs

.github/dependabot.yml

@@ -46,6 +46,10 @@ updates:
     directory: "/connector/countconnector"
     schedule:
       interval: "weekly"
+  - package-ecosystem: "gomod"
+    directory: "/connector/servicegraphconnector"
+    schedule:
+      interval: "weekly"
   - package-ecosystem: "gomod"
     directory: "/examples/demo/client"
     schedule:

connector/servicegraphconnector/Makefile

@@ -0,0 +1 @@
+include ../../Makefile.Common

connector/servicegraphconnector/README.md

@@ -0,0 +1,141 @@
+# Service Graph Connector
+
+| Status                   |                                                           |
+|------------------------- |---------------------------------------------------------- |
+| Stability                | [in development]                                          |
+| Supported pipeline types | See [Supported Pipeline Types](#supported-pipeline-types) |
+| Distributions            | []                                                        |
+
+## Supported Pipeline Types
+
+| [Exporter Pipeline Type] | [Receiver Pipeline Type] |
+| ------------------------ | ------------------------ |
+| traces                   | metrics                  |
+
+## Overview
+
+The service graphs connector builds a map representing the interrelationships between various services in a system.
+The connector will analyse trace data and generate metrics describing the relationship between the services.
+These metrics can be used by data visualization apps (e.g. Grafana) to draw a service graph.
+
+Service graphs are useful for a number of use-cases:
+
+* Infer the topology of a distributed system. As distributed systems grow, they become more complex. Service graphs can help you understand the structure of the system.
+* Provide a high level overview of the health of your system.
+Service graphs show error rates, latencies, among other relevant data.
+* Provide an historic view of a system’s topology.
+Distributed systems change very frequently,
+and service graphs offer a way of seeing how these systems have evolved over time.
+
+This component is based on [Grafana Tempo's service graph processor](https://github.com/grafana/tempo/tree/main/modules/generator/processor/servicegraphs).
+
+## How it works
+
+Service graphs work by inspecting traces and looking for spans with parent-children relationship that represent a request.
+The connector uses the OpenTelemetry semantic conventions to detect a myriad of requests.
+It currently supports the following requests:
+
+* A direct request between two services where the outgoing and the incoming span must have `span.kind` client and server respectively.
+* A request across a messaging system where the outgoing and the incoming span must have `span.kind` producer and consumer respectively.
+* A database request; in this case the connector looks for spans containing attributes `span.kind`=client as well as db.name.
+
+Every span that can be paired up to form a request is kept in an in-memory store,
+until its corresponding pair span is received or the maximum waiting time has passed.
+When either of these conditions are reached, the request is recorded and removed from the local store.
+
+Each emitted metrics series have the client and server label corresponding with the service doing the request and the service receiving the request.
+
+```
+traces_service_graph_request_total{client="app", server="db", connection_type="database"} 20
+```
+
+TLDR: The connector will try to find spans belonging to requests as seen from the client and the server and will create a metric representing an edge in the graph.
+
+## Metrics
+
+The following metrics are emitted by the connector:
+
+| Metric                                      | Type      | Labels                          | Description                                                  |
+|---------------------------------------------|-----------|---------------------------------|--------------------------------------------------------------|
+| traces_service_graph_request_total          | Counter   | client, server, connection_type | Total count of requests between two nodes                    |
+| traces_service_graph_request_failed_total   | Counter   | client, server, connection_type | Total count of failed requests between two nodes             |
+| traces_service_graph_request_server_seconds | Histogram | client, server, connection_type | Time for a request between two nodes as seen from the server |
+| traces_service_graph_request_client_seconds | Histogram | client, server, connection_type | Time for a request between two nodes as seen from the client |
+| traces_service_graph_unpaired_spans_total   | Counter   | client, server, connection_type | Total count of unpaired spans                                |
+| traces_service_graph_dropped_spans_total    | Counter   | client, server, connection_type | Total count of dropped spans                                 |
+
+Duration is measured both from the client and the server sides.
+
+Possible values for `connection_type`: unset, `messaging_system`, or `database`.
+
+Additional labels can be included using the `dimensions` configuration option. Those labels will have a prefix to mark where they originate (client or server span kinds).
+The `client_` prefix relates to the dimensions coming from spans with `SPAN_KIND_CLIENT`, and the `server_` prefix relates to the
+dimensions coming from spans with `SPAN_KIND_SERVER`.
+
+Since the service graph connector has to process both sides of an edge,
+it needs to process all spans of a trace to function properly.
+If spans of a trace are spread out over multiple instances, spans are not paired up reliably.
+A possible solution to this problem is using the [load balancing exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter)
+in a layer on front of collector instances running this connector.
+
+## Visualization
+
+Service graph metrics are natively supported by Grafana since v9.0.4.
+To run it, configure a Tempo data source's 'Service Graphs' by linking to the Prometheus backend where metrics are being sent:
+
+```yaml
+apiVersion: 1
+datasources:
+  # Prometheus backend where metrics are sent
+  - name: Prometheus
+    type: prometheus
+    uid: prometheus
+    url: <prometheus-url>
+    jsonData:
+        httpMethod: GET
+    version: 1
+  - name: Tempo
+    type: tempo
+    uid: tempo
+    url: <tempo-url>
+    jsonData:
+      httpMethod: GET
+      serviceMap:
+        datasourceUid: 'prometheus'
+    version: 1
+```
+
+## Example configuration
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      grpc:
+
+connectors:
+  servicegraph:
+    latency_histogram_buckets: [1,2,3,4,5]
+    dimensions:
+      - dimension-1
+      - dimension-2
+    store:
+      ttl: 1s
+      max_items: 10
+
+exporters:
+  prometheus/servicegraph:
+    endpoint: localhost:9090
+    namespace: servicegraph
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      exporters: [servicegraph]
+    metrics/servicegraph:
+      receivers: [servicegraph]
+      exporters: [prometheus/servicegraph]
+```
+
+[in development]: https://github.com/open-telemetry/opentelemetry-collector#development

connector/servicegraphconnector/factory.go

@@ -0,0 +1,19 @@
+// Copyright The OpenTelemetry Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//       http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package servicegraphconnector // import "github.com/open-telemetry/opentelemetry-collector-contrib/connector/servicegraphconnector"
+
+import "github.com/open-telemetry/opentelemetry-collector-contrib/processor/servicegraphprocessor"
+
+var NewFactory = servicegraphprocessor.NewConnectorFactory

connector/servicegraphconnector/go.mod

@@ -0,0 +1,41 @@
+module github.com/open-telemetry/opentelemetry-collector-contrib/connector/servicegraphconnector
+
+go 1.18
+
+require github.com/open-telemetry/opentelemetry-collector-contrib/processor/servicegraphprocessor v0.71.0
+
+require (
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/golang/protobuf v1.5.2 // indirect
+	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/knadh/koanf v1.5.0 // indirect
+	github.com/mitchellh/copystructure v1.2.0 // indirect
+	github.com/mitchellh/mapstructure v1.5.0 // indirect
+	github.com/mitchellh/reflectwalk v1.0.2 // indirect
+	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
+	github.com/modern-go/reflect2 v1.0.2 // indirect
+	go.opencensus.io v0.24.0 // indirect
+	go.opentelemetry.io/collector v0.71.0 // indirect
+	go.opentelemetry.io/collector/component v0.71.0 // indirect
+	go.opentelemetry.io/collector/confmap v0.71.0 // indirect
+	go.opentelemetry.io/collector/consumer v0.71.0 // indirect
+	go.opentelemetry.io/collector/featuregate v0.71.0 // indirect
+	go.opentelemetry.io/collector/pdata v1.0.0-rc5 // indirect
+	go.opentelemetry.io/collector/semconv v0.71.0 // indirect
+	go.opentelemetry.io/otel v1.13.0 // indirect
+	go.opentelemetry.io/otel/metric v0.36.0 // indirect
+	go.opentelemetry.io/otel/trace v1.13.0 // indirect
+	go.uber.org/atomic v1.10.0 // indirect
+	go.uber.org/multierr v1.9.0 // indirect
+	go.uber.org/zap v1.24.0 // indirect
+	golang.org/x/net v0.5.0 // indirect
+	golang.org/x/sys v0.4.0 // indirect
+	golang.org/x/text v0.6.0 // indirect
+	google.golang.org/genproto v0.0.0-20221202195650-67e5cbc046fd // indirect
+	google.golang.org/grpc v1.52.3 // indirect
+	google.golang.org/protobuf v1.28.1 // indirect
+)
+
+retract v0.65.0
+
+replace github.com/open-telemetry/opentelemetry-collector-contrib/processor/servicegraphprocessor => ../../processor/servicegraphprocessor/