Skip to content

Commit

Permalink
obsservice: ingest events
Browse files Browse the repository at this point in the history
This patch adds a couple of things:
1. an RPC endpoint to CRDB for streaming out events. This RPC service is
   called by the Obs Service.
2. a library in CRDB for exporting events.
3. code in the Obs Service for ingesting the events and writing them
   into a table in the sink cluster.

The first use of the event exporting is for the system.eventlog events.
All events written to that table are now also exported. Once the Obs
Service takes hold in the future, I hope we can remove system.eventlog.

The events are represented using OpenTelemetry protos. Unfortunately,
I've had to copy over the otel protos to our tree because I couldn't
figure out a vendoring solution. Problems encountered for vendoring are:
1. The repo where these protos live
(https://github.com/open-telemetry/opentelemetry-proto) is not
go-get-able. This means hacks are needed for our vendoring tools.
2. Some of the protos in that repo do not build with gogoproto (they
only build with protoc), because they use the new-ish "optional"
option on some fields. The logs protos that we use in this patch do not
have this problem, but others do (so we'll need to figure something out
in the future when dealing with the metrics proto).  FWIW, the
OpenTelemetry Collector ironically has the same problem (it uses
gogoproto), and it solved it through a sed that changes all the optional
fields to one-of's.
3. Even if you solve the first two problems, the next one is that we
already have a dependency on these compiled protos in our tree
(go.opentelemetry.io/proto/otlp). This repo contains generated code,
using protoc. We need it because of our use of the otlp trace exporter.
Bringing in the protos again, and building them with gogo, results in go
files that want to live in the same package/have the same import path.
So changing the import paths is needed.

Between all of these, I've given up - at least for the moment - and
decided to copy over to our tree the few protos that we actually need.
I'm also changing their import paths. You'll notice that there is a
script that codifies the process of transforming the needed protos from
their otel upstream.

Release note: None
  • Loading branch information
andreimatei committed Jul 19, 2022
1 parent 6e9f5a9 commit 8999d60
Show file tree
Hide file tree
Showing 36 changed files with 1,757 additions and 142 deletions.
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,7 @@
/pkg/util/stop/ @cockroachdb/kv-prs
/pkg/util/tracing @cockroachdb/obs-inf-prs
/pkg/workload/ @cockroachdb/sql-experience-noreview
/pkg/obs/ @cockroachdb/obs-inf-prs
/pkg/obsservice/ @cockroachdb/obs-inf-prs

# Own all bazel files to dev-inf, but don't request reviews for them
Expand Down
4 changes: 4 additions & 0 deletions pkg/gen/protobuf.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ PROTOBUF_SRCS = [
"//pkg/kv/kvserver/protectedts/ptstorage:ptstorage_go_proto",
"//pkg/kv/kvserver/readsummary/rspb:rspb_go_proto",
"//pkg/kv/kvserver:kvserver_go_proto",
"//pkg/obsservice/obspb/opentelemetry-proto/common/v1:v1_go_proto",
"//pkg/obsservice/obspb/opentelemetry-proto/logs/v1:v1_go_proto",
"//pkg/obsservice/obspb/opentelemetry-proto/resource/v1:v1_go_proto",
"//pkg/obsservice/obspb:obs_go_proto",
"//pkg/roachpb:roachpb_go_proto",
"//pkg/rpc:rpc_go_proto",
"//pkg/server/diagnostics/diagnosticspb:diagnosticspb_go_proto",
Expand Down
24 changes: 24 additions & 0 deletions pkg/obs/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
load("@io_bazel_rules_go//go:def.bzl", "go_library")

go_library(
name = "obs",
srcs = ["event_exporter.go"],
importpath = "github.com/cockroachdb/cockroach/pkg/obs",
visibility = ["//visibility:public"],
deps = [
"//pkg/obsservice/obspb",
"//pkg/obsservice/obspb/opentelemetry-proto/common/v1:common",
"//pkg/obsservice/obspb/opentelemetry-proto/logs/v1:logs",
"//pkg/obsservice/obspb/opentelemetry-proto/resource/v1:resource",
"//pkg/roachpb",
"//pkg/util/log",
"//pkg/util/mon",
"//pkg/util/stop",
"//pkg/util/syncutil",
"//pkg/util/timeutil",
"//pkg/util/uuid",
"@com_github_cockroachdb_errors//:errors",
"@com_github_cockroachdb_redact//:redact",
"@org_golang_google_grpc//peer",
],
)
15 changes: 15 additions & 0 deletions pkg/obs/doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
// Copyright 2022 The Cockroach Authors.
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.

package obs

// Package obs represents the client library for the Observability Service.
//
// The Obs Service lives in pkg/obsservice.
Loading

0 comments on commit 8999d60

Please sign in to comment.