From 855d6319ff1c31feec157ac10d6b9f5a45c62d27 Mon Sep 17 00:00:00 2001 From: Chris Smith Date: Mon, 4 Dec 2023 16:18:12 -0700 Subject: [PATCH] docs(main): add telemetry discussion to debug.md refs: googleapis/google-api-go-client#2127 refs: #8655 --- debug.md | 170 ++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 162 insertions(+), 8 deletions(-) diff --git a/debug.md b/debug.md index f2a608ce24d7..014cd29e2335 100644 --- a/debug.md +++ b/debug.md @@ -1,21 +1,31 @@ -# Debugging tips and tricks +# Logging, debugging and telemetry + +Logging, debugging and telemetry all capture data that can be used for +troubleshooting. Logging records specific events and transactions. Debugging +exposes values for immediate analysis. Telemetry is suitable for production use +and can serve both logging and monitoring purposes. Telemetry tracing follows +requests through a system to provide a view of component interactions. Telemetry +metrics collects data for significant performance indicators, offering insights +into a system's health. + +## Logging and debugging While working with the Go Client libraries you may run into some situations where you need a deeper level of understanding about what is going on in order to solve your problem. Here are some tips and tricks that you can use in these -cases. *Note* that many of the tips in this document will have a performance +cases. *Note* that many of the tips in this section will have a performance impact and are therefore not recommended for sustained production use. Use these tips locally or in production for a *limited time* to help get a better understanding of what is going on. -## HTTP based clients +### HTTP based clients All of our auto-generated clients have a constructor to create a client that uses HTTP/JSON instead of gRPC. Additionally a couple of our hand-written clients like Storage and Bigquery are also HTTP based. Here are some tips for debugging these clients. -### Try setting Go's HTTP debug variable +#### Try setting Go's HTTP debug variable Try setting the following environment variable for verbose Go HTTP logging: GODEBUG=http2debug=1. To read more about this feature please see the godoc for @@ -24,7 +34,7 @@ GODEBUG=http2debug=1. To read more about this feature please see the godoc for *WARNING*: Enabling this debug variable will log headers and payloads which may contain private information. -### Add in your own logging with an HTTP middleware +#### Add in your own logging with an HTTP middleware You may want to add in your own logging around HTTP requests. One way to do this is to register a custom HTTP client with a logging transport built in. Here is @@ -92,16 +102,16 @@ func main() { } ``` -## gRPC based clients +### gRPC based clients -### Try setting grpc-go's debug variables +#### Try setting grpc-go's debug variables Try setting the following environment variables for grpc-go: `GRPC_GO_LOG_VERBOSITY_LEVEL=99` `GRPC_GO_LOG_SEVERITY_LEVEL=info`. These are good for diagnosing connection level failures. For more information please see [grpc-go's debug documentation](https://pkg.go.dev/google.golang.org/grpc/examples/features/debugging#section-readme). -### Add in your own logging with a gRPC interceptors +#### Add in your own logging with a gRPC interceptors You may want to add in your own logging around gRPC requests. One way to do this is to register a custom interceptor that adds logging. Here is @@ -160,3 +170,147 @@ func main() { // Use the client } ``` + +## Telemetry + +The Google Cloud client libraries for Go still use the OpenCensus project by +default. However, opt-in support for +[OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) is now +available. The transition from OpenCensus to OpenTelemetry is covered in the +following sections. + +### Tracing (experimental) + +All clients contain experimental support for the propagation of both OpenCensus +and OpenTelemetry trace context to their receiving endpoints. However, apart +from spans created during context propagation, the generated clients do not +create spans. Only the following hand-written clients create spans for select +operations: + +* [cloud.google.com/go/bigquery](https://pkg.go.dev/cloud.google.com/go/bigquery) +* [cloud.google.com/go/bigtable](https://pkg.go.dev/cloud.google.com/go/bigtable) +* [cloud.google.com/go/datastore](https://pkg.go.dev/cloud.google.com/go/datastore) +* [cloud.google.com/go/firestore](https://pkg.go.dev/cloud.google.com/go/firestore) +* [cloud.google.com/go/spanner](https://pkg.go.dev/cloud.google.com/go/spanner) +* [cloud.google.com/go/storage](https://pkg.go.dev/cloud.google.com/go/storage) + +Currently, the spans created by these clients are for OpenCensus. However, +OpenCensus users are urged to transition to OpenTelemetry as soon as possible, +as explained in the next section. OpenTelemetry users can opt-in to experimental +OpenTelemetry support via an environment variable, as described below. + +#### OpenCensus + +**Warning:The OpenCensus project is obsolete and was archived on July 31st, +2023.** This means that any security vulnerabilities that are found will not be +patched. We recommend that you begin migrating to OpenCensus tracing to +OpenTelemetry, the successor project. + +Using the [OpenTelemetry-Go - OpenCensus Bridge](https://github.com/open-telemetry/opentelemetry-go/tree/main/bridge/opencensus), you can immediately begin exporting your traces with OpenTelemetry, even while +dependencies of your application remain instrumented with OpenCensus. If you do +not use the bridge, you will need to migrate your entire application and all of +its instrumented dependencies at once. For simple applications, this may be +possible, but we expect the bridge to be helpful if multiple libraries with +instrumentation are used. + +On May 29, 2024, six months after the +[release](https://github.com/googleapis/google-cloud-go/releases/tag/v0.111.0) +of experimental, opt-in support for OpenTelemetry tracing, the default tracing +support in the clients above will change from OpenCensus to OpenTelemetry, and +the experimental OpenCensus support will be marked as deprecated. To continue +using the OpenCensus support after this change, set the environment variable +`GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING` to the case-insensitive +value `opencensus` before loading the client library. + +```sh +export GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING=opencensus +``` + +On December 2nd, 2024, one year after the release of OpenTelemetry support, the +experimental and deprecated support for OpenCensus tracing will be removed. + +Please refer to the following resources: + +* [Sunsetting OpenCensus](https://opentelemetry.io/blog/2023/sunsetting-opencensus/) +* [OpenTelemetry-Go - OpenCensus Bridge](https://github.com/open-telemetry/opentelemetry-go/tree/main/bridge/opencensus) + +#### OpenTelemetry + +To opt-in to experimental OpenTelemetry tracing currently available in the +clients listed above, set the environment variable +`GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING` to the case-insensitive +value `opentelemetry` before loading the client library. + +```sh +export GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING=opentelemetry +``` + +On May 29, 2024, the default tracing support will change from OpenCensus to +OpenTelemetry, and this environment variable will no longer be needed. + +Please refer to the following resources: + +* [What is OpenTelemetry?](https://opentelemetry.io/docs/what-is-opentelemetry/) +* [Cloud Trace - Go and OpenTelemetry](https://cloud.google.com/trace/docs/setup/go-ot) +* On GCE, [use Ops Agent and OpenTelemetry](https://cloud.google.com/trace/docs/otlp) + +##### Configuring context propagation + +In order to pass options to OpenTelemetry trace context propagation, follow the +appropriate example for the client's underlying transport. + +###### Passing options in HTTP-based clients + +```go +ctx := context.Background() +trans, err := htransport.NewTransport(ctx, + http.DefaultTransport, + option.WithScopes(storage.ScopeFullControl), +) +if err != nil { + log.Fatal(err) +} +// An example of passing options to the otelhttp.Transport. +otelOpts := otelhttp.WithFilter(func(r *http.Request) bool { + return r.URL.Path != "/ping" +}) +hc := &http.Client{ + Transport: otelhttp.NewTransport(trans, otelOpts), +} +client, err := storage.NewClient(ctx, option.WithHTTPClient(hc)) +``` + +Note that scopes must be set manually in this user-configured solution. + +###### Passing options in gRPC-based clients + +```go +projectID := "..." +ctx := context.Background() + +// An example of passing options to grpc.WithStatsHandler. +otelOpts := otelgrpc.WithMessageEvents(otelgrpc.ReceivedEvents) +dialOpts := grpc.WithStatsHandler(otelgrpc.NewClientHandler(otelOpts)) + +ctx := context.Background() +c, err := datastore.NewClient(ctx, projectID, option.WithGRPCDialOption(dialOpts)) +if err != nil { + log.Fatal(err) +} +defer c.Close() +``` + + +### Metrics (experimental) + +The generated clients do not create metrics. Only the following hand-written +clients create experimental OpenCensus metrics: + +* [cloud.google.com/go/bigquery](https://pkg.go.dev/cloud.google.com/go/bigquery) +* [cloud.google.com/go/pubsub](https://pkg.go.dev/cloud.google.com/go/pubsub) +* [cloud.google.com/go/spanner](https://pkg.go.dev/cloud.google.com/go/spanner) + +#### OpenTelemetry + +The transition of the experimental metrics in the clients above from OpenCensus +to OpenTelemetry is still TBD. \ No newline at end of file