docs(main): add telemetry discussion to debug.md

refs: googleapis/google-api-go-client#2127
refs: googleapis#8655

debug.md

@@ -1,21 +1,31 @@
-# Debugging tips and tricks
+# Logging, debugging and telemetry
+
+Logging, debugging and telemetry all capture data that can be used for
+troubleshooting. Logging records specific events and transactions. Debugging
+exposes values for immediate analysis. Telemetry is suitable for production use
+and can serve both logging and monitoring purposes. Telemetry tracing follows
+requests through a system to provide a view of component interactions. Telemetry
+metrics collects data for significant performance indicators, offering insights
+into a system's health.
+
+## Logging and debugging
 
 While working with the Go Client libraries you may run into some situations
 where you need a deeper level of understanding about what is going on in order
 to solve your problem. Here are some tips and tricks that you can use in these
-cases. *Note* that many of the tips in this document will have a performance
+cases. *Note* that many of the tips in this section will have a performance
 impact and are therefore not recommended for sustained production use. Use these
 tips locally or in production for a *limited time* to help get a better
 understanding of what is going on.
 
-## HTTP based clients
+### HTTP based clients
 
 All of our auto-generated clients have a constructor to create a client that
 uses HTTP/JSON instead of gRPC. Additionally a couple of our hand-written
 clients like Storage and Bigquery are also HTTP based. Here are some tips for
 debugging these clients.
 
-### Try setting Go's HTTP debug variable
+#### Try setting Go's HTTP debug variable
 
 Try setting the following environment variable for verbose Go HTTP logging:
 GODEBUG=http2debug=1. To read more about this feature please see the godoc for
@@ -24,7 +34,7 @@ GODEBUG=http2debug=1. To read more about this feature please see the godoc for
 *WARNING*: Enabling this debug variable will log headers and payloads which may
 contain private information.
 
-### Add in your own logging with an HTTP middleware
+#### Add in your own logging with an HTTP middleware
 
 You may want to add in your own logging around HTTP requests. One way to do this
 is to register a custom HTTP client with a logging transport built in. Here is
@@ -92,16 +102,16 @@ func main() {
 }
 ```
 
-## gRPC based clients
+### gRPC based clients
 
-### Try setting grpc-go's debug variables
+#### Try setting grpc-go's debug variables
 
 Try setting the following environment variables for grpc-go:
 `GRPC_GO_LOG_VERBOSITY_LEVEL=99` `GRPC_GO_LOG_SEVERITY_LEVEL=info`. These are
 good for diagnosing connection level failures. For more information please see
 [grpc-go's debug documentation](https://pkg.go.dev/google.golang.org/grpc/examples/features/debugging#section-readme).
 
-### Add in your own logging with a gRPC interceptors
+#### Add in your own logging with a gRPC interceptors
 
 You may want to add in your own logging around gRPC requests. One way to do this
 is to register a custom interceptor that adds logging. Here is
@@ -160,3 +170,147 @@ func main() {
     // Use the client
 }
 ```
+
+## Telemetry
+
+The Google Cloud client libraries for Go still use the OpenCensus project by
+default. However, opt-in support for
+[OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) is now
+available. The transition from OpenCensus to OpenTelemetry is covered in the
+following sections.
+
+### Tracing (experimental)
+
+All clients contain experimental support for the propagation of both OpenCensus
+and OpenTelemetry trace context to their receiving endpoints. However, apart
+from spans created during context propagation, the generated clients do not
+create spans. Only the following hand-written clients create spans for select
+operations:
+
+* [cloud.google.com/go/bigquery](https://pkg.go.dev/cloud.google.com/go/bigquery)
+* [cloud.google.com/go/bigtable](https://pkg.go.dev/cloud.google.com/go/bigtable)
+* [cloud.google.com/go/datastore](https://pkg.go.dev/cloud.google.com/go/datastore)
+* [cloud.google.com/go/firestore](https://pkg.go.dev/cloud.google.com/go/firestore)
+* [cloud.google.com/go/spanner](https://pkg.go.dev/cloud.google.com/go/spanner)
+* [cloud.google.com/go/storage](https://pkg.go.dev/cloud.google.com/go/storage)
+
+Currently, the spans created by these clients are for OpenCensus. However,
+OpenCensus users are urged to transition to OpenTelemetry as soon as possible,
+as explained in the next section. OpenTelemetry users can opt-in to experimental
+OpenTelemetry support via an environment variable, as described below.
+
+#### OpenCensus
+
+**Warning:The OpenCensus project is obsolete and was archived on July 31st,
+2023.** This means that any security vulnerabilities that are found will not be
+patched. We recommend that you begin migrating to OpenCensus tracing to
+OpenTelemetry, the successor project.
+
+Using the [OpenTelemetry-Go - OpenCensus Bridge](https://github.com/open-telemetry/opentelemetry-go/tree/main/bridge/opencensus), you can immediately begin exporting your traces with OpenTelemetry, even while
+dependencies of your application remain instrumented with OpenCensus. If you do
+not use the bridge, you will need to migrate your entire application and all of
+its instrumented dependencies at once.  For simple applications, this may be
+possible, but we expect the bridge to be helpful if multiple libraries with
+instrumentation are used.
+
+On May 29, 2024, six months after the
+[release](https://github.com/googleapis/google-cloud-go/releases/tag/v0.111.0)
+of experimental, opt-in support for OpenTelemetry tracing, the default tracing
+support in the clients above will change from OpenCensus to OpenTelemetry, and
+the experimental OpenCensus support will be marked as deprecated. To continue
+using the OpenCensus support after this change, set the environment variable
+`GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING` to the case-insensitive
+value `opencensus` before loading the client library.
+
+```sh
+export GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING=opencensus
+```
+
+On December 2nd, 2024, one year after the release of OpenTelemetry support, the
+experimental and deprecated support for OpenCensus tracing will be removed.
+
+Please refer to the following resources:
+
+* [Sunsetting OpenCensus](https://opentelemetry.io/blog/2023/sunsetting-opencensus/)
+* [OpenTelemetry-Go - OpenCensus Bridge](https://github.com/open-telemetry/opentelemetry-go/tree/main/bridge/opencensus)
+
+#### OpenTelemetry
+
+To opt-in to experimental OpenTelemetry tracing currently available in the
+clients listed above, set the environment variable
+`GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING` to the case-insensitive
+value `opentelemetry` before loading the client library.
+
+```sh
+export GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING=opentelemetry
+```
+
+On May 29, 2024, the default tracing support will change from OpenCensus to
+OpenTelemetry, and this environment variable will no longer be needed.
+
+Please refer to the following resources:
+
+* [What is OpenTelemetry?](https://opentelemetry.io/docs/what-is-opentelemetry/)
+* [Cloud Trace - Go and OpenTelemetry](https://cloud.google.com/trace/docs/setup/go-ot)
+* On GCE, [use Ops Agent and OpenTelemetry](https://cloud.google.com/trace/docs/otlp)
+
+##### Configuring context propagation
+
+In order to pass options to OpenTelemetry trace context propagation, follow the
+appropriate example for the client's underlying transport.
+
+###### Passing options in HTTP-based clients
+
+```go
+ctx := context.Background()
+trans, err := htransport.NewTransport(ctx,
+        http.DefaultTransport,
+        option.WithScopes(storage.ScopeFullControl),
+)
+if err != nil {
+        log.Fatal(err)
+}
+// An example of passing options to the otelhttp.Transport.
+otelOpts := otelhttp.WithFilter(func(r *http.Request) bool {
+	return r.URL.Path != "/ping"
+})
+hc := &http.Client{
+	Transport: otelhttp.NewTransport(trans, otelOpts),
+}
+client, err := storage.NewClient(ctx, option.WithHTTPClient(hc))
+```
+
+Note that scopes must be set manually in this user-configured solution.
+
+######  Passing options in gRPC-based clients
+
+```go
+projectID := "..."
+ctx := context.Background()
+
+// An example of passing options to grpc.WithStatsHandler.
+otelOpts := otelgrpc.WithMessageEvents(otelgrpc.ReceivedEvents)
+dialOpts := grpc.WithStatsHandler(otelgrpc.NewClientHandler(otelOpts))
+
+ctx := context.Background()
+c, err := datastore.NewClient(ctx, projectID, option.WithGRPCDialOption(dialOpts))
+if err != nil {
+	log.Fatal(err)
+}
+defer c.Close()
+```
+
+
+### Metrics (experimental)
+
+The generated clients do not create metrics. Only the following hand-written
+clients create experimental OpenCensus metrics:
+
+* [cloud.google.com/go/bigquery](https://pkg.go.dev/cloud.google.com/go/bigquery)
+* [cloud.google.com/go/pubsub](https://pkg.go.dev/cloud.google.com/go/pubsub)
+* [cloud.google.com/go/spanner](https://pkg.go.dev/cloud.google.com/go/spanner)
+
+#### OpenTelemetry
+
+The transition of the experimental metrics in the clients above from OpenCensus
+to OpenTelemetry is still TBD.