Merge pull request #59 from grafana/instrument_with_pyroscope_sdk

Instrument with Pyroscope SDK

Dockerfile

@@ -12,7 +12,7 @@ ENV PUBLIC_BACKEND_WS_ENDPOINT=${PUBLIC_BACKEND_WS_ENDPOINT}
 RUN npm install && \
     npm run build
 
-FROM golang:1.20-bullseye as builder
+FROM golang:1.21-bullseye as builder
 
 WORKDIR /app
 COPY . ./

README.md

@@ -4,7 +4,7 @@
 
 ## What is QuickPizza? 🍕🍕🍕
 
-`QuickPizza` is a web application, used for demonstrations and workshops, that generates new and exciting pizza combinations! 
+`QuickPizza` is a web application, used for demonstrations and workshops, that generates new and exciting pizza combinations!
 
 The app is built using [SvelteKit](https://kit.svelte.dev/) for the frontend and [Go](https://go.dev/) for the backend.
 
@@ -83,7 +83,6 @@ k6 run -e BASE_URL=https://acmecorp.dev:3333 01.basic.js
 
 <details>
   <summary>Using k6 Docker image</summary>
-
   If you want to use the [k6 Docker image](https://hub.docker.com/r/grafana/k6) to run k6, you need to run the Quickpizza and k6 containers within the same network.
 
   First, create a Docker network. Then, run Quickpizza, assigning a hostname and connecting to the created network.
@@ -119,7 +118,7 @@ export QUICKPIZZA_LOG_LEVEL=debug
 
 The [docker-compose-local.yaml](./docker-compose-local.yaml) file is set up to run and orchestrate the QuickPizza, Grafana, Tempo, Loki, Prometheus, Pyroscope, and Grafana Agent containers.
 
-The Grafana Agent collects traces, metrics, logs and profiling data from the QuickPizza app, forwarding them to the Tempo, Prometheus, Loki and Pyroscope services. Finally, you can visualize and correlate data stored in these containers with the locally running Grafana instance. 
+The Grafana Agent collects traces, metrics, logs and profiling data from the QuickPizza app, forwarding them to the Tempo, Prometheus and Loki. Finally, you can visualize and correlate data stored in these containers with the locally running Grafana instance.
 
 To start the local environment, use the following command:
 
@@ -131,9 +130,19 @@ Like before, QuickPizza is available at [localhost:3333](http://localhost:3333).
 
 Then, you can visit the Grafana instance running at [localhost:3000](http://localhost:3000) to access QuickPizza data.
 
+Please refer to [agent-local.river](./contrib/agent-local.river) and [docker-compose-local.yaml](./docker-compose-local.yaml) to find the labels applied to the telemetry data.
+
+**Correlate Pyroscope data with k6 tests**
+
+Whenever there is a Pyroscope instance endpoint provided via `QUICKPIZZA_PYROSCOPE_ENDPOINT` environment variable, the QuickPizza app will emit profiling data to Pyroscope. You can visualize the profiling data with the Pyroscope data source in Grafana.
+
+To correlate the profiling data with the k6 test results, use the [k6 Pyroscope library](https://grafana.com/docs/k6/next/javascript-api/jslib/http-instrumentation-pyroscope/).
+
 ![Pyroscope Data Source](./docs/images/local-env-grafana-with-pyroscope.png)
 
-Please refer to [agent-local.river](./contrib/agent-local.river) and [docker-compose-local.yaml](./docker-compose-local.yaml) to find the labels applied to the telemetry data.
+Additional variables are available to configure the Pyroscope data source:
+- `QUICKPIZZA_PYROSCOPE_NAME` - the name of the service in Pyroscope (uses `quickpizza` by default)
+- `QUICKPIZZA_GRAFANA_CLOUD_USER` and `QUICKPIZZA_GRAFANA_CLOUD_PASSWORD` are the Basic auth credentials to authenticate with the Grafana Cloud instance.
 
 **Send k6 results to Prometheus and visualize them with prebuilt dashboards**
 
@@ -152,12 +161,12 @@ For detailed instructions about the different options of the k6 Prometheus outpu
 
 ### Grafana Cloud
 
-The [docker-compose-cloud.yaml](./docker-compose-cloud.yaml) file is set up to run the QuickPizza and Grafana Agent containers. 
+The [docker-compose-cloud.yaml](./docker-compose-cloud.yaml) file is set up to run the QuickPizza and Grafana Agent containers.
 
 In this setup, the Grafana Agent collects observability data from the QuickPizza app and forwards it to [Grafana Cloud](https://grafana.com/products/cloud/).
 
 You will need the following settings:
-1. The name of the [Grafana Cloud Stack](https://grafana.com/docs/grafana-cloud/account-management/cloud-portal/#your-grafana-cloud-stack) where the telemetry data will be stored. 
+1. The name of the [Grafana Cloud Stack](https://grafana.com/docs/grafana-cloud/account-management/cloud-portal/#your-grafana-cloud-stack) where the telemetry data will be stored.
 2. An [Access Policy Token](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/) that includes the following scopes for the selected Grafana Cloud Stack: `stacks:read`, `metrics:write`, `logs:write`, `traces:write`, and `profiles:write`.
 
 Then, create an `.env` file with the following environment variables and the values of the previous settings:
@@ -177,6 +186,16 @@ QuickPizza is available at [localhost:3333](http://localhost:3333). Click the `P
 
 Now, you can log in to [Grafana Cloud](https://grafana.com/products/cloud/) and explore QuickPizza's telemetry data on the Prometheus, Tempo, Loki, and Pyroscope instances of your Grafana Cloud Stack. Refer to [agent-cloud.river](./contrib/agent-cloud.river) and [docker-compose-cloud.yaml](./docker-compose-cloud.yaml) to find the labels applied to the telemetry data.
 
+***Enable profiling (Grafana Pyroscope)***
+
+To enable [Grafana Cloud Profiling](https://grafana.com/docs/grafana-cloud/monitor-applications/profiles/), add the `QUICKPIZZA_CONF_PYROSCOPE_URL` variable to the `.env` file, setting its value to your Pyroscope web URL:
+
+```bash
+QUICKPIZZA_CONF_PYROSCOPE_URL=
+```
+
+Restart the `docker-compose-cloud.yaml` environment.
+
 ***Enable Frontend observability (Grafana Faro)***
 
 Frontend observability is available exclusively in Grafana Cloud. To enable [Grafana Cloud Frontend Observability](https://grafana.com/docs/grafana-cloud/monitor-applications/frontend-observability/) for QuickPizza, add the `QUICKPIZZA_CONF_FARO_URL` variable to the `.env` file, setting its value to your Faro web URL:
@@ -191,7 +210,7 @@ Restart the `docker-compose-cloud.yaml` environment.
 
 **Send k6 results to Grafana Cloud Prometheus and visualize them with prebuilt dashboards**
 
-Just like in the local setup, we can output k6 result metrics to a Prometheus instance; in this case, it is provided by our Grafana Cloud Stack. 
+Just like in the local setup, we can output k6 result metrics to a Prometheus instance; in this case, it is provided by our Grafana Cloud Stack.
 
 ```bash
 K6_PROMETHEUS_RW_USERNAME=USERNAME \
@@ -204,11 +223,11 @@ For detailed instructions, refer to the [k6 output guide for Grafana Cloud Prome
 
 ## Deploy QuickPizza Docker image
 
-The [Dockerfile](./Dockerfile) contains the setup for running QuickPizza without collecting data with the Grafana agent. 
+The [Dockerfile](./Dockerfile) contains the setup for running QuickPizza without collecting data with the Grafana agent.
 
 You can use the Dockerfile or build a Docker image to deploy the QuickPizza app on any cloud provider that supports Docker deployments. For simplicity, here are the `Fly.io` instructions:
 
-1. [Authenticate using the fly CLI](https://fly.io/docs/speedrun/). 
+1. [Authenticate using the fly CLI](https://fly.io/docs/speedrun/).
 2. Then, run the CLI to deploy the application and set up the internal port `3333` that the server listens to.
     ```bash
     fly launch --internal-port 3333 --now
@@ -234,7 +253,7 @@ export QUICKPIZZA_DB="quickpizza.db"
 
 ## Deploy application to Kubernetes
 
-If you want to run a test that uses [xk6-disruptor](https://grafana.com/docs/k6/latest/testing-guides/injecting-faults-with-xk6-disruptor/first-steps/), or want to experiment with distributed tracing, you will need to deploy QuickPizza to Kubernetes. 
+If you want to run a test that uses [xk6-disruptor](https://grafana.com/docs/k6/latest/testing-guides/injecting-faults-with-xk6-disruptor/first-steps/), or want to experiment with distributed tracing, you will need to deploy QuickPizza to Kubernetes.
 
 For a detailed setup instructions, see the [QuickPizza Kubernetes guide](./docs/kubernetes-setup.md).
 

cmd/main.go

@@ -4,10 +4,12 @@ import (
 	"context"
 	http "net/http"
 	"os"
+	"runtime"
 	"strconv"
 	"strings"
 	"time"
 
+	"github.com/grafana/pyroscope-go"
 	"github.com/grafana/quickpizza/pkg/database"
 	qphttp "github.com/grafana/quickpizza/pkg/http"
 	"github.com/hashicorp/go-retryablehttp"
@@ -48,8 +50,18 @@ func main() {
 
 		server = server.WithTraceInstaller(installer)
 	}
-	// Always add profiling middleware.
-	server = server.WithProfiling()
+
+	// Add profiling middleware if Pyroscope is enabled
+	if cfg, ok := envPyroscopeConfig(); ok {
+		slog.Debug("enabling Pyroscope profiling")
+
+		runtime.SetMutexProfileFraction(5)
+		runtime.SetBlockProfileRate(5)
+
+		pyroscope.Start(cfg)
+
+		server = server.WithProfiling()
+	}
 
 	// Always add prometheus middleware.
 	server = server.WithPrometheus()
@@ -163,6 +175,43 @@ func clientFromEnv() *http.Client {
 	return retriableClient.StandardClient()
 }
 
+func envPyroscopeConfig() (pyroscope.Config, bool) {
+	pyroscopeAddr, ok := os.LookupEnv("QUICKPIZZA_PYROSCOPE_ENDPOINT")
+	if !ok {
+		return pyroscope.Config{}, false
+	}
+
+	svcName, ok := os.LookupEnv("QUICKPIZZA_PYROSCOPE_NAME")
+	if !ok {
+		svcName = "quickpizza"
+	}
+
+	return pyroscope.Config{
+		ApplicationName: svcName,
+		ServerAddress:   pyroscopeAddr,
+
+		BasicAuthUser:     os.Getenv("QUICKPIZZA_GRAFANA_CLOUD_USER"),
+		BasicAuthPassword: os.Getenv("QUICKPIZZA_GRAFANA_CLOUD_PASSWORD"),
+
+		// make configurable?
+		ProfileTypes: []pyroscope.ProfileType{
+			// these profile types are enabled by default:
+			pyroscope.ProfileCPU,
+			pyroscope.ProfileAllocObjects,
+			pyroscope.ProfileAllocSpace,
+			pyroscope.ProfileInuseObjects,
+			pyroscope.ProfileInuseSpace,
+
+			// these profile types are optional:
+			pyroscope.ProfileGoroutines,
+			pyroscope.ProfileMutexCount,
+			pyroscope.ProfileMutexDuration,
+			pyroscope.ProfileBlockCount,
+			pyroscope.ProfileBlockDuration,
+		},
+	}, true
+}
+
 func envServeAll() bool {
 	allSvcs, present := os.LookupEnv("QUICKPIZZA_ALL_SERVICES")
 	allSvcsB, _ := strconv.ParseBool(allSvcs)

contrib/agent-cloud.river

@@ -37,19 +37,6 @@ prometheus.scrape "default" {
   ]
 }
 
-pyroscope.scrape "scrape_profiles" {
-  targets = [
-    {
-      "__address__" = env("QUICKPIZZA_HOST"),
-      "service_name" = "quickpizza",
-      "instance" = "local",
-    },
-  ]
-  forward_to = [
-    module.git.grafana_cloud.exports.profiles_receiver,
-  ]
-}
-
 discovery.docker "all_containers" {
   host = "unix:///var/run/docker.sock"
 }

contrib/agent-local.river

@@ -58,26 +58,6 @@ prometheus.remote_write "default" {
   }
 }
 
-pyroscope.scrape "scrape_profiles" {
-  targets = [
-    {
-      "__address__" = env("QUICKPIZZA_HOST"),
-      "service_name" = "quickpizza",
-      "instance" = "local",
-    },
-  ]
-  forward_to = [
-    pyroscope.write.default.receiver,
-  ]
-}
-
-pyroscope.write "default" {
-  endpoint {
-    // TODO: Replace this with your pyroscpoe instance endpoint
-    url = env("PROFILES_ENDPOINT")
-  }
-}
-
 discovery.docker "all_containers" {
   host = "unix:///var/run/docker.sock"
 }

docker-compose-cloud.yaml

@@ -1,4 +1,4 @@
-version: '3.8'
+version: "3.8"
 
 services:
   quickpizza:
@@ -12,13 +12,15 @@ services:
       QUICKPIZZA_TRUST_CLIENT_TRACEID: 1
       # must be set with an .env file
       QUICKPIZZA_CONF_FARO_URL: "${QUICKPIZZA_CONF_FARO_URL}"
+      QUICKPIZZA_PYROSCOPE_ENDPOINT: "${QUICKPIZZA_CONF_PYROSCOPE_ENDPOINT}"
 
   agent:
     image: grafana/agent:latest
     volumes:
       - "./contrib/agent-cloud.river:/grafana-agent.river:Z"
       - "${DOCKER_SOCKET:-/var/run/docker.sock}:/var/run/docker.sock"
-    command: ["run", "/grafana-agent.river", "--server.http.listen-addr=0.0.0.0:12345"]
+    command:
+      ["run", "/grafana-agent.river", "--server.http.listen-addr=0.0.0.0:12345"]
     ports:
       - "12345:12345"
     environment:

docker-compose-local.yaml

@@ -1,5 +1,4 @@
-
-version: '3.8'
+version: "3.8"
 
 services:
   quickpizza:
@@ -11,6 +10,7 @@ services:
     environment:
       QUICKPIZZA_OTLP_ENDPOINT: http://agent:4318
       QUICKPIZZA_TRUST_CLIENT_TRACEID: 1
+      QUICKPIZZA_PYROSCOPE_ENDPOINT: http://pyroscope:4040
 
   loki:
     image: grafana/loki:2.8.3
@@ -34,10 +34,10 @@ services:
   tempo:
     image: grafana/tempo:2.1.0
     command:
-      - "-storage.trace.backend=local"                  # tell tempo where to permanently put traces
+      - "-storage.trace.backend=local" # tell tempo where to permanently put traces
       - "-storage.trace.local.path=/tmp/tempo/traces"
-      - "-storage.trace.wal.path=/tmp/tempo/wal"        # tell tempo where to store the wal
-      - "-auth.enabled=false"                           # disables the requirement for the X-Scope-OrgID header
+      - "-storage.trace.wal.path=/tmp/tempo/wal" # tell tempo where to store the wal
+      - "-auth.enabled=false" # disables the requirement for the X-Scope-OrgID header
       - "-server.http-listen-port=3200"
     ports:
       - "3200:3200"
@@ -60,7 +60,8 @@ services:
     volumes:
       - "./contrib/agent-local.river:/grafana-agent.river:Z"
       - "${DOCKER_SOCKET:-/var/run/docker.sock}:/var/run/docker.sock"
-    command: ["run", "/grafana-agent.river", "--server.http.listen-addr=0.0.0.0:12345"]
+    command:
+      ["run", "/grafana-agent.river", "--server.http.listen-addr=0.0.0.0:12345"]
     ports:
       - "12345:12345"
     environment:
@@ -69,7 +70,6 @@ services:
       METRICS_ENDPOINT: http://prometheus:9090/api/v1/write
       TRACES_ENDPOINT: http://tempo:4317
       LOGS_ENDPOINT: http://loki:3100/loki/api/v1/push
-      PROFILES_ENDPOINT: http://pyroscope:4040
     depends_on:
       - prometheus
       - quickpizza