Skip to content

Commit

Permalink
Use OTLP push mode in Prometheus tutorial (#4874)
Browse files Browse the repository at this point in the history 
Co-authored-by: Vishwesh Bankwar <[email protected]>
  • Loading branch information
@reyang @vishweshbankwar
reyang and vishweshbankwar authored Sep 22, 2023
1 parent 535c819 commit f468dd8
Show file tree
Hide file tree
Showing 4 changed files with 107 additions and 84 deletions.
7 changes: 6 additions & 1 deletion docs/metrics/getting-started-prometheus-grafana/Program.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,12 @@ public static void Main()
{
using var meterProvider = Sdk.CreateMeterProviderBuilder()
.AddMeter("MyCompany.MyProduct.MyLibrary")
.AddPrometheusHttpListener()
.AddConsoleExporter()
.AddOtlpExporter(options =>
{
options.Endpoint = new Uri("http://localhost:9090/api/v1/otlp/v1/metrics");
options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.HttpProtobuf;
})
.Build();

Console.WriteLine("Press any key to exit");
Expand Down
179 changes: 98 additions & 81 deletions docs/metrics/getting-started-prometheus-grafana/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Getting Started with Prometheus and Grafana

- [Export metrics from the application](#export-metrics-from-the-application)
- [Check results in the browser](#check-results-in-the-browser)
- [Check results in the console](#check-results-in-the-console)
- [Collect metrics using Prometheus](#collect-metrics-using-prometheus)
- [Configuration](#configuration)
- [Start Prometheus](#start-prometheus)
- [Install and run Prometheus](#install-and-run-prometheus)
- [View results in Prometheus](#view-results-in-prometheus)
- [Explore metrics using Grafana](#explore-metrics-using-grafana)
- [Final cleanup](#final-cleanup)
- [Learn more](#learn-more)

## Export metrics from the application
Expand All @@ -23,46 +23,73 @@ cd getting-started-prometheus
dotnet run
```

Add a reference to [Prometheus
Exporter Http Listener](../../../src/OpenTelemetry.Exporter.Prometheus.HttpListener/README.md):
Add reference to [Console
Exporter](../../../src/OpenTelemetry.Exporter.Console/README.md) and [OTLP
Exporter](../../../src/OpenTelemetry.Exporter.OpenTelemetryProtocol/README.md):

```sh
dotnet add package OpenTelemetry.Exporter.Prometheus.HttpListener --prerelease
dotnet add package OpenTelemetry.Exporter.Console
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
```

Now, we are going to make some small tweaks to the example in the
getting-started metrics `Program.cs` to make the metrics available via
OpenTelemetry Prometheus Exporter.
Now copy the code from [Program.cs](./Program.cs).

First, copy and paste everything from getting-started metrics
[example](../getting-started-console/Program.cs) to the Program.cs file of the
new console application (getting-started-prometheus) we've created.
### Check results in the console

And replace the below line:
Run the application again and we should see the metrics output from the console:

```csharp
.AddConsoleExporter()
```text
> dotnet run
Press any key to exit
Resource associated with Metric:
telemetry.sdk.name: opentelemetry
telemetry.sdk.language: dotnet
telemetry.sdk.version: 1.6.1-alpha.0.23
service.name: unknown_service:getting-started-prometheus-grafana
Export MyFruitCounter, Meter: MyCompany.MyProduct.MyLibrary/1.0
(2023-09-22T20:40:22.2586791Z, 2023-09-22T20:40:31.1582923Z] color: red name: apple LongSum
Value: 54
(2023-09-22T20:40:22.2586791Z, 2023-09-22T20:40:31.1582923Z] color: yellow name: lemon LongSum
Value: 63
(2023-09-22T20:40:22.2586791Z, 2023-09-22T20:40:31.1582923Z] color: green name: apple LongSum
Value: 18
...
```

with
Note that we have configured two exporters in the code:

```csharp
.AddPrometheusHttpListener()
using var meterProvider = Sdk.CreateMeterProviderBuilder()
...
.AddConsoleExporter()
.AddOtlpExporter(options =>
{
options.Endpoint = new Uri("http://localhost:9090/api/v1/otlp/v1/metrics");
options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.HttpProtobuf;
})
.Build();
```

`PrometheusHttpListener` is a wrapper that contains `PrometheusExporter`. With
`AddPrometheusHttpListener()`, OpenTelemetry `PrometheusExporter` will export
data via the endpoint defined by
[PrometheusHttpListenerOptions.UriPrefixes](../../../src/OpenTelemetry.Exporter.Prometheus.HttpListener/README.md#uriprefixes),
which is `http://localhost:9464/` by default.
When we ran the application, the `ConsoleExporter` was printing the metrics on
console, and the `OtlpExporter` was attempting to send the metrics to
`http://localhost:9090/api/v1/otlp/v1/metrics`.

Since we didn't have Prometheus server running, the metrics received by
`OtlpExporter` were simply dropped on the floor. In the next step, we are going
to learn about how to use Prometheus to collect and visualize the metrics.

```mermaid
graph LR
subgraph SDK
MeterProvider
MetricReader[BaseExportingMetricReader]
PrometheusHttpListener["PrometheusHttpListener<br/>(http://localhost:9464/)"]
MetricReader2[BaseExportingMetricReader]
ConsoleExporter
OtlpExporter
end
subgraph API
Expand All @@ -71,7 +98,9 @@ end
Instrument --> | Measurements | MeterProvider
MeterProvider --> | Metrics | MetricReader --> | Pull | PrometheusHttpListener
MeterProvider --> | Metrics | MetricReader --> | Push | OtlpExporter
MeterProvider --> | Metrics | MetricReader2 --> | Push | ConsoleExporter
```

Also, for our learning purpose, use a while-loop to keep increasing the counter
Expand All @@ -91,74 +120,37 @@ while (!Console.KeyAvailable)
}
```

After the above modifications, now our `Program.cs` should look like [this](./Program.cs).

### Check results in the browser

Start the application and keep it running. Now we should be able to see the
metrics at [http://localhost:9464/metrics](http://localhost:9464/metrics) from a
web browser:

![Browser UI](https://user-images.githubusercontent.com/17327289/151633547-736c6d91-62d2-4e66-a53f-2e16c44bfabc.png)

Now, we understand how we can configure `PrometheusHttpListener` to export metrics.
Next, we are going to learn about how to use Prometheus to collect the metrics.

## Collect metrics using Prometheus

### Install and run Prometheus

Follow the [first steps](https://prometheus.io/docs/introduction/first_steps/)
to download the [latest release](https://prometheus.io/download/) of Prometheus.

### Configuration

After finished downloading, extract it to a local location that's easy to
access. We will find the default Prometheus configuration YAML file in the
folder, named `prometheus.yml`.

Let's create a new file in the same location as where `prometheus.yml` locates,
and named the new file as `otel.yml` for this exercise. Then, copy and paste the
entire content below into the `otel.yml` file we have created just now.

```yaml
global:
scrape_interval: 10s
evaluation_interval: 10s
scrape_configs:
- job_name: "otel"
static_configs:
- targets: ["localhost:9464"]
```
access. Run the `prometheus(.exe)` server executable with feature flag
[otlp-receiver](https://prometheus.io/docs/prometheus/latest/feature_flags/#otlp-receiver)
enabled:

### Start Prometheus
Follow the instructions from
[starting-prometheus](https://prometheus.io/docs/introduction/first_steps/#starting-prometheus)
to start the Prometheus server and verify it has been started successfully.
Please note that we will need pass in `otel.yml` file as the argument:

```console
./prometheus --config.file=otel.yml
```sh
./prometheus --enable-feature=otlp-write-receiver
```

### View results in Prometheus

To use the graphical interface for viewing our metrics with Prometheus, navigate
to [http://localhost:9090/graph](http://localhost:9090/graph), and type
`MyFruitCounter` in the expression bar of the UI; finally, click the execute
button.
`MyFruitCounter_total` in the expression bar of the UI; finally, click the
execute button.

We should be able to see the following chart from the browser:

![Prometheus UI](https://user-images.githubusercontent.com/17327289/151636225-6e4ce4c7-09f3-4996-8ca5-d404a88d9195.png)

From the legend, we can see that the `instance` name and the `job` name are the
values we have set in `otel.yml`.

Congratulations!

Now we know how to configure Prometheus server and deploy OpenTelemetry
`PrometheusHttpListener` to export our metrics. Next, we are going to explore a tool
`OtlpExporter` to export our metrics. Next, we are going to explore a tool
called Grafana, which has powerful visualizations for the metrics.

## Explore metrics using Grafana
Expand Down Expand Up @@ -191,24 +183,49 @@ Feel free to find some handy PromQL
[here](https://promlabs.com/promql-cheat-sheet/).

In the below example, the query targets to find out what is the per-second rate
of increase of myFruitCounter over the past 5 minutes:
of increase of `MyFruitCounter_total` over the past 5 minutes:

![Grafana
UI](https://user-images.githubusercontent.com/17327289/151636769-138ecb4f-b44f-477b-88eb-247fc4340252.png)

## Final cleanup

In the end, remove the Console Exporter so we only have OTLP Exporter in the
final application:

```csharp
using var meterProvider = Sdk.CreateMeterProviderBuilder()
...
// Remove Console Exporter from the final application
// .AddConsoleExporter()
.AddOtlpExporter(options =>
{
options.Endpoint = new Uri("http://localhost:9090/api/v1/otlp/v1/metrics");
options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.HttpProtobuf;
})
.Build();
```

```sh
dotnet remove package OpenTelemetry.Exporter.Console
```

```mermaid
graph TD
subgraph Prometheus
PrometheusScraper
PrometheusDatabase
graph LR
subgraph SDK
MeterProvider
MetricReader[BaseExportingMetricReader]
OtlpExporter
end
subgraph API
Instrument["Meter(#quot;MyCompany.MyProduct.MyLibrary#quot;, #quot;1.0#quot;)<br/>Counter(#quot;MyFruitCounter#quot;)"]
end
PrometheusHttpListener["PrometheusHttpListener<br/>(listening at #quot;http://localhost:9464/#quot;)"] -->|HTTP GET| PrometheusScraper{{"Prometheus scraper<br/>(polling #quot;http://localhost:9464/metrics#quot; every 10 seconds)"}}
PrometheusScraper --> PrometheusDatabase[("Prometheus TSDB (time series database)")]
PrometheusDatabase -->|http://localhost:9090/graph| PrometheusUI["Browser<br/>(Prometheus Dashboard)"]
PrometheusDatabase -->|http://localhost:9090/api/| Grafana[Grafana Server]
Grafana -->|http://localhost:3000/dashboard| GrafanaUI["Browser<br/>(Grafana Dashboard)"]
Instrument --> | Measurements | MeterProvider
MeterProvider --> | Metrics | MetricReader --> | Push | OtlpExporter
```

## Learn more
Expand Down
3 changes: 2 additions & 1 deletion docs/metrics/getting-started-prometheus-grafana/getting-started-prometheus-grafana.csproj
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
<Project Sdk="Microsoft.NET.Sdk">
<ItemGroup>
<ProjectReference Include="$(RepoRoot)\src\OpenTelemetry.Exporter.Prometheus.HttpListener\OpenTelemetry.Exporter.Prometheus.HttpListener.csproj" />
<ProjectReference Include="$(RepoRoot)\src\OpenTelemetry.Exporter.Console\OpenTelemetry.Exporter.Console.csproj" />
<ProjectReference Include="$(RepoRoot)\src\OpenTelemetry.Exporter.OpenTelemetryProtocol\OpenTelemetry.Exporter.OpenTelemetryProtocol.csproj" />
</ItemGroup>
</Project>
2 changes: 1 addition & 1 deletion docs/trace/getting-started-jaeger/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ using var tracerProvider = Sdk.CreateTracerProviderBuilder()
.Build();
```

When we run the application, the `ConsoleExporter` was printing the traces on
When we ran the application, the `ConsoleExporter` was printing the traces on
console, and the `OtlpExporter` was attempting to send the traces to Jaeger
Agent via the default endpoint `http://localhost:4317`.

Expand Down

0 comments on commit f468dd8

Please sign in to comment.