Skip to content

Commit

Permalink
Update 1.0 host config and observability pages
Browse files Browse the repository at this point in the history
Signed-off-by: Eric Gregory <[email protected]>
  • Loading branch information
ericgregory committed Mar 19, 2024
1 parent 6983df9 commit 4d7a0b3
Show file tree
Hide file tree
Showing 5 changed files with 141 additions and 61 deletions.
22 changes: 13 additions & 9 deletions docs/deployment/observability/logs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,23 @@ sidebar_position: 2

In addition to host-level logging, wasmCloud supports log exports using the [OpenTelemetry (OTEL)](https://opentelemetry.io/) specification.

For more information on wasmCloud's implementation of OpenTelemetry, including advanced configuration options, see [Observability with OpenTelemetry](/docs/deployment/observability/observability-with-opentelemetry.mdx).
For more information on wasmCloud's implementation of OpenTelemetry, including advanced configuration options, see [Observability with OpenTelemetry](/docs/1.0/deployment/observability/observability-with-opentelemetry).

### Configuration

By default, wasmCloud hosts write `info` level logs to `stderr` in plaintext.

Enable the wasmCloud host to emit logs as an OTEL signal with the `OTEL_TRACES_EXPORTER` environment variable:
To enable all OTEL signals, including logs, use the `WASMCLOUD_OBSERVABILITY_ENABLED` environment variable:

```bash
OTEL_TRACES_EXPORTER=otlp
WASMCLOUD_OBSERVABILITY_ENABLED="true"
```

In v0.82, this enables all supported OTEL signals&mdash;not just traces. (A more accurately descriptive environment variable and command line argument is to come in v1.0.)
To enable only logs, use the `WASMCLOUD_LOGS_ENABLED` environment variable:

```bash
WASMCLOUD_LOGS_ENABLED="true"
```

Enabling OTEL signals automatically structures logs for OTEL export. (In this case, local `stderr` logs remain plaintext.)

Expand All @@ -30,7 +34,7 @@ The endpoint for exports may be overwritten with the `OTEL_EXPORTER_OTLP_ENDPOIN
OTEL_EXPORTER_OTLP_ENDPOINT=http://my-custom-host:4318
```

These configurations can also be set with the `--otel-traces-exporter` and `--otel-exporter-otlp-endpoint` [host config](/docs/reference/host-config) options.
See the [host config](/docs/1.0/reference/host-config) options for equivalent host configuration CLI flags.

### Log levels

Expand All @@ -41,7 +45,7 @@ The wasmCloud host supports the following log levels:
- `debug`
- `trace`

To override the log level, use the `--log-level` [host config](/docs/reference/host-config) argument.
To override the log level, use the `--log-level` [host config](/docs/1.0/reference/host-config) argument.

:::warning

Expand All @@ -53,14 +57,14 @@ Take care when enabling `trace` or `debug` level logs in a production environmen

In a production environment, it's more common to emit logs in a structured format and aggregate logs from all hosts in a centralized location. Enabling OTEL signals automatically structures JSON logs for OTEL export, while local `stderr` logs remain plaintext.

To enable structured log output *without* enabling OTEL signals, use the `--enable-structured-logging` [host config](/docs/reference/host-config) argument. This will cause the host to emit logs as JSON, which are suitable for forwarding to a log aggregator.
To enable structured log output *without* enabling OTEL signals, use the `--enable-structured-logging` [host config](/docs/1.0/reference/host-config) argument. This will cause the host to emit logs as JSON, which are suitable for forwarding to a log aggregator.

```json
{"timestamp":"2024-01-09T17:54:59.113161Z","level":"INFO","fields":{"message":"wasmCloud host started","host_id":"NDBYMWNPI3GSXBNM6H4WG6JTVB7K42KQWPOPICMIBVUXQL36ISJZHK4D"},"target":"wasmcloud_host::wasmbus"}
```

### Viewing logs with Grafana and Loki

A set of sample configurations and Docker Compose files are available to help launch a local observability environment. After following the instructions to [**launch a wasmCloud observability ecosystem**](/docs/deployment/observability/observability-with-opentelemetry#launch-a-wasmcloud-observability-ecosystem), you can navigate to [http://localhost:5050/explore](http://localhost:5050/explore), click the dropdown in the upper left, and select **"Loki"** to view logs.
A set of sample configurations and Docker Compose files are available to help launch a local observability environment. After following the instructions to [**launch a wasmCloud observability ecosystem**](/docs/1.0/deployment/observability/observability-with-opentelemetry#launch-a-wasmcloud-observability-ecosystem), you can navigate to [http://localhost:5050/explore](http://localhost:5050/explore), click the dropdown in the upper left, and select **"Loki"** to view logs.

Refer to the `OTEL_` environment variables in the [Docker Compose file](https://github.com/wasmCloud/wasmCloud/blob/main/examples/docker/docker-compose-otel.yml) for a complete example configuration. The example uses Grafana and Loki, but wasmCloud works with any OTEL-compatible backend.
Refer to the environment variables in the [Docker Compose file](https://github.com/wasmCloud/wasmCloud/blob/main/examples/docker/docker-compose-otel.yml) for a complete example configuration. The example uses Grafana and Loki, but wasmCloud works with any OTEL-compatible backend.
60 changes: 56 additions & 4 deletions docs/deployment/observability/metrics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,62 @@ description: "How to get aggregate data on the availability and performance of a
sidebar_position: 4
---

:::info
The wasmCloud host can export metrics using the [OpenTelemetry](https://opentelemetry.io/) (OTEL) specification.

🚧 Under construction 🚧
For more information on wasmCloud's implementation of OpenTelemetry, including advanced configuration options, see [Observability with OpenTelemetry](/docs/deployment/observability/observability-with-opentelemetry.mdx).

Design for exporting [metrics via OpenTelemetry](https://opentelemetry.io/docs/specs/otel/metrics/) is underway. See the [RFC](https://github.com/wasmCloud/wasmCloud/issues/664) for more information.
### Configuration

:::
By default, metrics are disabled in the wasmCloud host. The host can be configured using [environment variables or host config flags](https://wasmcloud.com/docs/reference/host-config).

To enable all OTEL signals, including metrics, use the `WASMCLOUD_OBSERVABILITY_ENABLED` environment variable:

```bash
WASMCLOUD_OBSERVABILITY_ENABLED="true"
```

To enable only metrics, use the `WASMCLOUD_METRICS_ENABLED` environment variable:

```bash
WASMCLOUD_METRICS_ENABLED="true"
```

By default, wasmCloud hosts export signals via HTTP/OpenTelemetry Protocol (`otlp`) on port 4318.

The endpoint for metrics exports may be overwritten with the `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` environment variable:

```bash
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://my-custom-host:4318
```

See the [host config](/docs/1.0/reference/host-config) options for equivalent host configuration CLI flags.

# Format

Metrics are exported in [OpenTelemetry format](/docs/1.0/deployment/observability/observability-with-opentelemetry). The [OTEL collector](https://opentelemetry.io/docs/collector/) converts OTEL-formatted metrics to [Prometheus format](https://prometheus.io/docs/concepts/data_model/) for use with Prometheus systems.

### Metric options

The following specific metrics are included with wasmCloud:

- **Actors**:
* Time to handle invocation in nanoseconds (`wasmcloud_host.handle_rpc_message.duration`)
* Count of the number of invocations (`wasmcloud_host.actor.invocations`)
* Count of errors (`wasmcloud_host.actor.errors`)

User feedback and proposals for specific metrics is encouraged&mdash;[join the Slack](https://slack.wasmcloud.com/) or [open an issue](https://github.com/wasmCloud/wasmCloud/issues) to discuss your ideas.

### Viewing metrics with Grafana and Prometheus

A set of sample configurations and Docker Compose files are available to help launch a local observability environment. After following the instructions to [**launch a wasmCloud observability ecosystem**](/docs/1.0/deployment/observability/observability-with-opentelemetry#launch-a-wasmcloud-observability-ecosystem), there are two different ways to query metrics emitted by the wasmCloud host:

* **Prometheus' built-in query interface** at [http://localhost:9090/graph](http://localhost:9090/graph).
* **Grafana's "Explore" interface** at [http://localhost:5050/explore](http://localhost:5050/explore). Once you are in the "View," select "Prometheus" from the dropdown on the top-left corner and under the "Metric" field select the metric you would like to explore.

Depending on what you are trying to accomplish with the metrics, one or the other interface may be a better fit for you.

* If you are in the process of emitting new metrics, or you are interested in exploring the metrics in a Prometheus-native interface, the Prometheus built-in query interface is probably the better fit.

* If you are interested in seeing how the metrics would look in an interface that you might use in production, or you are looking to develop a metrics dashboard, the Grafana interface is the better fit.

Refer to the environment variables in the [Docker Compose file](https://github.com/wasmCloud/wasmCloud/blob/main/examples/docker/docker-compose-otel.yml) for a complete example configuration. The example uses Grafana and Prometheus, but wasmCloud works with any OTEL-compatible backend.
26 changes: 12 additions & 14 deletions docs/deployment/observability/observability-with-opentelemetry.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: "Using OpenTelemetry for wasmCloud observability"
sidebar_position: 1
---

wasmCloud is designed to support cloud native observability standards using [OpenTelemetry (OTEL)](https://opentelemetry.io/). OpenTelemetry is an open source project defining common APIs and standards for telemetry in cloud native systems. As of v0.82.0, wasmCloud emits **traces** and **logs** as OTEL [**signals**](https://opentelemetry.io/docs/concepts/signals/), with **metrics** support slated for v1.0.
wasmCloud is designed to support cloud native observability standards using [OpenTelemetry (OTEL)](https://opentelemetry.io/). OpenTelemetry is an open source project defining common APIs and standards for telemetry in cloud native systems. As of v1.0, wasmCloud emits **traces**, **logs**, and **metrics** as OTEL [**signals**](https://opentelemetry.io/docs/concepts/signals/).

### Data flow

Expand All @@ -14,28 +14,28 @@ A wasmCloud observability ecosystem must include an [**OTEL collector**](https:/

The OTEL documentation provides detailed instructions on how to [install and run the OTEL collector](https://opentelemetry.io/docs/collector/quick-start/). Additionally, you can find a [sample configuration for an OTEL collector](https://github.com/wasmCloud/wasmCloud/blob/main/examples/docker/otel-collector.yaml) in the wasmCloud observability ecosystem demo on GitHub.

While the OTEL collector is the recommended point of configuration for complex signal routing, there are a set of core configurations available at the level of a [wasmCloud host](https://wasmcloud.com/docs/concepts/hosts).
While the OTEL collector is the recommended point of configuration for complex signal routing, there are a set of core configurations available at the level of a [wasmCloud host](https://wasmcloud.com/docs/concepts/hosts). The host can be configured using [environment variables or host config flags](/docs/1.0/reference/host-config).

### Configuration

Enable the wasmCloud host to emit OTEL signals with the `OTEL_TRACES_EXPORTER` environment variable:
Enable the wasmCloud host to emit all OTEL signals with the `WASMCLOUD_OBSERVABILITY_ENABLED` environment variable:

```bash
OTEL_TRACES_EXPORTER=otlp
WASMCLOUD_OBSERVABILITY_ENABLED="true"
```

In v0.82, this enables all supported OTEL signals at once&mdash;not just traces. (A more accurately descriptive environment variable and command line argument is to come in v1.0.)

By default, OTEL signals are exported on port 4318, the standard **HTTP/OpenTelemetry Protocol (`otlp`)** port. wasmCloud only supports the `otlp` format. The endpoint for `otlp` exports may be overwritten with the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable:
By default, OTEL signals are exported on port 4318, the standard **HTTP/OpenTelemetry Protocol (`otlp`)** port. wasmCloud only supports the `otlp` format. The endpoint for all `otlp` exports may be overwritten with the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable:

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://my-custom-host:4318
```

While these options are typically passed as part of [host configuration](https://wasmcloud.com/docs/reference/host-config), you can launch a wasmCloud observability ecosystem locally by passing environment variables to `wash up`:
It is also possible to enable and overwrite endpoints for only specific signals&mdash;see the pages for [traces](/docs/1.0/deployment/observability/tracing), [logs](/docs/1.0/deployment/observability/logs), and [metrics](/docs/1.0/deployment/observability/metrics) for more details.

While these options are typically passed as part of [host configuration](/docs/1.0/reference/host-config), you can launch a wasmCloud observability ecosystem locally by passing environment variables to `wash up`:

```bash
OTEL_TRACES_EXPORTER=otlp wash up
WASMCLOUD_OBSERVABILITY_ENABLED="true" wash up
```

### Launch a wasmCloud observability ecosystem
Expand Down Expand Up @@ -67,7 +67,7 @@ docker compose -f docker-compose-otel.yml up
Make sure a wasmCloud host is running locally with OTEL signals enabled:

```bash
OTEL_TRACES_EXPORTER=otlp wash up
WASMCLOUD_OBSERVABILITY_ENABLED="true" wash up
```

Now you can access Grafana's "Explore" interface at [http://localhost:5050/explore](http://localhost:5050/explore).
Expand All @@ -76,16 +76,14 @@ Now you can access Grafana's "Explore" interface at [http://localhost:5050/explo

Because wasmCloud's implementation of OpenTelemetry observability uses upstream OTEL libraries, a variety of advanced configuration options are available via environment variables defined in the upstream [OTEL SDK](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/) and [OTLP Exporter SDK](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/).

For example, users have several options for [trace sampling](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration), [log batch management](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#batch-logrecord-processor), and setting [per signal overrides for endpoints](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_traces_endpoint).

Not all upstream options are supported&mdash;for example, the wasmCloud host does not support changing format to gRPC.
For example, users have several options for [trace sampling](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration) and [log batch management](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#batch-logrecord-processor). Not all upstream options are supported&mdash;for example, the wasmCloud host does not support changing format to gRPC.

:::info[Note]
In cases where the system receives conflicting configurations for the same option via environment variable and command line argument, upstream OpenTelemetry libraries prioritize configurations set via environment variable. Refer to the [OpenTelemetry documentation](https://opentelemetry.io/docs/languages/sdk-configuration/) for more details.
:::

### Further reading

Signal-specific options on the wasmCloud host are detailed on the [traces](/docs/deployment/observability/tracing.mdx), [logs](/docs/deployment/observability/logs.mdx), and [metrics](/docs/deployment/observability/metrics.mdx) pages. For more information on OpenTelemetry, refer to the [project documentation](https://opentelemetry.io/docs/).
Signal-specific options on the wasmCloud host are detailed on the [traces](/docs/1.0/deployment/observability/tracing), [logs](/docs/1.0/deployment/observability/logs), and [metrics](/docs/1.0/deployment/observability/metrics) pages. For more information on OpenTelemetry, refer to the [project documentation](https://opentelemetry.io/docs/).


22 changes: 15 additions & 7 deletions docs/deployment/observability/tracing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,29 +8,37 @@ Traces are essential for understanding the flow of messages through a distribute

The wasmCloud host can record and export distributed traces using the [OpenTelemetry](https://opentelemetry.io/) (OTEL) specification. If you haven't worked with distributed traces before, [this primer](https://opentelemetry.io/docs/concepts/observability-primer/#distributed-traces) is a great place to start.

For more information on wasmCloud's implementation of OpenTelemetry, including advanced configuration options, see [Observability with OpenTelemetry](/docs/deployment/observability/observability-with-opentelemetry.mdx).
For more information on wasmCloud's implementation of OpenTelemetry, including advanced configuration options, see [Observability with OpenTelemetry](/docs/1.0/deployment/observability/observability-with-opentelemetry).

### Configuration

By default, tracing is disabled in the wasmCloud host. To enable traces, use the `OTEL_TRACES_EXPORTER` environment variable:
By default, tracing is disabled in the wasmCloud host. The host can be configured using [environment variables or host config flags](/docs/1.0/reference/host-config).

To enable all OTEL signals, including traces, use the `WASMCLOUD_OBSERVABILITY_ENABLED` environment variable:

```bash
WASMCLOUD_OBSERVABILITY_ENABLED="true"
```

To enable only traces, use the `WASMCLOUD_TRACES_ENABLED` environment variable:

```bash
OTEL_TRACES_EXPORTER=otlp
WASMCLOUD_TRACES_ENABLED="true"
```

By default, wasmCloud hosts export traces via HTTP/OpenTelemetry Protocol (`otlp`) on port 4318.

The endpoint for trace exports may be overwritten with the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable:
The endpoint for trace exports may be overwritten with the `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` environment variable:

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://my-custom-host:4318
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://my-custom-host:4318
```

These configurations can also be set with the `--otel-traces-exporter` and `--otel-exporter-otlp-endpoint` [host config](/docs/reference/host-config) options.
See the [host config](/docs/1.0/reference/host-config) options for equivalent host configuration CLI flags.

### Viewing traces with Grafana and Tempo

A set of sample configurations and Docker Compose files are available to help launch a local observability environment. After following the instructions to [**launch a wasmCloud observability ecosystem**](/docs/deployment/observability/observability-with-opentelemetry#launch-a-wasmcloud-observability-ecosystem), you can navigate to [http://localhost:5050/explore](http://localhost:5050/explore), click the dropdown in the upper left, and select **"Tempo"** to view traces.
A set of sample configurations and Docker Compose files are available to help launch a local observability environment. After following the instructions to [**launch a wasmCloud observability ecosystem**](/docs/1.0/deployment/observability/observability-with-opentelemetry#launch-a-wasmcloud-observability-ecosystem), you can navigate to [http://localhost:5050/explore](http://localhost:5050/explore), click the dropdown in the upper left, and select **"Tempo"** to view traces.

There are several ways to query traces in Tempo. To see all traces from the host, change the "Query type" tab to "Search," then click the "Service Name" dropdown and select "wasmcloud-host." You can also increase the "Limit" field to something more than the default (20). To search, press Shift-Enter. You can click on any of the Trace IDs to view all the spans associated with the trace.

Expand Down
Loading

0 comments on commit 4d7a0b3

Please sign in to comment.