Skip to content

Commit

Permalink
Update the Agent Concepts topics (#5956)
Browse files Browse the repository at this point in the history 
* Update grammar spelling and syntax

* Updates for semantic line breaks

* More cleanup of grammar spelling and word use

* Another pass at cleanup and consistency

* Additional edits nad updates for grammar style

* Fix image link

* Update images grammar spelling xrefs

* More language cleanup with focus on passive

* Convert links to doc references

* Tidy up punctuation and fix broken links

* Update docs/sources/flow/config-language/syntax.md
  • Loading branch information
@clayton-cornell
clayton-cornell authored Dec 14, 2023
1 parent 59af364 commit b42073f
Show file tree
Hide file tree
Showing 19 changed files with 423 additions and 1,046 deletions.
304 changes: 0 additions & 304 deletions docs/sources/assets/concepts_example_pipeline.svg
View file

This file was deleted.

216 changes: 0 additions & 216 deletions docs/sources/assets/flow_referencing_exports_diagram.svg
View file

This file was deleted.

2 changes: 1 addition & 1 deletion docs/sources/flow/concepts/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,6 @@ weight: 100

# Concepts

This section explains primary concepts of {{< param "PRODUCT_NAME" >}}.
This section explains the primary concepts of {{< param "PRODUCT_NAME" >}}.

{{< section >}}
45 changes: 17 additions & 28 deletions docs/sources/flow/concepts/clustering.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,32 +15,25 @@ weight: 500

# Clustering (beta)

Clustering enables a fleet of {{< param "PRODUCT_ROOT_NAME" >}}s to work together for workload distribution
and high availability. It helps create horizontally scalable deployments with
minimal resource and operational overhead.
Clustering enables a fleet of {{< param "PRODUCT_ROOT_NAME" >}}s to work together for workload distribution and high availability.
It helps create horizontally scalable deployments with minimal resource and operational overhead.

To achieve this, {{< param "PRODUCT_NAME" >}} makes use of an eventually consistent model that
assumes all participating {{< param "PRODUCT_ROOT_NAME" >}}s are interchangeable and converge on using the
same configuration file.
To achieve this, {{< param "PRODUCT_NAME" >}} makes use of an eventually consistent model that assumes all participating
{{< param "PRODUCT_ROOT_NAME" >}}s are interchangeable and converge on using the same configuration file.

The behavior of a standalone, non-clustered {{< param "PRODUCT_ROOT_NAME" >}} is the same as if it was a
single-node cluster.
The behavior of a standalone, non-clustered {{< param "PRODUCT_ROOT_NAME" >}} is the same as if it were a single-node cluster.

You configure clustering by passing `cluster` command-line flags to the [run][]
command.
You configure clustering by passing `cluster` command-line flags to the [run][] command.

## Use cases

### Target auto-distribution

Target auto-distribution is the most basic use case of clustering; it allows
scraping components running on all peers to distribute scrape load between
themselves. For target auto-distribution to work correctly, all {{< param "PRODUCT_ROOT_NAME" >}} in the
same cluster must be able to reach the same service discovery APIs and must be
able to scrape the same targets.
Target auto-distribution is the most basic use case of clustering.
It allows scraping components running on all peers to distribute the scrape load between themselves.
Target auto-distribution requires that all {{< param "PRODUCT_ROOT_NAME" >}} in the same cluster can reach the same service discovery APIs and scrape the same targets.

You must explicitly enable target auto-distribution on components by defining a
`clustering` block, such as:
You must explicitly enable target auto-distribution on components by defining a `clustering` block.

```river
prometheus.scrape "default" {
Expand All @@ -52,18 +45,15 @@ prometheus.scrape "default" {
}
```

A cluster state change is detected when a new node joins or an existing node goes away. All participating components locally
recalculate target ownership and re-balance the number of targets they’re
scraping without explicitly communicating ownership over the network.
A cluster state change is detected when a new node joins or an existing node leaves.
All participating components locally recalculate target ownership and re-balance the number of targets they’re scraping without explicitly communicating ownership over the network.

Target auto-distribution allows you to dynamically scale the number of {{< param "PRODUCT_ROOT_NAME" >}}s to distribute workload during peaks.
It also provides resiliency because targets are automatically picked up by one of the node peers if a node goes away.
It also provides resiliency because targets are automatically picked up by one of the node peers if a node leaves.

{{< param "PRODUCT_NAME" >}} uses a fully-local consistent hashing algorithm to distribute
targets, meaning that, on average, only ~1/N of the targets are redistributed.
{{< param "PRODUCT_NAME" >}} uses a local consistent hashing algorithm to distribute targets, meaning that, on average, only ~1/N of the targets are redistributed.

Refer to component reference documentation to discover whether it supports
clustering, such as:
Refer to component reference documentation to discover whether it supports clustering, such as:

- [prometheus.scrape][]
- [pyroscope.scrape][]
Expand All @@ -72,9 +62,8 @@ clustering, such as:

## Cluster monitoring and troubleshooting

To monitor your cluster status, you can check the {{< param "PRODUCT_NAME" >}} UI [clustering page][].
The [debugging][] topic contains some clues to help pin down probable
clustering issues.
You can use the {{< param "PRODUCT_NAME" >}} UI [clustering page][] to monitor your cluster status.
Refer to [Debugging clustering issues][debugging] for additional troubleshooting information.

{{% docs/reference %}}
[run]: "/docs/agent/ -> /docs/agent/<AGENT_VERSION>/flow/reference/cli/run.md#clustering-beta"
Expand Down
104 changes: 40 additions & 64 deletions docs/sources/flow/concepts/component_controller.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,7 @@ weight: 200

# Component controller

The _component controller_ is the core part of {{< param "PRODUCT_NAME" >}} which manages
components at runtime.
The _component controller_ is the core part of {{< param "PRODUCT_NAME" >}} which manages components at runtime.

The component controller is responsible for:

Expand All @@ -25,16 +24,12 @@ The component controller is responsible for:

## Component graph

As discussed in [Components][], a relationship between components is created
when an expression is used to set the argument of one component to an exported
field of another component.
A relationship between [components][Components] is created when an expression is used to set the argument of one component to an exported field of another component.

The set of all components and the relationships between them define a [Directed
Acyclic Graph][DAG] (DAG), which informs the component controller which
references are valid and in what order components must be evaluated.
The set of all components and the relationships between them define a [Directed Acyclic Graph][DAG] (DAG),
which informs the component controller which references are valid and in what order components must be evaluated.

For a configuration file to be valid, components must not reference themselves or
contain a cyclic reference:
For a configuration file to be valid, components must not reference themselves or contain a cyclic reference.

```river
// INVALID: local.file.some_file can not reference itself:
Expand All @@ -55,87 +50,68 @@ local.file "b" {

## Component evaluation

A component is _evaluated_ when its expressions are computed into concrete
values. The computed values are then used to configure the component's runtime
behavior. The component controller is finished loading once all components are
evaluated, configured, and running.
A component is _evaluated_ when its expressions are computed into concrete values.
The computed values configure the component's runtime behavior.
The component controller is finished loading once all components are evaluated, configured, and running.

The component controller only evaluates a given component after evaluating all
of that component's dependencies. Components that do not depend on other
components can be evaluated at any time during the evaluation process.
The component controller only evaluates a given component after evaluating all of that component's dependencies.
Components that don't depend on other components can be evaluated anytime during the evaluation process.

## Component reevaluation

As mentioned in [Components][], a component is dynamic: a component can update
its exports any number of times throughout its lifetime.
A [component][Components] is dynamic. A component can update its exports any number of times throughout its lifetime.

When a component updates its exports, a _controller reevaluation_ is triggered:
the component controller reevaluates any component which references the changed
component, any components which reference those components, and so on, until
all affected components are reevaluated.
A _controller reevaluation_ is triggered when a component updates its exports.
The component controller reevaluates any component that references the changed component, any components that reference those components,
and so on, until all affected components are reevaluated.

## Component health

At any given time, a component can have one of the following health states:

1. Unknown: default state, the component isn't running yet.
2. Healthy: the component is working as expected.
3. Unhealthy: the component is not working as expected.
4. Exited: the component has stopped and is no longer running.
1. Unknown: The default state. The component isn't running yet.
1. Healthy: The component is working as expected.
1. Unhealthy: The component isn't working as expected.
1. Exited: The component has stopped and is no longer running.

By default, the component controller determines the health of a component. The
component controller marks a component as healthy as long as that component is
running and its most recent evaluation succeeded.
By default, the component controller determines the health of a component.
The component controller marks a component as healthy as long as that component is running and its most recent evaluation succeeded.

Some components can report their own component-specific health information. For
example, the `local.file` component reports itself as unhealthy if the file it
was watching gets deleted.
Some components can report their own component-specific health information.
For example, the `local.file` component reports itself as unhealthy if the file it was watching gets deleted.

The overall health of a component is determined by combining the
controller-reported health of the component with the component-specific health
information.
The overall health of a component is determined by combining the controller-reported health of the component with the component-specific health information.

An individual component's health is independent of the health of any other
components it references: a component can be marked as healthy even if it
references an exported field of an unhealthy component.
An individual component's health is independent of the health of any other components it references.
A component can be marked as healthy even if it references an exported field of an unhealthy component.

## Handling evaluation failures

When a component fails to evaluate, it is marked as unhealthy with the reason
for why the evaluation failed.
When a component fails to evaluate, it's marked as unhealthy with the reason for why the evaluation failed.

When an evaluation fails, the component continue operating as normal: it
continues using its previous set of evaluated arguments, and it can continue
exporting new values.
When an evaluation fails, the component continues operating as normal.
The component continues using its previous set of evaluated arguments and can continue exporting new values.

This prevents failure propagation: if your `local.file` component which watches
API keys suddenly stops working, other components continues using the last
valid API key until the component returns to a healthy state.
This behavior prevents failure propagation.
If your `local.file` component, which watches API keys, suddenly stops working, other components continue using the last valid API key until the component returns to a healthy state.

## In-memory traffic

Components which expose HTTP endpoints, such as [prometheus.exporter.unix][],
can expose an internal address which will completely bypass the network and
communicate in-memory. This allows components within the same process to
communicate with one another without needing to be aware of any network-level
protections such as authentication or mutual TLS.
Components that expose HTTP endpoints, such as [prometheus.exporter.unix][], can expose an internal address that completely bypasses the network and communicate in-memory.
Components within the same process can communicate with one another without needing to be aware of any network-level protections such as authentication or mutual TLS.

The internal address defaults to `agent.internal:12345`. If this address
collides with a real target on your network, change it to something unique
using the `--server.http.memory-addr` flag in the [run][] command.
The internal address defaults to `agent.internal:12345`.
If this address collides with a real target on your network, change it to something unique using the `--server.http.memory-addr` flag in the [run][] command.

Components must opt-in to using in-memory traffic. See the individual
documentation for components to learn if in-memory traffic is supported.
Components must opt-in to using in-memory traffic.
Refer to the individual documentation for components to learn if in-memory traffic is supported.

## Updating the configuration file

Both the `/-/reload` HTTP endpoint and the `SIGHUP` signal can be used to
inform the component controller to reload the configuration file. When this happens,
the component controller will synchronize the set of running components with
the ones in the configuration file, removing components which are no longer defined in
the configuration file and creating new components which were added to the configuration
file. All components managed by the controller will be reevaluated after
reloading.
The `/-/reload` HTTP endpoint and the `SIGHUP` signal can inform the component controller to reload the configuration file.
When this happens, the component controller synchronizes the set of running components with the ones in the configuration file,
removing components no longer defined in the configuration file and creating new components added to the configuration file.
All components managed by the controller are reevaluated after reloading.

[DAG]: https://en.wikipedia.org/wiki/Directed_acyclic_graph

Expand Down
Loading

0 comments on commit b42073f

Please sign in to comment.