Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add some guidelines about the use of metrics metadata and dimensions #5270

Merged
merged 1 commit into from
Feb 23, 2023
Merged

Add some guidelines about the use of metrics metadata and dimensions #5270

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 37 additions & 6 deletions docs/generic_guidelines.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,25 +22,56 @@ All fields produced by an integration must be mapped by `fields.yml`. This guara

As part of the field definition, there are two settings that add metadata which will help Kibana graphing it:

- `unit` applies to all data types, defines the units of the field.
- `metric_type` applies to metric events only, to be added to metric fields, it defines their type.
- `unit` applies to all data types, defines the units of the field. Some
examples of units are `byte` or `ms`. When using `percent` for percentages,
the convention is to use 1 for 100%. You can find the full list of supported
units in the [package spec](https://github.com/elastic/package-spec/blob/ff8286d0c40ad76bb082e9c8ea78f4551c2519c1/spec/integration/data_stream/fields/fields.spec.yml#L103).
- `metric_type` applies to metric events only, to be added to metric fields,
it defines their metric type. It can be of type `gauge` or `counter`. Counters
are used for metrics that always increase over time, as number of visits.
Gauges are used for amounts that can increase or decrease over time, as the
memory used.

Elasticsearch docs details the [expected values for these two fields](https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-field-meta.html).

Other applications, like Kibana, can use the information provided by this
metadata when accessing these fields. The `unit` is used when formatting the
values of the field, and the `metric_type` can be used to provide better defaults
when quering the data.

##### Specify dimensions

A set of fields of a data stream can be defined as dimensions. A set of dimensions
with the same values identify a single time serie. It is important to choose wisely
the set of fields, they should be the minimal set of dimensions required to
properly identify any time serie included in the data stream. Too few dimensions can
mix data of multiple time series into a single one, too many can impact performance.
with the same values identify a single time serie.

It is important to choose wisely the set of fields, they should be the minimal set
of dimensions required to properly identify any time serie included in the data stream.
Too few dimensions can mix data of multiple time series into a single one, too many can
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we provide specific guideline on lower bound and higher bound on dimensions count? Otherwise, this question will come up in the future, in multiple forums.

I am also trying to understand, why elastic TSDB enforces a lower bound on dimensions (too few dimensions is an issue) and where-as in prometheus TSDB metrics can be ingested without labels(dimension) and does not break anything there.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we provide specific guideline on lower bound and higher bound on dimensions count? Otherwise, this question will come up in the future, in multiple forums.

This is quite dependant of the context. And even for the same object it may vary depending on the available data. For example if you are monitoring containers in Kubernetes pods, and you have the container ID, this may be the only dimension needed, but if you don't have the container ID, you may need to use at least the container name, the pod name and the pod namespace as dimensions.

There are things like HTTP requests where we may want to store lots of dimensions, such as the method, source IP, url path, query parameters, important headers, backend handling the request and so on. For these cases we have a higher bound of 16 dimensions, that is checked by Elasticsearch itself. But this limit can be raised per data stream if needed, using the index.mapping.dimension_fields.limit option.

I am also trying to understand, why elastic TSDB enforces a lower bound on dimensions (too few dimensions is an issue) and where-as in prometheus TSDB metrics can be ingested without labels(dimension) and does not break anything there.

It is actually similar with Prometheus, it is not that things "break", but there can be misbehaviours.
For example if you ingest Kubernetes pod metrics only with the label name, and you have pods with the same name in different namespaces, you are going to have the same kind of problems also in Prometheus (only one metric is stored for the same timestamp, and mixed metrics of different pods in the same time serie).
Thus you need to use the name and the namespace as labels. If you add the namespace label later, you are creating new time series, disconnected from the previous ones, even for pods where their name was unique.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jsoriano for the clarifications. I am good for the document to the merged.

There is a separate discussion happening for dimensions in separate threads, to clarify/close some of these aspects.

impact performance.

A field can be configured as a dimension by setting `dimension: true` on its
definition.

Only fields of certain data types can be defined as dimensions. These data types
include keywords, IPs and numeric types.

Some guidelines to take into account when chosing dimensions:
- They can affect ingestion performance, it is recommended to have as few dimensions as
possible. When selecting dimensions, try to avoid redundant ones, as unique
identifiers and names that refer to the same object.
- Be also careful with having too few dimensions. There can be only one document
with the same timestamp for a given set of dimensions. This can lead to data
loss if different objects produce the same dimensions.
- Changing dimensions can be a breaking change. A different set of dimensions
produces a different time serie, even if they select the same data.

Declaring dimensions is a requisite to use TSDB indexes. These indexes are
optimized for time series use cases, bringing disk storage savings and additional
queries and aggregations.

TSDB indexes can be enabled in data streams by setting `elasticsearch.index_mode: time_series`
in their manifests.

#### Logs and Metrics UI compatibility

When applicable an integrataion package should provide the relevant fields for the Logs and Metrics Apps. This is especially relevant for integrations that are focused on compute-resources (VMs, containers, etc.).
Expand Down