From d5fd9af2aee59115688579f03da60df915e98b1c Mon Sep 17 00:00:00 2001 From: Alan Conway Date: Fri, 3 Nov 2023 11:40:20 -0400 Subject: [PATCH] Added enhancements/cluster-logging/content-filter.md --- .../cluster-logging/content-filter.md | 212 ++++++++++++++++++ 1 file changed, 212 insertions(+) create mode 100644 enhancements/cluster-logging/content-filter.md diff --git a/enhancements/cluster-logging/content-filter.md b/enhancements/cluster-logging/content-filter.md new file mode 100644 index 00000000000..324ebdf1a7b --- /dev/null +++ b/enhancements/cluster-logging/content-filter.md @@ -0,0 +1,212 @@ +--- +title: content-filter +authors: + - "@alanconway" +reviewers: + - "@jcantrill" +approvers: + - "@jcantrill" +api-approvers: + - "@jcantrill" +creation-date: 2023-11-03 +last-updated: 2023-11-03 +tracking-link: + - [LOG-2155 Filter log messages based on metadata and content.](https://issues.redhat.com/browse/LOG-2155) +see-also: [] +replaces: [] +superseded-by: [] +--- + +# Content filters for log records + +## Summary + +Allow users to reduce the volume of log data by: +1. Dropping unwanted log records completely. +2. Pruning unwanted fields from log records. + +The new prune/drop content filters use the same framework as the kube-api-audit filter. +This framework can be extended with new types of filters in future. + +**NOTE**: Content filters are distinct from input selectors. +Input selectors select or ignore entire log _streams_ based on _source metadata_. +Content filters _edit_ log streams (remove and modify records) based on _record content_. + +## Motivation + +Collecting all logs from a cluster produces a large amount of data, which can be expensive to transport and store. +A lot of log data is low-value noise that does not need to be stored. + +The ideal solution is to configure each application to generate only valuable log data, however: +- Not all applications can be tuned to remove noise while keeping important data. +- Configuration of applications is not always under the control of the cluster administrator. +- The value of logs varies depending on use: for example debugging a specific problem vs. monitoring overall health. + +### User Stories + +* As a logging administrator I want to drop log records by severity or other field values in the record. +* As a logging administrator I want to prune log records by removing pod annotations, or other uninteresting fields. + +### Goals + +The user can configure the ClusterLogForwarder to drop and prune log records according to their requirements. + +### Non-Goals + +No complex record transformations, for example adding new fields with computed values. +More complex filters MAY be added in future, but not as part of this enhancement. + +## Proposal + +### Workflow Description + +**log administrator** is a human responsible for configuring log collection, storage and forwarding in a cluster. + +1. The log administrator creates named `filter` sections in a `ClusterLogForwarder` +2. Pipelines that require filtering have a `filterRef` field with a list of filter names. +3. The collector edits log streams according to the filters before forwarding. + +### API Extensions + +A new `filters` section in the `ClusterLogForwarder` allows named filters to be defined. +This proposal defines two types of filter "prune" and "drop". + +#### Prune filters + +A "prune" filter removes fields from each record passing through the filter. + +##### API + +``` yaml +spec: + filters: + - name: "" # User selects filter name + type: prune + prune: + - matches: "" # Regular expression, remove all fields with paths that match. + - notMatches: "" # Regular expression, remove all fields with paths that do NOT match. +``` + +##### Examples + +``` yaml + spec: + filters: + - name: foo + type: prune + prune: + - matches: "kubernetes\\.flat_labels|pipeline_metadata" + pipelines: + - name: bar + filterRefs: ["foo"] +``` + +``` yaml +- "PruneK8sLabels": +```yaml +- name: PruneK8sLabels + type: prune + prune: + - in: + - .kubernetes.flat_labels + - .kubernetes.pod_labels + - .kubernetes.namespace_labels + - .kubernetes.annotations + +``` + +#### Drop filters + +##### API + +A drop filter applies a sequence of tests to a log record and drops the record if any test passes. +Each test contains a sequence of conditions, all conditions must true for the test to pass. + +``` yaml +spec: + filters: + - name: # Provided by the user + type: drop + drop: + - test: + - field: # JSON path to the field + # Requires exactly one of the following conditions. + matches: # regular expression match + notMatches: # regular expression does not match +``` + +Note: +- If _all_ conditions in a test are true, the test passes. +- If _any_ test in the drop filter passes, the record is dropped. +- If there is an error evaluating a condition (e.g. a missing field), that condition evaluates to false. + Evaluation continues as normal. + +The drop filter is equivalent to a boolean OR of AND clauses. Any boolean expression can be reduced to this form. + +##### Example + +``` yaml +filters: + - name: important + type: drop + drop: + - tests: + - field: .kubernetes.namespace_name + notMatches: "very-important" # Keep everything from this namespace. + - field: .level # Keep important levels + matches: "warning|error|critical" +``` + +### Implementation Details + +### JSON path + +Need to document the rules for field paths, these are a subset of the JSON path spec. +In fact we will use the same subset as Vector does, but we should describe the rules explicitly +and not refer to vector docs, so we don't create assumptions in the API about the underlying collector. + +### Metrics + +Vector provides metrics for records in/out of each vector node. +Records dropped by filters can be computed from this. +We should provide _brief_ hints and links to these metrics, but not duplicate vector docs. + +### Risks and Mitigations + +- Performance impact, need to benchmark slowdown due to rule evaluation, but balance against reduction in data volume. +- No new security issues. +- Extensions to ClusterLogForwarder will be reviewed as part of normal code review. + +### Drawbacks + +- Possible performance impact (negative and positive) +- User can damage log records in ways that make them invalid. + +## Design Details + +### Upgrade / Downgrade Strategy + +This is a new feature, opt-in and backward compatible. Should not cause any upgrade issues. + +### Version Skew Strategy + + +### Operational Aspects of API Extensions + +#### Failure Modes + +Invalid filters SHOULD be detected at reconcile time. +Any errors at run time will cause a filter to be ignored, will not cause overall failure. + +#### Support Procedures +Support calls complaining of missing log records or missing data in log records: check if the data was intentionally removed by a filter. +### Test Plan +### Graduation Criteria +#### Dev Preview -> Tech Preview +#### Tech Preview -> GA +#### Removing a deprecated feature + +## Implementation History + +## Alternatives +