Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Align elastic-agent-standalone manifest with the kubernetes package changes #29595

Merged
merged 5 commits into from
Dec 23, 2021
Merged

Align elastic-agent-standalone manifest with the kubernetes package changes #29595

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ba0f804
align elastic-agent-standalone manifest with the managed version
tetianakravchenko Dec 23, 2021
83a7432
revetn docker image version
tetianakravchenko Dec 23, 2021
d9b1436
remove ES_HOST used to run test locally
tetianakravchenko Dec 23, 2021
7c00ce9
set default values for container parser implicitly
tetianakravchenko Dec 23, 2021
0040823
remove skip_older as it is a default value anyway
tetianakravchenko Dec 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 72 additions & 7 deletions deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ data:
meta:
package:
name: kubernetes
version: 0.2.8
version: 1.9.0
data_stream:
namespace: default
streams:
Expand All @@ -54,6 +54,7 @@ data:
- event
period: 10s
add_metadata: true
skip_older: true
ChrsMark marked this conversation as resolved.
Show resolved Hide resolved
- data_stream:
dataset: kubernetes.state_container
type: metrics
Expand All @@ -72,6 +73,15 @@ data:
hosts:
- 'kube-state-metrics:8080'
period: 10s
- data_stream:
dataset: kubernetes.state_daemonset
type: metrics
metricsets:
- state_daemonset
add_metadata: true
hosts:
- 'kube-state-metrics:8080'
period: 10s
- data_stream:
dataset: kubernetes.state_deployment
type: metrics
Expand Down Expand Up @@ -214,20 +224,75 @@ data:
fields:
ecs.version: 1.12.0
- name: container-log
type: logfile
type: filestream
use_output: default
meta:
package:
name: log
version: 0.4.6
name: kubernetes
version: 1.9.0
data_stream:
namespace: default
streams:
- data_stream:
dataset: generic
symlinks: true
dataset: kubernetes.container_logs
type: logs
prospector.scanner.symlinks: true
parsers:
- container:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we should keep the manifest as simple as possible and leave the defaults with not comments. The extra fields are documented for anyone who wants to change settings. I would just leave

parsers:
        - container: ~

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is current default - see https://github.com/elastic/integrations/blob/master/packages/kubernetes/data_stream/container_logs/agent/stream/stream.yml.hbs#L7-L10, was introduced in elastic/integrations#2345 to make it possible to configure those default values

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Defaults are all streams and auto format. See in https://github.com/elastic/integrations/blob/master/packages/kubernetes/data_stream/container_logs/manifest.yml#L27.
For filestream input if we don't set them like

- container: ~

it takes the defaults.
For additionalParsersConfig we just added the possibility for a user to add another parser via advanced options.
The default is still commented out.

Copy link
Contributor Author

@tetianakravchenko tetianakravchenko Dec 23, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, ok, I understood what do you mean, I thought to set it the same way as done in the package, but you are right - in the end it is the same.
setting default values implicitly - 7c00ce9

stream: all
format: auto
# - ndjson:
# target: json
# - multiline:
# type: pattern
# pattern: '^\['
# negate: true
# match: after
paths:
- /var/log/containers/*${kubernetes.container.id}.log
- name: audit-log
type: filestream
use_output: default
meta:
package:
name: kubernetes
version: 1.9.0
data_stream:
namespace: default
streams:
- data_stream:
dataset: kubernetes.audit_logs
type: logs
exclude_files:
- .gz$
parsers:
- ndjson:
add_error_key: true
target: kubernetes_audit
paths:
- /var/log/kubernetes/kube-apiserver-audit.log
processors:
- rename:
fields:
- from: kubernetes_audit
to: kubernetes.audit
- script:
id: dedot_annotations
lang: javascript
source: |
function process(event) {
var audit = event.Get("kubernetes.audit");
for (var annotation in audit["annotations"]) {
var annotation_dedoted = annotation.replace(/\./g,'_')
event.Rename("kubernetes.audit.annotations."+annotation, "kubernetes.audit.annotations."+annotation_dedoted)
}
return event;
} function test() {
var event = process(new Event({ "kubernetes": { "audit": { "annotations": { "authorization.k8s.io/decision": "allow", "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-scheduler\" of ClusterRole \"system:kube-scheduler\" to User \"system:kube-scheduler\"" } } } }));
if (event.Get("kubernetes.audit.annotations.authorization_k8s_io/decision") !== "allow") {
throw "expected kubernetes.audit.annotations.authorization_k8s_io/decision === allow";
}
}
- name: system-metrics
type: system/metrics
use_output: default
Expand Down Expand Up @@ -332,7 +397,7 @@ data:
meta:
package:
name: kubernetes
version: 0.2.8
version: 1.9.0
data_stream:
namespace: default
streams:
Expand Down
79 changes: 72 additions & 7 deletions deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-daemonset-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ data:
meta:
package:
name: kubernetes
version: 0.2.8
version: 1.9.0
data_stream:
namespace: default
streams:
Expand All @@ -54,6 +54,7 @@ data:
- event
period: 10s
add_metadata: true
skip_older: true
- data_stream:
dataset: kubernetes.state_container
type: metrics
Expand All @@ -72,6 +73,15 @@ data:
hosts:
- 'kube-state-metrics:8080'
period: 10s
- data_stream:
dataset: kubernetes.state_daemonset
type: metrics
metricsets:
- state_daemonset
add_metadata: true
hosts:
- 'kube-state-metrics:8080'
period: 10s
- data_stream:
dataset: kubernetes.state_deployment
type: metrics
Expand Down Expand Up @@ -214,20 +224,75 @@ data:
fields:
ecs.version: 1.12.0
- name: container-log
type: logfile
type: filestream
use_output: default
meta:
package:
name: log
version: 0.4.6
name: kubernetes
version: 1.9.0
data_stream:
namespace: default
streams:
- data_stream:
dataset: generic
symlinks: true
dataset: kubernetes.container_logs
type: logs
prospector.scanner.symlinks: true
parsers:
- container:
stream: all
format: auto
# - ndjson:
# target: json
# - multiline:
# type: pattern
# pattern: '^\['
# negate: true
# match: after
paths:
- /var/log/containers/*${kubernetes.container.id}.log
- name: audit-log
type: filestream
use_output: default
meta:
package:
name: kubernetes
version: 1.9.0
data_stream:
namespace: default
streams:
- data_stream:
dataset: kubernetes.audit_logs
type: logs
exclude_files:
- .gz$
parsers:
- ndjson:
add_error_key: true
target: kubernetes_audit
paths:
- /var/log/kubernetes/kube-apiserver-audit.log
processors:
- rename:
fields:
- from: kubernetes_audit
to: kubernetes.audit
- script:
id: dedot_annotations
lang: javascript
source: |
function process(event) {
var audit = event.Get("kubernetes.audit");
for (var annotation in audit["annotations"]) {
var annotation_dedoted = annotation.replace(/\./g,'_')
event.Rename("kubernetes.audit.annotations."+annotation, "kubernetes.audit.annotations."+annotation_dedoted)
}
return event;
} function test() {
var event = process(new Event({ "kubernetes": { "audit": { "annotations": { "authorization.k8s.io/decision": "allow", "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-scheduler\" of ClusterRole \"system:kube-scheduler\" to User \"system:kube-scheduler\"" } } } }));
if (event.Get("kubernetes.audit.annotations.authorization_k8s_io/decision") !== "allow") {
throw "expected kubernetes.audit.annotations.authorization_k8s_io/decision === allow";
}
}
- name: system-metrics
type: system/metrics
use_output: default
Expand Down Expand Up @@ -332,7 +397,7 @@ data:
meta:
package:
name: kubernetes
version: 0.2.8
version: 1.9.0
data_stream:
namespace: default
streams:
Expand Down