Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Fix id for config map #37545

Merged
merged 4 commits into from
Jan 8, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion deploy/kubernetes/filebeat-kubernetes.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -113,14 +113,15 @@ data:
filebeat.yml: |-
filebeat.inputs:
- type: filestream
id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
id: kubernetes-container-logs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These substitutions should work only in autodiscover, not in regular input configuration, AFAIK.

I see @gsantoro made this change in #37401
How was it tested?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was not correctly tested in that PR, this is a fix to that

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@constanca-m My point is rather if this was not tested, how can we trust the rest of the changes from that PR? Perhaps we need to ask @gsantoro to re-test those changes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hello @rdner , sorry that was my fault. It was an unintended change. I'll do some testing now anyway

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have now tested this. it works as expected with this PR changes. thanks @constanca-m for catching this

paths:
- /var/log/containers/*.log
parsers:
- container: ~
prospector:
scanner:
fingerprint.enabled: true
fingerprint.length: 300
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@constanca-m

By default the length for fingerprint is 1024, which is what is causing the error. We can reduce it to 300 (minimum possible is 64), and that solves the problem.

It's not an error, it's a warning and it's totally normal to have it until the file is large enough to be picked up. The warning needs to be there so the user knows why there are no events coming from a file yet.

Reducing the fingerprint might cause collisions in file identity, lowering the value must be done only if the user is 100% sure that the new amount of characters is always unique for each file. Please keep the value as it is now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the explanation @rdner , so the warning does not cause unexpected behavior? I will remove that line in that case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@constanca-m The warning is just to notify the user why the ingestion of the file got delayed. Perhaps we could re-phrase this warning to be more clear and less error-looking. Ideas are welcome.

It does not cause any incorrect/unexpected behaviour, it's a normal state of things.

symlinks: true
file_identity.fingerprint: ~
processors:
Expand Down
3 changes: 2 additions & 1 deletion deploy/kubernetes/filebeat/filebeat-configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,15 @@ data:
filebeat.yml: |-
filebeat.inputs:
- type: filestream
id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
id: kubernetes-container-logs
paths:
- /var/log/containers/*.log
parsers:
- container: ~
prospector:
scanner:
fingerprint.enabled: true
fingerprint.length: 300
symlinks: true
file_identity.fingerprint: ~
processors:
Expand Down
3 changes: 2 additions & 1 deletion dev-tools/kubernetes/filebeat/manifest.debug.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -113,14 +113,15 @@ data:
filebeat.yml: |-
filebeat.inputs:
- type: filestream
id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
id: kubernetes-container-logs
paths:
- /var/log/containers/*.log
parsers:
- container: ~
prospector:
scanner:
fingerprint.enabled: true
fingerprint.length: 300
symlinks: true
file_identity.fingerprint: ~
processors:
Expand Down
3 changes: 2 additions & 1 deletion dev-tools/kubernetes/filebeat/manifest.run.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -113,14 +113,15 @@ data:
filebeat.yml: |-
filebeat.inputs:
- type: filestream
id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
id: kubernetes-container-logs
paths:
- /var/log/containers/*.log
parsers:
- container: ~
prospector:
scanner:
fingerprint.enabled: true
fingerprint.length: 300
symlinks: true
file_identity.fingerprint: ~
processors:
Expand Down
Loading