Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tag truncated events by max_bytes #7022

Closed
ppf2 opened this issue May 4, 2018 · 0 comments
Closed

Tag truncated events by max_bytes #7022

ppf2 opened this issue May 4, 2018 · 0 comments
Assignees
@kvch
Labels
enhancement Filebeat Filebeat

Comments

@ppf2
Copy link
Member

ppf2 commented May 4, 2018

While max_bytes is a useful feature to protect against unexpected large messages, it does its job silently today. There are no logging messages indicating a message being truncated, and the resulting published event also has no tagging indicating that the event has been truncated. If a user looks at the resulting event, they may think that the truncated version is really the original message. It will be helpful for us to add a tag to all the events that have been truncated via max_bytes so it is easier from an auditing point of view.
@ph ph assigned ph and unassigned ph May 5, 2018
@ppf2 ppf2 changed the title Add a tag when messages are truncated via max_bytes Better debugging for max_bytes settings May 8, 2018
@ppf2 ppf2 changed the title Better debugging for max_bytes settings Tag truncated events by max_bytes May 10, 2018
@kvch kvch self-assigned this Aug 16, 2018
@kvch kvch mentioned this issue Aug 16, 2018
Merged
kvch added a commit that referenced this issue Aug 30, 2018
@kvch
Add tag "truncated" to "log.flags" if incoming line is longer than co…
0884236 
…nfigured limit (#7991)

A new field is added to store the flags of an event named "log.flags".
If a message is truncated, "truncated" flag is added to the list.

Example event with "truncated" flag:

{
  "@timestamp": "2018-08-16T13:00:46.759Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "host": {
    "name": "sleipnir"
  },
  "source": "/home/n/test.log",
  "offset": 33,
  "log": {
    "flags": [
       "truncated"
    ],
  },
  "message": "test line",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "hostname": "sleipnir",
    "version": "7.0.0-alpha1",
    "name": "sleipnir"
  }
}

Closes #7022
@kvch kvch mentioned this issue Aug 30, 2018
Merged
kvch added a commit to kvch/beats that referenced this issue Aug 31, 2018
@kvch
Add tag "truncated" to "log.flags" if incoming line is longer than co…
06cc06e 
…nfigured limit (elastic#7991)

A new field is added to store the flags of an event named "log.flags".
If a message is truncated, "truncated" flag is added to the list.

Example event with "truncated" flag:

{
  "@timestamp": "2018-08-16T13:00:46.759Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "host": {
    "name": "sleipnir"
  },
  "source": "/home/n/test.log",
  "offset": 33,
  "log": {
    "flags": [
       "truncated"
    ],
  },
  "message": "test line",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "hostname": "sleipnir",
    "version": "7.0.0-alpha1",
    "name": "sleipnir"
  }
}

Closes elastic#7022
(cherry picked from commit 0884236)
kvch added a commit that referenced this issue Sep 3, 2018
@kvch
Cherry-pick #7991 to 6.x: Add tag "truncated" to "log.flags" if incom…
74ac00e 
…ing line is longer than configured limit (#8165)

* Add tag "truncated" to "log.flags" if incoming line is longer than configured limit (#7991)

A new field is added to store the flags of an event named "log.flags".
If a message is truncated, "truncated" flag is added to the list.

Example event with "truncated" flag:

{
  "@timestamp": "2018-08-16T13:00:46.759Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "host": {
    "name": "sleipnir"
  },
  "source": "/home/n/test.log",
  "offset": 33,
  "log": {
    "flags": [
       "truncated"
    ],
  },
  "message": "test line",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "hostname": "sleipnir",
    "version": "7.0.0-alpha1",
    "name": "sleipnir"
  }
}

Closes #7022
(cherry picked from commit 0884236)

* fix changelog && rebase
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kvch kvch

Labels
enhancement Filebeat Filebeat
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ph @kvch @ppf2