[Filebeat] Fix RFC5424 date format in system/syslog in pipeline #12529
Conversation
The date format used by the system/syslog pipeline has a couple of issues: - Since Elasticsearch switched to Java time from Joda time, it won't correctly parse timezone offsets that use a colon to separate hours and minutes. - RFC5424 allows for 1 to 6 digits for the fractional second. This updates the pipeline to also accept 3 decimals (milliseconds).
adriansr
added
the
needs_backport
| delete_key(obj, "@timestamp") | ||
|
|
||
| # Remove timestamp from syslog tests except for the log file that needs it. | ||
| if (obj["event.dataset"] == "system.syslog" and filename != "tz-offset.log"): | ||
| delete_key(obj, "@timestamp") | ||
|
|
There was a problem hiding this comment.
its a bit hacky but I think for now we don't have choice.
There was a problem hiding this comment.
Definitely hacky :)
Note to reviewers: This is needed because in most syslog logs, the year is missing from the timestamp, so Elasticsearch will set it to the current year and that will cause the tests to fail every new year.
There was a problem hiding this comment.
Yeah should the explanation you gave be in the comment in the code? This is hacky enough after a while we probably won't remember why it's added 😄(If there is no better solution.)
There was a problem hiding this comment.
should the explanation you gave be in the comment in the code?
Right!
|
@ph @kaiyan-sheng Turns out that in order for it to work, the hack needs to be a little uglier. Are you still OK with it? (see the latest commits). Otherwise, we need to find out a more elegant way of having tests keep the timestamp selectively. |
…stic#12529) The date format used by the system/syslog pipeline has a couple of issues: - Since Elasticsearch switched to Java time from Joda time, it won't correctly parse timezone offsets that use a colon to separate hours and minutes. - RFC5424 allows for 1 to 6 digits for the fractional second. This updates the pipeline to also accept 3 decimals (milliseconds). (cherry picked from commit d30b4dc)
adriansr
added
v7.2.0
and removed
needs_backport
) (#12543) The date format used by the system/syslog pipeline has a couple of issues: - Since Elasticsearch switched to Java time from Joda time, it won't correctly parse timezone offsets that use a colon to separate hours and minutes. - RFC5424 allows for 1 to 6 digits for the fractional second. This updates the pipeline to also accept 3 decimals (milliseconds). (cherry picked from commit d30b4dc)
…stic#12529) (elastic#12543) The date format used by the system/syslog pipeline has a couple of issues: - Since Elasticsearch switched to Java time from Joda time, it won't correctly parse timezone offsets that use a colon to separate hours and minutes. - RFC5424 allows for 1 to 6 digits for the fractional second. This updates the pipeline to also accept 3 decimals (milliseconds). (cherry picked from commit dd4d52f)
The date format used by the system/syslog pipeline has a couple of issues with RFC5424 messages:
Since Elasticsearch switched from Joda time to Java time, it won't correctly parse timezone offsets that use a colon to separate hours and minutes.
RFC5424 allows for 1 to 6 digits for the fractional second. This updates the pipeline to also accept 3 decimals (milliseconds).