[Filebeat][Checkpoint module] field [@timestamp] already exists

[Bernhard-Fluehmann]

One of a customer reported the following error when using the filebeat checkpoint module:
error.message: field [@timestamp] already exists

This seems to be related to how the ingest pipeline sets the timestamps, especially when the log contains a checkpoint.time field. Since none of the logs of test/checkpoint.log contains this field, the problem may not appear in the unit tests.

https://github.com/RealStuff/beats/tree/checkpoint-fix contains a fix for the problem.
Basically it renames the @timestamp field created by filebeat to event.created. In addition, rename processors were replaced by date processors to create the final @timestamp field.

Please let me know what you think about this fix. If you find it usable, I will create a pull request.

For confirmed bugs, please report:

• Version: 7.8
• Operating System: CentOS
• Discuss Forum URL:
• Steps to Reproduce: Send checkpoint log containing time field to filebeat checkpoint module.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[hazcod]

I am seeing this on my fresh elastic 7.8 cluster too. @Bernhard-Fluehmann thank you for the fork, can you please open up a PR?

[P1llus]

I will try to take a look at this as well, timestamp should have been either overwritten or grabbed from the syslog header, but might be some niche cases.

[Bernhard-Fluehmann]

@hazcod Thank you for your report
@P1llus I have just created a pull-request with a fix and description of the problem. Please let me know if you need more information

[bevano8]

@Bernhard-Fluehmann Will this be fixed in the next filebeat release? Whether its 7.8.2 or 7.9?

[Bernhard-Fluehmann]

@bevano8 I hope in 7.9 but it depends on Elastic