Convert Filebeat system.syslog to ECS #9135

webmat · 2018-11-16T20:32:06Z

Caveats

Syslog messages could be received from hosts and devices. Currently setting host.hostname. What should we do with device.hostname?
ECS is missing a timezone field, but should have one. Currently leaving to beats.timezone.

ECS Fields temporarily defined locally

host.hostname, added to system/_meta/fields.yml

Renames

system.syslog.hostname => host.hostname
system.syslog.program => process.name
system.syslog.pid => process.pid
system.syslog.message => message

TODO

Noticed

We're not storing event.original. Should we offer that as an option?
- Potentially a broader question than this specific module.
- Being discussed in Filebeat modules: keep raw message #8083.
There's no test for module option convert_timezone. Is there a way to add a test for this? Added to [WIP] Filebeat module issues found during ECS conversion #9208

andrewkroh · 2018-11-17T05:01:32Z

filebeat/module/system/syslog/test/darwin-syslog-sample.log-expected.json

-        "system.syslog.program": "GoogleSoftwareUpdateAgent", 
+        "message": "2016-12-13 11:35:28.421 GoogleSoftwareUpdateAgent[21412/0x700007399000] [lvl=2] -[KSUpdateEngine updateAllExceptProduct:] KSUpdateEngine updating all installed products, except:'com.google.Keystone'.", 
+        "process.name": "GoogleSoftwareUpdateAgent", 
+        "process.pid": "21412", 


IIRC process.pid needs converted to number.

I have the same issue in an other PR. I need to investigate if during the testing the ECS templates are also properly loaded.

This PR has not changed the fact that the PID was never properly cast to an int.

It's a trivial thing to fix in the grok, however. I'd be ok with fixing that in places where I see these values improperly cast. Could this introduce a breaking change, though? Visualizations are not affected by the field in _source being a string or an int, so that part should be fine. But are there other things that could be affected by these values going from a string to an int?

Actually haproxy/log has it the worst. It has a mix of int and strings, both for the PID and for the port. If this was a problem, we would have heard about it by now. I'll go around and do this small improvement on all of my open PRs.

It's a rabbit hole, though. Now I notice that IIS response code and response times are not converted to ints...

I don't think it's a grok issue here. It's more of a template issue in the tests.

No, all of these fields have proper long mappings. It really is as simple as adding :int at the end of the GROK expression, so that the value in the JSON document is a integer rather than a quoted string.

I've actually gone deep in the rabbit hole and changed all of the occurrences I saw in this PR and all of the other ones I'm working on at the moment.

webmat · 2018-11-20T13:37:06Z

@ruflin Ready for review. Make sure to check out caveats. Sorry on this app I can’t add the “review“ tag 😂

ruflin · 2018-11-20T13:42:57Z

Modified title and assigned label ;-)

filebeat/module/system/syslog/ingest/pipeline.json

dev-tools/ecs-migration.yml

filebeat/module/system/syslog/ingest/pipeline.json

ruflin · 2018-11-20T13:47:43Z

I think this one is ready to go. We can do further adjustments in follow up PR's.

webmat

Test failure was a random network hiccup. Will re-trigger the build by pushing the deletion of system.syslog.timestamp and rebasing. Should be good for merging after that.

dev-tools/ecs-migration.yml

filebeat/module/system/syslog/ingest/pipeline.json

ruflin

For the caveats:

Let's go with host.hostname for now. I don't think we can tell if it comes from a device or host
I'm good with keeping beat.timezone for now until we figured out things in ECS. Let's add this to a "todo" list somewhere so we don't forget
event.original: Lets handle this in Filebeat modules: keep raw message #8083 as proposed, not here.

webmat · 2018-11-21T20:09:20Z

@ruflin Trying to add field aliases towards their ECS equivalent seems not to make these fields into aliases in the mapping. It also does the wrong thing in the documentation generated.

Steps to confirming the mapping:

Rebuild make update && make filebeat.test && make filebeat
Clean up to be extra sure: DELETE test-filebeat-modules
Run system/syslog tests INTEGRATION_TESTS=1 GENERATE=1 TESTING_FILEBEAT_MODULES=system nosetests tests/system/test_modules.py
Check the mapping again GET test-filebeat-modules/_mapping

Result: I'm seeing all of the aliased fields as normal keyword fields, not field aliases. Do you know what could be causing this? Is there an issue with the way I specify my aliases: commit?

Also, check out how all of the system.syslog fields I'm aliasing get replaced directly with their ECS equivalent here, that doesn't seem ok.

My take would be to

Do the declarative part anyway. All field declarations that should be aliases are made so right now, and we merge as is
Open issue to get the aliases to make it to the mapping (unless I'm doing something wrong)
Open issue to generate correct documentation for alias fields.

Perhaps should now simply say "deprecated field, please use X in the future. This od field name will remain as an alias until version 8, and then be removed" or something to that effect

webmat · 2018-11-22T18:05:00Z

Only failure appears to be with this flaky test: #9207

ruflin

I opened a PR to skip the failing test. Ready to be merged.

- system.syslog.hostname => host.hostname - system.syslog.program => process.name - system.syslog.pid => process.pid - system.syslog.message => message

ruflin mentioned this pull request Nov 16, 2018

Transition Beats to ECS #8655

Closed

webmat self-assigned this Nov 16, 2018

webmat added in progress Pull request is currently in progress. module Filebeat Filebeat ecs labels Nov 16, 2018

webmat force-pushed the ecs-system-syslog branch from ba62d3c to c4f68a1 Compare November 16, 2018 20:46

andrewkroh reviewed Nov 17, 2018

View reviewed changes

webmat force-pushed the ecs-system-syslog branch from c4f68a1 to 40d8ee1 Compare November 20, 2018 04:24

webmat removed the in progress Pull request is currently in progress. label Nov 20, 2018

ruflin changed the title ~~WIP Convert Filebeat system.syslog to ECS~~ Convert Filebeat system.syslog to ECS Nov 20, 2018

ruflin added the review label Nov 20, 2018

ruflin approved these changes Nov 20, 2018

View reviewed changes

filebeat/module/system/syslog/ingest/pipeline.json Show resolved Hide resolved

dev-tools/ecs-migration.yml Show resolved Hide resolved

filebeat/module/system/syslog/ingest/pipeline.json Show resolved Hide resolved

webmat commented Nov 20, 2018

View reviewed changes

dev-tools/ecs-migration.yml Show resolved Hide resolved

filebeat/module/system/syslog/ingest/pipeline.json Show resolved Hide resolved

webmat force-pushed the ecs-system-syslog branch from 40d8ee1 to 57078b7 Compare November 20, 2018 16:17

ruflin approved these changes Nov 21, 2018

View reviewed changes

webmat force-pushed the ecs-system-syslog branch from 57078b7 to b3ebfe6 Compare November 21, 2018 20:00

webmat mentioned this pull request Nov 22, 2018

Convert Filebeat haproxy.log to ECS #9117

Merged

7 tasks

webmat force-pushed the ecs-system-syslog branch from b3ebfe6 to edc9cf7 Compare November 22, 2018 15:34

ruflin approved these changes Nov 22, 2018

View reviewed changes

Mathieu Martin added 5 commits November 22, 2018 13:49

Space. The final frontier.

58a9c80

Add missing ECS host.hostname temporarily

6fff947

Perform all the straightforward renames at once.

bfcc341

- system.syslog.hostname => host.hostname - system.syslog.program => process.name - system.syslog.pid => process.pid - system.syslog.message => message

Convert PID to an int

69f61b4

Shove all renames so far in ecs-migration.yml

1b5ba4a

Mathieu Martin added 3 commits November 22, 2018 13:49

Changelog

7e99526

Remove system.syslog.timestamp

712d3bd

Attempt at aliasing system.syslog fields to their ECS counterpart.

dc67401

webmat force-pushed the ecs-system-syslog branch from edc9cf7 to dc67401 Compare November 22, 2018 18:50

webmat merged commit ff83dcf into elastic:master Nov 22, 2018

webmat deleted the ecs-system-syslog branch November 22, 2018 18:58

webmat mentioned this pull request Dec 17, 2018

Fix recently translated Filebeat modules where integer coercions were added #9598

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert Filebeat system.syslog to ECS #9135

Convert Filebeat system.syslog to ECS #9135

webmat commented Nov 16, 2018 •

edited

Loading

andrewkroh Nov 17, 2018

ruflin Nov 19, 2018

webmat Nov 19, 2018

webmat Nov 19, 2018 •

edited

Loading

webmat Nov 19, 2018 •

edited

Loading

ruflin Nov 20, 2018

webmat Nov 20, 2018

webmat commented Nov 20, 2018

ruflin commented Nov 20, 2018

ruflin commented Nov 20, 2018

webmat left a comment

ruflin left a comment

webmat commented Nov 21, 2018 •

edited

Loading

webmat commented Nov 22, 2018

ruflin left a comment

Convert Filebeat system.syslog to ECS #9135

Convert Filebeat system.syslog to ECS #9135

Conversation

webmat commented Nov 16, 2018 • edited Loading

andrewkroh Nov 17, 2018

Choose a reason for hiding this comment

ruflin Nov 19, 2018

Choose a reason for hiding this comment

webmat Nov 19, 2018

Choose a reason for hiding this comment

webmat Nov 19, 2018 • edited Loading

Choose a reason for hiding this comment

webmat Nov 19, 2018 • edited Loading

Choose a reason for hiding this comment

ruflin Nov 20, 2018

Choose a reason for hiding this comment

webmat Nov 20, 2018

Choose a reason for hiding this comment

webmat commented Nov 20, 2018

ruflin commented Nov 20, 2018

ruflin commented Nov 20, 2018

webmat left a comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

webmat commented Nov 21, 2018 • edited Loading

webmat commented Nov 22, 2018

ruflin left a comment

Choose a reason for hiding this comment

webmat commented Nov 16, 2018 •

edited

Loading

webmat Nov 19, 2018 •

edited

Loading

webmat Nov 19, 2018 •

edited

Loading

webmat commented Nov 21, 2018 •

edited

Loading