sophosxg-module initial release

[StefanSa]

What does this PR do?

This PR Introduces the SophosXG filebeat module. Focusing currently on SophosXG Firewall, but should include other Sophos products as separate PR's later on.

Why is it important?

Adding more supported products to the filebeat portfolio.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

INTEGRATION_TESTS=1 BEAT_STRICT_PERMS=false TESTING_FILEBEAT_MODULES=sophosxg nosetests -v -s tests/system/test_xpack_modules.py

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[cla-checker-service]

💚 CLA has been signed

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[P1llus]

This is a PR that was planned with @StefanSa over email, we will try to collaborate and apply any changes needed before we can start a review.

Thanks a lot for the contribution!

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #19591 updated]

• Start Time: 2020-07-14T17:28:24.304+0000

• Duration: 103 min 10 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4237
Skipped	679
Total	4916

[P1llus]

cla/check

[adriansr]

jenkins, test this

[adriansr]

Jenkins, run tests please

[P1llus]

Jenkins, run tests please

[leehinman]

Awesome. a few spelling mistakes to fix and some data type suggestions.

[Bernhard-Fluehmann]

@StefanSa Thank you very much for your work. It would be great have this module available soon. I am working on a similar project for one of our customers. The difference is that it needs to support Sophos UTM, especially the SG5xx product family. Do you know if these platforms are compatible in terms of logs or if two different modules are possible? If they are different, would it be possible to merge them together into one module or would it require two separate modules?

[StefanSa]

@StefanSa Thank you very much for your work. It would be great have this module available soon. I am working on a similar project for one of our customers. The difference is that it needs to support Sophos UTM, especially the SG5xx product family. Do you know if these platforms are compatible in terms of logs or if two different modules are possible? If they are different, would it be possible to merge them together into one module or would it require two separate modules?

@Bernhard-Fluehmann
You do not only have to thank me alone, without the great support from elastic in particular from @P1llus (of course also from all the others here) it was difficult to do this.
I can't really tell you that because, i don't know the UTM syslog message.
If you would send me a few examples of UTM syslog messages, i can take a look at them.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[Bernhard-Fluehmann]

@StefanSa It seems that the structure of the logs is similar but unfortunately the log header and field names are different. Please have a look at our repo . Under x-pack you find the sophosutm module which contains the current state of our work.
Even if the platforms are different, it would still be nice if they could be part of the same module. What do you think about? E.g. use sophos as module name and have dataset xg and utm instead of firewall? or Name the datasets firewall-xg and firewall-utm. Please let me know your thoughts.

[P1llus]

@Bernhard-Fluehmann I think in the long run, most likely for a separate release, we would need to think about renaming the module to sophos, and have the filesets be called sophosxg and sophosutm.

I might be able to convert the sophosxg one to follow that convention

[Bernhard-Fluehmann]

@P1llus This is great, thanks. It seems that the official namings are is xg-firewall and sg-utm, while utm is the legacy platform and sg firewalls can be migrated to xg os.
Putting them into the same module as different datasets with own input config (like seen in the cisco module) would be great. This would allow for additional datasets like sophos email etc. in the future.
One thought about field name homologation. The most important fields are converted to ECS already, which makes both log types partially compatible. At the moment, non ECS fields would be different for both datasets. Since I expect more ECS fields to be available in the future, we can leave them alone for the moment. Since UTM seems to be legacy I can think to rename them to xg names as well if required.

BTW, I have some troubles building beats on my macbook. Since I have updated go and xcode command line tools make release works. But now the testsuite fails with the following error:
make: *** No rule to make target '../x-pack/filebeat/build/package/filebeat-darwin-amd64.dmg/beat-pkg-root/Library/Application', needed by 'filebeat.test'. Stop.
make[1]: *** [system-tests-environment] Error 2
Do you know where the problem may be?

[P1llus]

@Bernhard-Fluehmann Yeah the pointer here will be to have it similar to the other modules, where the module name is the vendor and the fileset is the product. That way we can keep the separation between XG, UTM and any other module later on.

I think when this PR is merged, you can pull the latest update from our master, add your new fileset if you want, to the sophos module (il make sure to rename the folders before merging), and create a PR adding the fileset.

In terms of building, the way its built has changed a bit maybe, depending on what you are trying to do here.

If you are in the ./beats/x-pack/filebeat directory, make sure to delete the build folder and that you have the correct go version.
Currently this is built with 1.13.X, though I think the merges happening now is also going to 1.14.X.

I would recommend checking out what can be wrong, by ensuring that you have the correct go version and grab a clone of our repo and try to run it there.

If you are merging with master from time to time there might also be that you happen to be on a broken commit, should try to rebase with the newest master as well.

[andrewkroh]

LGTM. But can you undo the go.sum changes.

[P1llus]

jenkins run tests

[adriansr]

jenkins, test this

[adriansr]

LGTM

[P1llus]

jenkins, test this