Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add RFC5424 format support for syslog input #20246

Closed
wants to merge 11 commits into from
Closed

Add RFC5424 format support for syslog input #20246

wants to merge 11 commits into from

Conversation

wph95
Copy link
Contributor

@wph95 wph95 commented Jul 27, 2020 •
edited
Loading

What does this PR do?

Resolves #6872
Add RFC5424 format support for syslog input

Why is it important?

Syslog input only support rfc3164, btw rfc3164 is obsoleted by rfc5424

More and more software is using rfc5424 instead of rfc3164.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Syslog
    • inputs config support RFC5424
    • detect format automatically
  • Ragel
    • HEADER
      • PRI
      • VERSION
      • TIMESTAMP
      • HOSTNAME
      • APP-NAME
      • PROCID
      • MSGID
    • STRUCTURED-DATA
      • SD-ELEMENT
      • SD-ID
      • SD-PARAM
      • Change Control
    • MSG
  • TEST
    • all examples in RFC5424 document
    • Some special boundary case tests

How to test this PR locally

Related issues

closes # #6872

Use cases

Screenshots

Logs

@wph95
- add PRI schema
d8d403d 
- add VERSION schema
@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

1 similar comment
@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 27, 2020
@cla-checker-service
Copy link

cla-checker-service bot commented Jul 27, 2020 •
edited
Loading

💚 CLA has been signed

@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 27, 2020 •
edited by jenkins-beats-ci bot
Loading

❕ Build Aborted

The PR is not allowed to run in the CI yet

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20246 updated]

  • Reason: The PR is not allowed to run in the CI yet

  • Start Time: 2020-08-07T11:58:27.089+0000

  • Duration: 4 min 32 sec

  • Commit: 7b26515

Steps errors

Expand to view the steps failures

  • Name: Error signal

    • Description: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permission

    • Duration: 0 min 0 sec

    • Start Time: 2020-08-07T12:01:53.649+0000

    • log

Log output

Expand to view the last 100 lines of log output 

[2020-08-07T12:01:56.107Z] Stage "Elastic Agent x-pack" skipped due to earlier failure(s)
[2020-08-07T12:01:56.109Z] Stage "Elastic Agent x-pack Windows" skipped due to earlier failure(s)
[2020-08-07T12:01:56.110Z] Stage "Elastic Agent Mac OS X" skipped due to earlier failure(s)
[2020-08-07T12:01:56.111Z] Stage "Filebeat oss" skipped due to earlier failure(s)
[2020-08-07T12:01:56.112Z] Stage "Filebeat x-pack" skipped due to earlier failure(s)
[2020-08-07T12:01:56.113Z] Stage "Filebeat Mac OS X" skipped due to earlier failure(s)
[2020-08-07T12:01:56.114Z] Stage "Filebeat x-pack Mac OS X" skipped due to earlier failure(s)
[2020-08-07T12:01:56.116Z] Stage "Filebeat Windows" skipped due to earlier failure(s)
[2020-08-07T12:01:56.117Z] Stage "Filebeat x-pack Windows" skipped due to earlier failure(s)
[2020-08-07T12:01:56.118Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:56.119Z] Stage "Auditbeat oss Linux" skipped due to earlier failure(s)
[2020-08-07T12:01:56.120Z] Stage "Auditbeat crosscompile" skipped due to earlier failure(s)
[2020-08-07T12:01:56.121Z] Stage "Auditbeat oss Mac OS X" skipped due to earlier failure(s)
[2020-08-07T12:01:56.122Z] Stage "Auditbeat oss Windows" skipped due to earlier failure(s)
[2020-08-07T12:01:56.123Z] Stage "Auditbeat x-pack" skipped due to earlier failure(s)
[2020-08-07T12:01:56.125Z] Stage "Auditbeat x-pack Mac OS X" skipped due to earlier failure(s)
[2020-08-07T12:01:56.126Z] Stage "Auditbeat x-pack Windows" skipped due to earlier failure(s)
[2020-08-07T12:01:56.127Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:56.127Z] Stage "Libbeat x-pack" skipped due to earlier failure(s)
[2020-08-07T12:01:56.128Z] Stage "Metricbeat OSS Unit tests" skipped due to earlier failure(s)
[2020-08-07T12:01:56.129Z] Stage "Metricbeat OSS Go Integration tests" skipped due to earlier failure(s)
[2020-08-07T12:01:56.131Z] Stage "Metricbeat OSS Python Integration tests" skipped due to earlier failure(s)
[2020-08-07T12:01:56.132Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-08-07T12:01:56.133Z] Stage "Metricbeat crosscompile" skipped due to earlier failure(s)
[2020-08-07T12:01:56.134Z] Stage "Metricbeat Mac OS X" skipped due to earlier failure(s)
[2020-08-07T12:01:56.135Z] Stage "Metricbeat x-pack Mac OS X" skipped due to earlier failure(s)
[2020-08-07T12:01:56.136Z] Stage "Metricbeat Windows" skipped due to earlier failure(s)
[2020-08-07T12:01:56.138Z] Stage "Metricbeat x-pack Windows" skipped due to earlier failure(s)
[2020-08-07T12:01:56.139Z] Stage "Packetbeat OSS" skipped due to earlier failure(s)
[2020-08-07T12:01:56.140Z] Stage "dockerlogbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:56.140Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:56.141Z] Stage "Winlogbeat Windows x-pack" skipped due to earlier failure(s)
[2020-08-07T12:01:56.142Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:56.143Z] Stage "Journalbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:56.144Z] Stage "Generators" skipped due to earlier failure(s)
[2020-08-07T12:01:56.144Z] Stage "Kubernetes" skipped due to earlier failure(s)
[2020-08-07T12:01:56.239Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:56.241Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:56.241Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-08-07T12:01:56.243Z] Stage "Packetbeat OSS" skipped due to earlier failure(s)
[2020-08-07T12:01:56.244Z] Stage "dockerlogbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:56.245Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:56.246Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:56.247Z] Stage "Journalbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:56.248Z] Stage "Generators" skipped due to earlier failure(s)
[2020-08-07T12:01:56.782Z] Failed in branch Elastic Agent x-pack
[2020-08-07T12:01:56.783Z] Failed in branch Elastic Agent x-pack Windows
[2020-08-07T12:01:56.784Z] Failed in branch Elastic Agent Mac OS X
[2020-08-07T12:01:56.785Z] Failed in branch Filebeat oss
[2020-08-07T12:01:56.785Z] Failed in branch Filebeat x-pack
[2020-08-07T12:01:56.786Z] Failed in branch Filebeat Mac OS X
[2020-08-07T12:01:56.786Z] Failed in branch Filebeat x-pack Mac OS X
[2020-08-07T12:01:56.787Z] Failed in branch Filebeat Windows
[2020-08-07T12:01:56.787Z] Failed in branch Filebeat x-pack Windows
[2020-08-07T12:01:56.788Z] Failed in branch Auditbeat oss Linux
[2020-08-07T12:01:56.788Z] Failed in branch Auditbeat crosscompile
[2020-08-07T12:01:56.789Z] Failed in branch Auditbeat oss Mac OS X
[2020-08-07T12:01:56.789Z] Failed in branch Auditbeat oss Windows
[2020-08-07T12:01:56.790Z] Failed in branch Auditbeat x-pack
[2020-08-07T12:01:56.791Z] Failed in branch Auditbeat x-pack Mac OS X
[2020-08-07T12:01:56.791Z] Failed in branch Auditbeat x-pack Windows
[2020-08-07T12:01:56.792Z] Failed in branch Libbeat x-pack
[2020-08-07T12:01:56.792Z] Failed in branch Metricbeat OSS Unit tests
[2020-08-07T12:01:56.793Z] Failed in branch Metricbeat OSS Go Integration tests
[2020-08-07T12:01:56.793Z] Failed in branch Metricbeat OSS Python Integration tests
[2020-08-07T12:01:56.794Z] Failed in branch Metricbeat crosscompile
[2020-08-07T12:01:56.794Z] Failed in branch Metricbeat Mac OS X
[2020-08-07T12:01:56.795Z] Failed in branch Metricbeat x-pack Mac OS X
[2020-08-07T12:01:56.796Z] Failed in branch Metricbeat Windows
[2020-08-07T12:01:56.796Z] Failed in branch Metricbeat x-pack Windows
[2020-08-07T12:01:56.797Z] Failed in branch Winlogbeat Windows x-pack
[2020-08-07T12:01:56.797Z] Failed in branch Kubernetes
[2020-08-07T12:01:57.058Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:57.060Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:57.061Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-08-07T12:01:57.062Z] Stage "Packetbeat OSS" skipped due to earlier failure(s)
[2020-08-07T12:01:57.062Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:57.063Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:57.064Z] Stage "Generators" skipped due to earlier failure(s)
[2020-08-07T12:01:57.104Z] Failed in branch dockerlogbeat
[2020-08-07T12:01:57.105Z] Failed in branch Journalbeat
[2020-08-07T12:01:57.469Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:57.471Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:57.472Z] Stage "Packetbeat OSS" skipped due to earlier failure(s)
[2020-08-07T12:01:57.473Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-08-07T12:01:57.474Z] Stage "Generators" skipped due to earlier failure(s)
[2020-08-07T12:01:57.518Z] Failed in branch Metricbeat x-pack
[2020-08-07T12:01:57.519Z] Failed in branch Winlogbeat
[2020-08-07T12:01:57.729Z] Failed in branch Heartbeat
[2020-08-07T12:01:57.730Z] Failed in branch Libbeat
[2020-08-07T12:01:57.731Z] Failed in branch Packetbeat OSS
[2020-08-07T12:01:57.731Z] Failed in branch Functionbeat
[2020-08-07T12:01:57.732Z] Stage "Generators" skipped due to earlier failure(s)
[2020-08-07T12:01:57.806Z] Failed in branch Generators
[2020-08-07T12:01:58.342Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-20246
[2020-08-07T12:01:58.581Z] [INFO] getVaultSecret: Getting secrets
[2020-08-07T12:01:58.667Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-08-07T12:01:59.403Z] + chmod 755 generate-build-data.sh
[2020-08-07T12:01:59.403Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20246/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20246/runs/9 ABORTED 212053
[2020-08-07T12:01:59.403Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20246/runs/9/steps/?limit=10000 -o steps-info.json

wph95 added 3 commits July 28, 2020 00:22
@wph95
- make test work
f1d2dc3 
- clean code
- change parser.go to parser/parser_rfc5424
@wph95
- Add TIMESTAMP format
eef6387
@wph95
- Add "AppName" format
e395b7b 
- Add "ProcID" format
- Add "MsgID" format
- Add "HostName" format
- improve test
@andresrc andresrc added the Team:Services (Deprecated) Label for the former Integrations-Services team label Jul 28, 2020
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 28, 2020
@andresrc
Copy link
Contributor

Related: #15467

@wph95 wph95 marked this pull request as ready for review July 28, 2020 11:56
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@wph95
- syslog input config support rfc5424
147fdad 
- add auto detect format
@wph95
Copy link
Contributor Author

wph95 commented Jul 28, 2020 •
edited
Loading

Major code logic and core test completed
Looking forward to the code review :)
I'll also keep updating it to add more tests (some boundary tests for rfc5424)

@kvch kvch self-requested a review July 28, 2020 12:48
@botelastic
Copy link

botelastic bot commented Sep 6, 2020

Hi!
We just realized that we haven't looked into this PR in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it in as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Sep 6, 2020
@botelastic
Copy link

botelastic bot commented Oct 6, 2020

Hi!
This PR has been stale for a while and we're going to close it as part of our cleanup procedure.
We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team.
Feel free to re-open this PR if you think it should stay open and is worth rebasing.
Thank you for your contribution!

@botelastic botelastic bot closed this Oct 6, 2020
@wph95 wph95 mentioned this pull request Dec 25, 2020
Closed
@raciasolvo
Copy link

@wph95 Hi! Could you re-open PR, please?

@wph95 wph95 mentioned this pull request Feb 10, 2021
Merged
27 tasks
@faec faec mentioned this pull request Apr 15, 2021
Closed
27 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kvch kvch Awaiting requested review from kvch

Assignees
No one assigned
Labels
Stalled Team:Services (Deprecated) Label for the former Integrations-Services team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Syslog input to support RFC5424
4 participants
@wph95 @elasticmachine @andresrc @raciasolvo