Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Add timezone config option to decode_cef and syslog input #27727

Merged

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Sep 3, 2021

What does this PR do?

CEF messages that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a timezone option to the decode_cef processor and cef module to allow the
time zone to be specified when a timestamp does not contain an offset or zone.

CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15

This also replaces the deprecated import "4d63.com/tz" with Go's relatively new built-in
time/tzdata package. The timestamp processor was updated.

While I was adding the a timezone config type I made the syslog input's timezone configurable too.

Fixes #27232

Why is it important?

Timestamps were being interpreted incorrectly.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Do a manual test of the module with netcat.

Related issues

Logs

2021-09-03T10:08:32.659-0400 DEBUG [processors] processors/processor.go:120 Generated new processors: rename=[{From:message To:event.original}], decode_cef={"Field":"event.original","TargetField":"cef","IgnoreMissing":false,"IgnoreFailure":false,"ID":"","ECS":true,"Timezone":"America/New_York"}, community_id=[target=network.community_id, fields=[source_ip=source.ip, source_port=source.port, destination_ip=destination.ip, destination_port=destination.port, transport_protocol=network.transport, icmp_type=icmp.type, icmp_code=icmp.code], seed=0], add_fields={"ecs":{"version":"1.11.0"}}

Manual test

filebeat.modules:
  - module: cef
    log:
      enabled: true
      var:
        syslog_host: localhost
        syslog_port: 9003
        timezone: America/New_York

output.console.pretty: true
{
  "@timestamp": "2021-08-04T15:31:15.000Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.0.0",
    "pipeline": "filebeat-8.0.0-cef-log-pipeline",
    "truncated": false
  },
  "input": {
    "type": "syslog"
  },
  "observer": {
    "vendor": "Aruba Networks",
    "product": "ClearPass",
    "version": "6.8.7.120583"
  },
  "message": "RADIUS Accounting",
  "ecs": {
    "version": "1.11.0"
  },
  "agent": {
    "type": "filebeat",
    "version": "8.0.0",
    "ephemeral_id": "e5b474fd-98af-44d6-9fae-a1cbdbee99f5",
    "id": "7292d176-7008-484e-b7d0-008ce53fe838",
    "name": "mac15"
  },
  "fileset": {
    "name": "log"
  },
  "cef": {
    "extensions": {
      "deviceReceiptTime": "2021-08-04T15:31:15.000Z"
    },
    "version": "0",
    "device": {
      "vendor": "Aruba Networks",
      "product": "ClearPass",
      "version": "6.8.7.120583",
      "event_class_id": "2002"
    },
    "name": "RADIUS Accounting",
    "severity": "1"
  },
  "log": {
    "source": {
      "address": "127.0.0.1:60040"
    }
  },
  "tags": [
    "cef",
    "forwarded"
  ],
  "service": {
    "type": "cef"
  },
  "event": {
    "severity": 1,
    "module": "cef",
    "dataset": "cef.log",
    "original": "CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15",
    "code": "2002"
  }
}

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 3, 2021
@botelastic
Copy link

botelastic bot commented Sep 3, 2021

This pull request doesn't have a Team:<team> label.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 3, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-09-03T18:23:58.087+0000

  • Duration: 234 min 45 sec

  • Commit: ffcfd85

Test stats 🧪

Test Results
Failed 0
Passed 53858
Skipped 5325
Total 59183

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 53858
Skipped 5325
Total 59183

@andrewkroh andrewkroh force-pushed the feature/fb/decode-cef-timezone branch 3 times, most recently from 7e55931 to e62fc7c Compare September 3, 2021 14:00
CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the
time zone to be specified when a timestamp does not contain an offset or zone.

    CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15

Note that the CEF module receives messages using the syslog input. The syslog input does not have
a configurable time zone and always assumes timestamps without time zones are given in the machine's
local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module.

This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in
`time/tzdata` package. The `timestamp` processor was updated.

Fixes elastic#27232
@andrewkroh andrewkroh marked this pull request as ready for review September 3, 2021 14:22
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@andrewkroh andrewkroh force-pushed the feature/fb/decode-cef-timezone branch from e62fc7c to c47f3c9 Compare September 3, 2021 14:23
@andrewkroh
Copy link
Member Author

run tests

@andrewkroh andrewkroh changed the title [Filebeat] Add timezone config option to decode_cef [Filebeat] Add timezone config option to decode_cef and syslog input Sep 3, 2021
@andrewkroh andrewkroh merged commit b3497ca into elastic:master Sep 7, 2021
mergify bot pushed a commit that referenced this pull request Sep 7, 2021
…27727)

CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the
time zone to be specified when a timestamp does not contain an offset or zone.

    CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15

Note that the CEF module receives messages using the syslog input. The syslog input does not have
a configurable time zone and always assumes timestamps without time zones are given in the machine's
local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module.

This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in
`time/tzdata` package. The `timestamp` processor was updated.

While I was adding the a timezone config type I made the syslog input's timezone configurable too.

Fixes #27232

(cherry picked from commit b3497ca)
ninaspitfire pushed a commit that referenced this pull request Sep 9, 2021
…27727)

CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the
time zone to be specified when a timestamp does not contain an offset or zone.

    CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15

Note that the CEF module receives messages using the syslog input. The syslog input does not have
a configurable time zone and always assumes timestamps without time zones are given in the machine's
local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module.

This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in
`time/tzdata` package. The `timestamp` processor was updated.

While I was adding the a timezone config type I made the syslog input's timezone configurable too.

Fixes #27232

(cherry picked from commit b3497ca)
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Sep 9, 2021
* master: (39 commits)
  [Heartbeat] Move JSON tests from python->go (elastic#27816)
  docs: simplify permissions for Dockerfile COPY (elastic#27754)
  Osquerybeat: Fix osquery logger plugin severy levels mapping (elastic#27789)
  [Filebeat] Update compatibility function to remove processor description on ES < 7.9.0 (elastic#27774)
  warn log entry and no validation failure when both queue_url and buck… (elastic#27612)
  libbeat/cmd/instance: ensure test config file has appropriate permissions (elastic#27178)
  [Heartbeat] Add httpcommon options to ZipURL (elastic#27699)
  Add a header round tripper option to httpcommon (elastic#27509)
  [Elastic Agent] Add validation to ensure certificate paths are absolute. (elastic#27779)
  Rename dashboards according to module.yml files for master (elastic#27749)
  Refactor vagrantfile, add scripts for provisioning with docker/kind (elastic#27726)
  Accept syslog dates with leading 0 (elastic#27775)
  [Filebeat] Add timezone config option to decode_cef and syslog input (elastic#27727)
  [Filebeat] Threatintel compatibility updates (elastic#27323)
  Add support for ephemeral containers in elastic agent dynamic provider (elastic#27707)
  [Filebeat] Integration tests in CI for AWS-S3 input (elastic#27491)
  Fix flakyness of TestFilestreamEmptyLine (elastic#27705)
  [Filebeat] kafka v2 using parsers (elastic#27335)
  Update Kafka version parsing / supported range (elastic#27720)
  Update Sarama to 1.29.1 (elastic#27717)
  ...
andrewkroh added a commit that referenced this pull request Sep 9, 2021
…27727) (#27780)

CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the
time zone to be specified when a timestamp does not contain an offset or zone.

    CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15

Note that the CEF module receives messages using the syslog input. The syslog input does not have
a configurable time zone and always assumes timestamps without time zones are given in the machine's
local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module.

This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in
`time/tzdata` package. The `timestamp` processor was updated.

While I was adding the a timezone config type I made the syslog input's timezone configurable too.

Fixes #27232

(cherry picked from commit b3497ca)

Co-authored-by: Andrew Kroh <[email protected]>
andrewkroh added a commit to andrewkroh/integrations that referenced this pull request Sep 14, 2021
Expose the `timezone` config option for the `decode_cef` processor. It is an IANA time zone or time offset
(e.g. `+0200`) to use when interpreting timestamps without a time zone in the CEF message.

Relates: elastic/beats#27727
andrewkroh added a commit to elastic/integrations that referenced this pull request Sep 27, 2021
Expose the `timezone` config option for the `decode_cef` processor. It is an IANA time zone or time offset
(e.g. `+0200`) to use when interpreting timestamps without a time zone in the CEF message.

Relates: elastic/beats#27727
Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021
…lastic#27727)

CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the
time zone to be specified when a timestamp does not contain an offset or zone.

    CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15

Note that the CEF module receives messages using the syslog input. The syslog input does not have
a configurable time zone and always assumes timestamps without time zones are given in the machine's
local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module.

This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in
`time/tzdata` package. The `timestamp` processor was updated.

While I was adding the a timezone config type I made the syslog input's timezone configurable too.

Fixes elastic#27232
eyalkraft pushed a commit to build-security/integrations that referenced this pull request Mar 30, 2022
Expose the `timezone` config option for the `decode_cef` processor. It is an IANA time zone or time offset
(e.g. `+0200`) to use when interpreting timestamps without a time zone in the CEF message.

Relates: elastic/beats#27727
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v7.16.0 Automated backport with mergify enhancement Filebeat Filebeat
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Filebeat CEF module need the option to set a timezone or an offset
3 participants