Integrations that parse timestamps without a year fail after new year

[jsoriano]

Some integrations have started failing after the change of year. These integrations are likely parsing timestamps without a year.

These tests should be fixed by checking first if the year is actually not found in the logs, or if there is some parsing issue.

You can find the failing packages for example in these builds:

• https://fleet-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/main/632/tests
• https://fleet-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations%2FPR-4940/detail/PR-4940/1/tests

Pinging owners of affected packages: @elastic/security-external-integrations @elastic/obs-service-integrations @elastic/elastic-agent-data-plane

cc @elastic/ecosystem

[ebeahan]

Looking at a handful of the SEI-owned packages, it's because the original logs don't include a year at all in the timestamp. This is expected for BSD/RFC 3164 syslog message timestamps (Mmm dd hh:mm:ss).

@jsoriano any suggestions of how to address? Have we ever hit this issue with other integrations in the past?

[jsoriano]

@jsoriano any suggestions of how to address? Have we ever hit this issue with other integrations in the past?

Hey @ebeahan,

After confirming that the source log/event doesn't contain a year, I think there are a couple of options in the test configuration:

• Hard-code a timestamp, as is done for example in

  integrations/packages/zeek/data_stream/dhcp/_dev/test/pipeline/test-dhcp.log-config.yml

  Line 2 in 27f5518

  "@timestamp": "2020-04-28T11:07:58.223Z"

• Define a pattern to validate the timestamp, as is done for example in

  integrations/packages/cyberarkpas/data_stream/audit/_dev/test/pipeline/test-common-config.yml

  Line 2 in 27f5518

  "@timestamp": "^[0-9]{4}(-[0-9]{2}){2}T[0-9]{2}(:[0-9]{2}){2}\\.[0-9]{3}Z$"

Not sure though if the second option will solve this problem.

Another option is to just regenerate the expected results, but they will fail again next year 🙂

[narph]

Define a pattern to validate the timestamp, as is done for example in
integrations/packages/cyberarkpas/data_stream/audit/_dev/test/pipeline/test-common-config.yml

Line 2 in 27f5518

"@timestamp": "^[0-9]{4}(-[0-9]{2}){2}T[0-9]{2}(:[0-9]{2}){2}\.[0-9]{3}Z$"
Not sure though if the second option will solve this problem.

pfsense is one of the affected packages

pfsense/log test-pfsense-unbound.log:
--- want
+++ got
@@ -1,7 +1,7 @@
 {
     "expected": [
         {
-            "@timestamp": "2022-08-15T16:19:02.000-04:00",
+            "@timestamp": "2023-08-15T16:19:02.000-04:00",
             "client": {
                 "address": "192.168.1.1",
                 "ip": "192.168.1.1"

╭─────────┬─────────────┬───────────┬──────────────────────────┬─────────────────────────────────────────────────────────────────────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT
                 │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────┼─────────────────────────────────────────────────────────────────────────┼──────────────┤
│ pfsense │ log         │ pipeline  │ test-opensense.log       │ FAIL: test case failed: Expected results are different from actual ones │    39.8652ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-bsd.log     │ FAIL: test case failed: Expected results are different from actual ones │    13.0515ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-dhcp.log    │ FAIL: test case failed: Expected results are different from actual ones │    17.1696ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-haproxy.log │ FAIL: test case failed: Expected results are different from actual ones │    14.0588ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-ipsec.log   │ PASS
                 │    11.4619ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-openvpn.log │ FAIL: test case failed: Expected results are different from actual ones │    10.0666ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-phpfpm.log  │ FAIL: test case failed: Expected results are different from actual ones │       9.43ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-squid.log   │ FAIL: test case failed: Expected results are different from actual ones │     7.2031ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-syslog.log  │ PASS
                 │    10.8438ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-unbound.log │ FAIL: test case failed: Expected results are different from actual ones │    10.0712ms │
╰─────────┴─────────────┴───────────┴──────────────────────────┴─────────────────────────────────────────────────────────────────────────┴──────────────╯

I've added:

dynamic_fields:
  "@timestamp": "^[0-9]{4}(-[0-9]{2}){2}T[0-9]{2}(:[0-9]{2}){2}\\.[0-9]{3}"

to https://github.com/elastic/integrations/blob/main/packages/pfsense/data_stream/log/_dev/test/pipeline/test-common-config.yml

and tests passed

--- Test results for package: pfsense - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ pfsense │ log         │ pipeline  │ test-opensense.log       │ PASS   │     36.612ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-bsd.log     │ PASS   │    16.5897ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-dhcp.log    │ PASS   │    15.7219ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-haproxy.log │ PASS   │    12.9254ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-ipsec.log   │ PASS   │    14.0902ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-openvpn.log │ PASS   │    12.6828ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-phpfpm.log  │ PASS   │    10.6016ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-squid.log   │ PASS   │     8.8069ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-syslog.log  │ PASS   │    11.4988ms │
│ pfsense │ log         │ pipeline  │ test-pfsense-unbound.log │ PASS   │    10.3019ms │
╰─────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: pfsense - END   ---

@jsoriano, @andrewkroh should we go with the second option for the security packages?

[andrewkroh]

Using a dynamic field to handle the year SGTM.

[jsoriano]

Yeah, sounds good, do you have a PR with the change for pfsense?

[narph]

Yeah, sounds good, do you have a PR with the change for pfsense?

we'll soon link a pr with changes for all security packages

[ebeahan]

Security external integrations have updated our integrations: #4984

[jsoriano]

Thanks!

[jsoriano]

It looks like all related issues have been fixed. Thanks all!