Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pkg/stanza] Fix serverity range unmarshaling #16339

Merged
merged 1 commit into from
Nov 17, 2022

Conversation

djaglowski
Copy link
Member

This functionality was broken during refactoring of the unmarshaling logic.

Previously, various yaml types could be used to represent a severity level. This was designed to be as open as possible to future implementations. However, it is now clear that only some keys are valid anyways. Namely, string and ints that represent OTel's native severity levels. Both strings and ints are now treated as strings when unmarshaling, which is fully compatible with the sev map lookup code.

Resolves #16336

@djaglowski djaglowski force-pushed the pkg-stanza-sev-range-fix branch 2 times, most recently from dffbc96 to 7e679a3 Compare November 17, 2022 14:28
This functionality was broken during refactoring of the unmarshaling logic.

Previously, various yaml types could be used to represent a severity level.
This was designed to be as open as possible to future implementations. However,
it is now clear that only some keys are valid anyways. Namely, string and ints
that represent OTel's native severity levels. Both strings and ints are now treated
as strings when unmarshaling, which is fully compatible with the sev map lookup code.
@djaglowski djaglowski force-pushed the pkg-stanza-sev-range-fix branch from 7e679a3 to 8648bd2 Compare November 17, 2022 15:55
@djaglowski djaglowski marked this pull request as ready for review November 17, 2022 16:31
@djaglowski djaglowski requested review from a team, TylerHelmuth and schmikei November 17, 2022 16:31
Copy link
Contributor

@schmikei schmikei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can confirm this works for me!

stdout
➜ ./bin/otelcontribcol_darwin_amd64 --config common_event_format_logs.yaml
2022/11/17 11:38:59 proto: duplicate proto type registered: jaeger.api_v2.PostSpansRequest
2022/11/17 11:38:59 proto: duplicate proto type registered: jaeger.api_v2.PostSpansResponse
2022-11-17T11:38:59.795-0500	info	service/telemetry.go:110	Setting up own telemetry...
2022-11-17T11:38:59.795-0500	info	service/telemetry.go:140	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
2022-11-17T11:38:59.795-0500	info	components/components.go:30	In development component. May change in the future.	{"kind": "exporter", "data_type": "logs", "name": "logging", "stability": "in development"}
2022-11-17T11:38:59.796-0500	info	service/service.go:89	Starting otelcontribcol...	{"Version": "v0.64.0-66-g8648bd23e2", "NumCPU": 12}
2022-11-17T11:38:59.796-0500	info	extensions/extensions.go:41	Starting extensions...
2022-11-17T11:38:59.796-0500	info	pipelines/pipelines.go:74	Starting exporters...
2022-11-17T11:38:59.796-0500	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "logs", "name": "logging"}
2022-11-17T11:38:59.796-0500	info	pipelines/pipelines.go:82	Exporter started.	{"kind": "exporter", "data_type": "logs", "name": "logging"}
2022-11-17T11:38:59.796-0500	info	pipelines/pipelines.go:86	Starting processors...
2022-11-17T11:38:59.796-0500	info	pipelines/pipelines.go:98	Starting receivers...
2022-11-17T11:38:59.796-0500	info	pipelines/pipelines.go:102	Receiver is starting...	{"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-11-17T11:38:59.796-0500	info	adapter/receiver.go:55	Starting stanza receiver	{"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-11-17T11:38:59.796-0500	info	pipelines/pipelines.go:106	Receiver started.	{"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-11-17T11:38:59.796-0500	info	service/service.go:106	Everything is ready. Begin running and processing data.
2022-11-17T11:38:59.997-0500	info	fileconsumer/file.go:159	Started watching file	{"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/Users/keithschmitt/cel.log"}
2022-11-17T11:39:00.096-0500	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "#logs": 4}
config
receivers:
  filelog:
    include: [/Users/keithschmitt/cel.log]
    start_at: beginning
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2})\s+((?P<hostname>[^\s]+)\s+)?(?P<cef_headers>[\d\D]+)'
        timestamp:
          parse_from: attributes.timestamp
          layout_type: gotime
          layout: 'Jan 02 15:04:05'
          location: UTC

      - type: csv_parser
        parse_from: attributes.cef_headers
        header: 'version|device_vendor|device_product|device_version|signature_id|name|severity|extensions'
        delimiter: "|"

      - type: regex_parser
        if: 'attributes.version != nil and attributes.version matches "CEF:[^\\|]*"'
        regex: 'CEF:(?P<version>[^\|]*)'
        parse_from: attributes.version

      - type: severity_parser
        if: 'attributes.severity != nil'
        parse_from: attributes.severity
        preset: none
        mapping:
          info:
            - min: 0
              max: 3
            - low
          warn:
            - min: 4
              max: 6
            - medium
          error:
            - min: 7
              max: 8
            - high
          fatal:
            - min: 9
              max: 10
            - very-high

      - id: add_log_type
        type: add
        field: attributes.log_type
        value: cel
exporters:
  logging:


service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [logging]

Thanks @djaglowski 🚀

@djaglowski djaglowski merged commit cd1fc02 into open-telemetry:main Nov 17, 2022
@djaglowski djaglowski deleted the pkg-stanza-sev-range-fix branch November 17, 2022 17:45
JaredTan95 pushed a commit to openinsight-proj/opentelemetry-collector-contrib that referenced this pull request Nov 21, 2022
This functionality was broken during refactoring of the unmarshaling logic.

Previously, various yaml types could be used to represent a severity level.
This was designed to be as open as possible to future implementations. However,
it is now clear that only some keys are valid anyways. Namely, string and ints
that represent OTel's native severity levels. Both strings and ints are now treated
as strings when unmarshaling, which is fully compatible with the sev map lookup code.
shalper2 pushed a commit to shalper2/opentelemetry-collector-contrib that referenced this pull request Dec 6, 2022
This functionality was broken during refactoring of the unmarshaling logic.

Previously, various yaml types could be used to represent a severity level.
This was designed to be as open as possible to future implementations. However,
it is now clear that only some keys are valid anyways. Namely, string and ints
that represent OTel's native severity levels. Both strings and ints are now treated
as strings when unmarshaling, which is fully compatible with the sev map lookup code.
@plantfansam plantfansam mentioned this pull request Jul 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[pkg/stanza] Severity Parser isRange decode is expecting wrong type
4 participants