Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: unroll processor #2021

Merged
merged 16 commits into from
Dec 3, 2024
Merged

feat: unroll processor #2021

merged 16 commits into from
Dec 3, 2024

Conversation

schmikei
Copy link
Contributor

@schmikei schmikei commented Dec 2, 2024

Proposed Change

Still need to write some documentation on this but the idea is to create new log entries out of arrays. The start of this is working exclusively with the body of a log entry.

Sample config

receivers:
  filelog:
    include: [ ./test.json ]
    start_at: beginning
  filelog/2:
    include: [ ./test2.json ]
    start_at: beginning
processors:
  transform:
    log_statements:
      - context: log
        statements:
          - set(body, Split(body, "\\n"))
          # - unroll(body)
  transform/split-by-comma:
    log_statements:
      - context: log
        statements:
          - set(body, Split(body, ","))
  transform/set-attr:
    log_statements:
      - context: log
        statements:
          - set(attributes["owner"], "Keith")
  unroll:
  unroll/body:
    unroll_key: ""

exporters:
  file:
    path: ./test/output.json
  debug:
    verbosity: detailed
  googlecloud/otel-agent-dev:
    log:
        compression: gzip
        default_log_name: Keiths-Macbook-Pro.local
        resource_filters:
            - regex: .*
    metric:
        compression: gzip
    project: otel-agent-dev


service:
  pipelines:
    # logs:
    #   receivers: [filelog]
    #   processors: [transform, unroll]
    #   exporters: [debug, file, googlecloud/otel-agent-dev]
    logs/2:
      receivers: [filelog/2]
      processors: [transform/split-by-comma, unroll/body, transform/set-attr]
      exporters: [debug, file, googlecloud/otel-agent-dev]

This for example can take in a log file like this

1,2,3,4,5,6,7,8

And it becomes multiple log events:

image
Checklist
  • Changes are tested
  • CI has passed

processor/unrollprocessor/processor.go Outdated Show resolved Hide resolved
processor/unrollprocessor/processor.go Outdated Show resolved Hide resolved
processor/unrollprocessor/processor.go Outdated Show resolved Hide resolved
@schmikei schmikei marked this pull request as ready for review December 3, 2024 16:33
@schmikei schmikei requested review from dpaasman00 and a team as code owners December 3, 2024 16:33
Copy link
Contributor

@dpaasman00 dpaasman00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple nits + what I messaged you on the side about

processor/unrollprocessor/README.md Outdated Show resolved Hide resolved
processor/unrollprocessor/go.mod Outdated Show resolved Hide resolved
@dpaasman00 dpaasman00 self-requested a review December 3, 2024 20:10
Copy link
Contributor

@dpaasman00 dpaasman00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good!

@schmikei schmikei merged commit 97f5f27 into release/v1.66.0 Dec 3, 2024
15 checks passed
@schmikei schmikei deleted the feat/unroll-processor branch December 3, 2024 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants