sample component overwrite sample_rate field from the original events

[richardfan1126]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

The sample component will add a field sample_rate into the log event.

If the original event has the field with the same name, it will be overwritten, which cause data loss.

Configuration

sources:
  http:
    type: http_server
    address: 0.0.0.0:8080

transforms:
  unwrap:
    type: remap
    inputs:
      - http
    source: |
      . = parse_json!(string!(.message))
  sample:
    type: sample
    inputs:
      - unwrap
    rate: 1

sinks:
  console:
    type: console
    inputs:
      - sample
    encoding:
      codec: json

Version

0.31.0

Debug Output

No response

Example Data

Input data:

{"foo":"bar","sample_rate":"testing"}

Output data:

{"foo":"bar","sample_rate":"1"}

Additional Context

No response

References

No response

[jszwedko]

Hi @richardfan1126 !

This is an issue we are hoping to address through a new opt-in feature we released in v0.31.0 to have Vector add its metadata fields to a separate namespace. See https://vector.dev/blog/log-namespacing/ for a description of how this works and how to enable it. Would you want to try that out and see how it works for you? You can leave any feedback on it here: #17796.

[richardfan1126]

@jszwedko Thanks for the suggestion, Log Namespacing is a great feature.

However, it seems like only apply to source and have no effect on sample component.

Here's my testing:

• Configuration:

  schema:
    log_namespace: true
  
  sources:
    http:
      type: http_server
      address: 0.0.0.0:8080
  
  transforms:
    unwrap:
      type: remap
      inputs:
        - http
      source: |
        . = parse_json!(string!(.message))
    sample:
      type: sample
      inputs:
        - unwrap
      rate: 1
  
  sinks:
    console:
      type: console
      inputs:
        - sample
      encoding:
        codec: json

• Input:

  {"foo":"bar","sample_rate":"testing"}

• Output

  {"foo":"bar","sample_rate":"1"}

[jszwedko]

Thanks for flagging this @richardfan1126 . #18259 should resolve it by moving the sample_rate field to metadata.