Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

file input: optional file_name_resolved and file_path_resolved labels #364

Merged
merged 2 commits into from
Jul 19, 2021

Conversation

jsirianni
Copy link
Member

@jsirianni jsirianni commented Jul 15, 2021

Description of Changes

Add optional file_name_resolved and file_path_resolved labels to file input

Ported from OTEL: https://github.com/open-telemetry/opentelemetry-log-collection/pull/189/files

With new labels enabled

{
  "timestamp": "2021-07-15T19:39:39.289773231-04:00",
  "severity": 0,
  "labels": {
    "file_name": "in-symlink.json",
    "file_name_resolved": "test.json",
    "file_path": "./log/in-symlink.json",
    "file_path_resolved": "/tmp/test.json"
  },
  "record": {
    "key-symlink": "value"
  }
}
{
  "timestamp": "2021-07-15T19:39:39.289788021-04:00",
  "severity": 0,
  "labels": {
    "file_name": "in.json",
    "file_name_resolved": "in.json",
    "file_path": "./log/in.json",
    "file_path_resolved": "/home/jsirianni/go/src/github.com/observiq/stanza/log/in.json"
  },
  "record": {
    "key": "value"
  }
}

Config

pipeline:
- type: file_input
  include: 
  - ./log/in.json
  - ./log/in-symlink.json
  include_file_path: true
  include_file_name_resolved: true
  include_file_path_resolved: true
  start_at: beginning
- type: json_parser
- type: stdout

Please check that the PR fulfills these requirements

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • Add a changelog entry (for non-trivial bug fixes / features)
  • CI passes

@jsirianni jsirianni requested a review from djaglowski July 15, 2021 23:51
@codecov
Copy link

codecov bot commented Jul 15, 2021

Codecov Report

Merging #364 (15a78e4) into master (0019926) will decrease coverage by 0.14%.
The diff coverage is 84.91%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #364      +/-   ##
==========================================
- Coverage   73.33%   73.19%   -0.14%     
==========================================
  Files         124      124              
  Lines        7948     7974      +26     
==========================================
+ Hits         5828     5836       +8     
- Misses       1629     1644      +15     
- Partials      491      494       +3     
Impacted Files Coverage Δ
operator/builtin/input/file/reader.go 71.05% <60.00%> (-1.40%) ⬇️
operator/builtin/input/file/config.go 80.00% <100.00%> (+2.86%) ⬆️
operator/builtin/input/file/file.go 79.06% <100.00%> (-3.66%) ⬇️
operator/buffer/memory.go 75.33% <0.00%> (-4.67%) ⬇️
operator/builtin/output/forward/forward.go 60.49% <0.00%> (-1.23%) ⬇️
operator/builtin/output/otlp/otlp.go 68.13% <0.00%> (+3.30%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0019926...15a78e4. Read the comment docs.

@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 1.4654696 +0.06888926 127.57207 -0.35573578
1 5000 5.017335 -0.15506506 143.12595 +8.293381
1 10000 10.603407 +0.5514803 145.74124 +1.5323181
1 50000 52.67129 +3.5157433 178.35547 +1.3708191
1 100000 83.623116 -16.75943 1080.219 +836.9307
10 100 1.982852 +0.051787138 135.11853 +4.852371
10 500 6.0347166 +0.08624411 136.81923 -3.0370483
10 1000 12.05211 +0.776042 151.88861 +6.9226837
10 5000 56.27653 +0.12049866 182.16837 +7.0234375
10 10000 73.08633 -32.21685 602.31885 +374.56076

@jsirianni jsirianni merged commit 291404b into master Jul 19, 2021
@jsirianni jsirianni deleted the file-path-resolved branch July 19, 2021 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants