Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Utilize resources #105

Merged
merged 5 commits into from
Aug 31, 2020
Merged

Utilize resources #105

merged 5 commits into from
Aug 31, 2020

Conversation

camdencheek
Copy link
Contributor

@camdencheek camdencheek commented Aug 27, 2020

This commit makes a series of changes to make use of the new top-level
resource field.

The contents of $resource use the same resource standard as Open
Telemetry. See here for details: https://github.com/open-telemetry/opentelemetry-specification/tree/master/specification/resource/semantic_conventions

This adds support for accessing fields with dots in them using syntax like
$record['field.with.dots'].

This also modifies the host_metadata, k8s_event_input,
and k8s_metadata_decorator operators to add resource identifiers to
the entry.

Additionally, it adds support in the google_cloud_output for using the
resource field to add logs to a monitored resource. In the process, it
moves google_cloud_output into its own package for better organization
of its multiple files.

Includes a fix for a minor issue where rendering plugin ID in the template would set it as <no value> if using a default plugin ID.

Includes a fix for an issue where not all k8s events had a LastTimestamp, so now we prefer EventTimestamp, but fall back to LastTimestamp, then FirstTimestamp

Description of Changes

Please check that the PR fulfills these requirements

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • Add a changelog entry (for non-trivial bug fixes / features)
  • CI passes

This commit makes a series of changes to make use of the new top-level
resource field.

The contents of `$resource` use the same resource standard as Open
Telemetry. See here for details: https://github.com/open-telemetry/opentelemetry-specification/tree/master/specification/resource/semantic_conventions

This adds support for accessing fields with dots in them using syntax like
`$record['field.with.dots']`.

This also modifies the `host_metadata`, `k8s_event_input`,
and `k8s_metadata_decorator` operators to add resource identifiers to
the entry.

Additionally, it adds support in the `google_cloud_output` for using the
resource field to add logs to a monitored resource. In the process, it
moves `google_cloud_output` into its own package for better organization
of its multiple files.
@camdencheek camdencheek requested a review from djaglowski August 27, 2020 19:16
@codecov
Copy link

codecov bot commented Aug 27, 2020

Codecov Report

Merging #105 into master will decrease coverage by 0.90%.
The diff coverage is 57.58%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #105      +/-   ##
==========================================
- Coverage   73.17%   72.27%   -0.90%     
==========================================
  Files          71       75       +4     
  Lines        4294     4508     +214     
==========================================
+ Hits         3142     3258     +116     
- Misses        876      967      +91     
- Partials      276      283       +7     
Impacted Files Coverage Δ
operator/builtin/output/googlecloud/resource.go 13.43% <13.43%> (ø)
operator/builtin/input/k8s_events.go 40.16% <33.33%> (-3.51%) ⬇️
operator/builtin/output/googlecloud/payload.go 49.61% <49.61%> (ø)
...ator/builtin/transformer/k8s_metadata_decorator.go 43.07% <76.47%> (+0.43%) ⬆️
entry/field.go 81.82% <80.33%> (-4.39%) ⬇️
operator/builtin/output/googlecloud/severity.go 90.91% <90.91%> (ø)
entry/label_field.go 100.00% <100.00%> (ø)
entry/record_field.go 90.24% <100.00%> (+1.46%) ⬆️
entry/resource_field.go 100.00% <100.00%> (ø)
...perator/builtin/output/googlecloud/google_cloud.go 46.76% <100.00%> (ø)
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 949959c...bc433f9. Read the comment docs.

@observIQ observIQ deleted a comment from djaglowski Aug 31, 2020
@observIQ observIQ deleted a comment from djaglowski Aug 31, 2020
@observIQ observIQ deleted a comment from djaglowski Aug 31, 2020
@djaglowski
Copy link
Member

Log Files Logs / Second CPU Avg (%) CPU Avg Δ (%) Memory Avg (MB) Memory Avg Δ (MB)
1 1000 3.6035423 -0.32753897 35.774113 +2.27034
1 5000 9.258792 -1.0516052 42.69774 -1.0237045
1 10000 16.741867 +0.6380253 58.011314 +1.2265625
1 50000 72.68873 +0.39237213 205.96512 -2.555359
1 100000 145.14624 +1.2762909 332.74905 +30.696777
10 100 4.5691113 -0.5689082 31.117052 -2.718481
10 500 11.586083 -0.15560722 44.712555 -2.3917007
10 1000 19.396875 +0.41363907 58.13524 +1.1724167
10 5000 78.536674 +0.7763443 195.98962 -3.5595398
10 10000 165.85332 +14.856201 315.40207 -3.832428

@camdencheek camdencheek merged commit e614037 into master Aug 31, 2020
@camdencheek camdencheek deleted the resources branch August 31, 2020 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants