Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logstasher instrumentation #2559

Merged
merged 56 commits into from
Jul 11, 2024
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
867f4c1
Instrumentation skeleton
hannahramadan Apr 15, 2024
82f1f43
Merge branch 'dev' into logstasher_instrumentation
hannahramadan Jun 11, 2024
ec7de88
Merge branch 'dev' into logstasher_instrumentation
hannahramadan Jun 13, 2024
5531741
log decorator: support hashes
fallwith Jun 21, 2024
2485d79
log decorator: return early for frozen hashes
fallwith Jun 21, 2024
b1f3f0b
Merge pull request #2729 from newrelic/mark_williams
hannahramadan Jun 21, 2024
b302925
Add instrumenting methods
hannahramadan Jun 21, 2024
36f0a05
Instrumentation updates
hannahramadan Jun 24, 2024
693a991
Rubocop: spacing
hannahramadan Jun 24, 2024
d0ad2d2
Add tests
hannahramadan Jun 25, 2024
5cc4d09
Rubocop
hannahramadan Jun 25, 2024
b0816c6
Update lib/new_relic/agent/instrumentation/logstasher/instrumentation.rb
hannahramadan Jun 25, 2024
f00a319
Test updates
hannahramadan Jun 25, 2024
acda525
test and instrumentation udpates
hannahramadan Jun 25, 2024
4827fec
test updates
hannahramadan Jun 25, 2024
032e67d
Test updates
hannahramadan Jun 27, 2024
e0b714b
Merge branch 'dev' into logstasher_instrumentation
hannahramadan Jun 27, 2024
f8542b2
Rubocop
hannahramadan Jun 27, 2024
fce3e68
Envfile update
hannahramadan Jun 27, 2024
9ccd046
require ostruct
hannahramadan Jun 27, 2024
8b88ba6
Add changelog and version gate
hannahramadan Jun 27, 2024
6ad0e5f
Test updates
hannahramadan Jun 28, 2024
c3f3089
rubocop beep beep
hannahramadan Jun 28, 2024
ba44298
More tests
hannahramadan Jun 28, 2024
072e97c
metrics enabled test
hannahramadan Jun 28, 2024
20304b0
message change
hannahramadan Jun 28, 2024
df672a7
test updates
hannahramadan Jun 28, 2024
c8a7d04
test: record nil on kaboom
hannahramadan Jun 28, 2024
df447d6
beep beep rubocop
hannahramadan Jun 28, 2024
7462d5e
Apply suggestions from code review
hannahramadan Jun 28, 2024
de96a74
Apply suggestions from code review
hannahramadan Jul 1, 2024
4fabc5b
Create shared methods
hannahramadan Jul 1, 2024
229c7d1
Monitoring conditions?
hannahramadan Jul 1, 2024
278c972
Update pin
hannahramadan Jul 1, 2024
45e6bc4
Always capitalize severity level
hannahramadan Jul 1, 2024
f689fb7
Don't send empty strings
hannahramadan Jul 1, 2024
84c64b3
Spruce up attributes test
hannahramadan Jul 1, 2024
3546a3c
Upcase severity levels
hannahramadan Jul 2, 2024
6eef9a6
Fix name
hannahramadan Jul 2, 2024
32036e5
Update linking metadata tests
hannahramadan Jul 2, 2024
559ecd4
Small cleanups
hannahramadan Jul 2, 2024
02d09b3
Use initialize value
hannahramadan Jul 3, 2024
29a78c0
Use singletone class
hannahramadan Jul 3, 2024
18ca504
revert initialize changes
hannahramadan Jul 3, 2024
1ab9aa2
remove required file
hannahramadan Jul 3, 2024
1a70059
Use *args
hannahramadan Jul 3, 2024
3168617
All keys are strings
hannahramadan Jul 3, 2024
6dc33a4
revert to_s
hannahramadan Jul 3, 2024
fecf021
up ruby version
hannahramadan Jul 3, 2024
07353c7
Trigger ci run
hannahramadan Jul 9, 2024
752c1f2
Merge branch 'dev' into logstasher_instrumentation
hannahramadan Jul 9, 2024
c8e2d8e
Update CHANGELOG.md
hannahramadan Jul 9, 2024
9e2fa2d
Add comment
hannahramadan Jul 9, 2024
015d13b
Apply suggestions from code review
hannahramadan Jul 9, 2024
068acc3
add envfile comment
hannahramadan Jul 9, 2024
cf27f6c
Use less than 7.1
hannahramadan Jul 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions lib/new_relic/agent/configuration/default_source.rb
Original file line number Diff line number Diff line change
Expand Up @@ -1595,6 +1595,15 @@ def self.enforce_fallback(allowed_values: nil, fallback: nil)
:allowed_from_server => false,
:description => 'Controls auto-instrumentation of Ruby standard library Logger at start-up. May be one of: `auto`, `prepend`, `chain`, `disabled`.'
},
:'instrumentation.logstasher' => {
:default => instrumentation_value_from_boolean(:'application_logging.enabled'),
:documentation_default => 'auto',
:public => true,
:type => String,
:dynamic_name => true,
:allowed_from_server => false,
:description => 'Controls auto-instrumentation of the LogStasher library at start-up. May be one of [auto|prepend|chain|disabled].'
},
:'instrumentation.memcache' => {
:default => 'auto',
:public => true,
Expand Down
26 changes: 26 additions & 0 deletions lib/new_relic/agent/instrumentation/logstasher.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# This file is distributed under New Relic's license terms.
# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
# frozen_string_literal: true

require_relative 'logstasher/instrumentation'
require_relative 'logstasher/chain'
require_relative 'logstasher/prepend'

DependencyDetection.defer do
named :logstasher

depends_on do
defined?(LogStasher) &&
NewRelic::Agent.config[:'application_logging.enabled']
end

executes do
NewRelic::Agent.logger.info('Installing LogStasher instrumentation')

if use_prepend?
prepend_instrument LogStasher, NewRelic::Agent::Instrumentation::LogStasher::Prepend
else
chain_instrument NewRelic::Agent::Instrumentation::LogStasher::Chain
end
end
end
21 changes: 21 additions & 0 deletions lib/new_relic/agent/instrumentation/logstasher/chain.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# This file is distributed under New Relic's license terms.
# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
# frozen_string_literal: true

module NewRelic::Agent::Instrumentation
module LogStasher::Chain
def self.instrument!
::LogStasher.class_eval do
include NewRelic::Agent::Instrumentation::LogStasher

alias_method(:build_logstash_event_without_new_relic, :build_logstash_event)

def build_logstash_event(data, tags)
build_logstash_event_with_new_relic(data, tags) do
build_logstash_event_without_new_relic(data, tags)
end
end
end
end
end
end
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# This file is distributed under New Relic's license terms.
# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
# frozen_string_literal: true

module NewRelic::Agent::Instrumentation
module LogStasher
INSTRUMENTATION_NAME = 'LogStasher'
hannahramadan marked this conversation as resolved.
Show resolved Hide resolved

def self.enabled?
NewRelic::Agent.config[:'instrumentation.logstasher'] != 'disabled'
end

def build_logstash_event_with_new_relic(data, tags)
data = yield
hannahramadan marked this conversation as resolved.
Show resolved Hide resolved
log = data.instance_variable_get(:@data)

::NewRelic::Agent.record_instrumentation_invocation(INSTRUMENTATION_NAME)
::NewRelic::Agent.agent.log_event_aggregator.record_json(log)
::NewRelic::Agent::LocalLogDecorator.decorate(log)

data
end
end
end
13 changes: 13 additions & 0 deletions lib/new_relic/agent/instrumentation/logstasher/prepend.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# This file is distributed under New Relic's license terms.
# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
# frozen_string_literal: true

module NewRelic::Agent::Instrumentation
module LogStasher::Prepend
include NewRelic::Agent::Instrumentation::LogStasher

def build_logstash_event(data, tags)
build_logstash_event_with_new_relic(data, tags) { super }
end
end
end
9 changes: 8 additions & 1 deletion lib/new_relic/agent/local_log_decorator.rb
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@ def decorate(message)
return message unless decorating_enabled?

metadata = NewRelic::Agent.linking_metadata

if message.is_a?(Hash)
message.merge!(metadata) unless message.frozen?
return
end

formatted_metadata = " NR-LINKING|#{metadata[ENTITY_GUID_KEY]}|#{metadata[HOSTNAME_KEY]}|" \
"#{metadata[TRACE_ID_KEY]}|#{metadata[SPAN_ID_KEY]}|" \
"#{escape_entity_name(metadata[ENTITY_NAME_KEY])}|"
Expand All @@ -23,7 +29,8 @@ def decorate(message)

def decorating_enabled?
NewRelic::Agent.config[:'application_logging.enabled'] &&
NewRelic::Agent::Instrumentation::Logger.enabled? &&
(NewRelic::Agent::Instrumentation::Logger.enabled? ||
NewRelic::Agent::Instrumentation::LogStasher.enabled?) &&
NewRelic::Agent.config[:'application_logging.local_decorating.enabled']
end

Expand Down
72 changes: 71 additions & 1 deletion lib/new_relic/agent/log_event_aggregator.rb
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,42 @@ def record(formatted_message, severity)
nil
end

def record_json(log)
return unless enabled?

severity = log['level'] || 'UNKNOWN'

if NewRelic::Agent.config[METRICS_ENABLED_KEY]
hannahramadan marked this conversation as resolved.
Show resolved Hide resolved
@counter_lock.synchronize do
@seen += 1
@seen_by_severity[severity] += 1
end
end

return if severity_too_low?(severity)
# The message key only exists if users manually add log statements
return if log.key?(:message) && (log.key?(:message).nil? || llog.key?(:message).empty?)
fallwith marked this conversation as resolved.
Show resolved Hide resolved
return unless NewRelic::Agent.config[FORWARDING_ENABLED_KEY]
return if @high_security

txn = NewRelic::Agent::Transaction.tl_current
priority = LogPriority.priority_for(txn)
message = get_json_message(log)
fallwith marked this conversation as resolved.
Show resolved Hide resolved

return txn.add_log_event(create_json_event(priority, message, severity, log)) if txn

@lock.synchronize do
@buffer.append(priority: priority) do
create_json_event(priority, message, severity, log)
end
end
rescue
end

def get_json_message(log)
log['message'] || ''
end

def record_batch(txn, logs)
# Ensure we have the same shared priority
priority = LogPriority.priority_for(txn)
Expand Down Expand Up @@ -121,6 +157,36 @@ def create_event(priority, formatted_message, severity)
]
end

def create_json_event(priority, formatted_message, severity, log)
formatted_message = truncate_message(formatted_message)

event = LinkingMetadata.append_trace_linking_metadata({
LEVEL_KEY => severity,
MESSAGE_KEY => formatted_message,
TIMESTAMP_KEY => Process.clock_gettime(Process::CLOCK_REALTIME) * 1000
})

add_json_event_attributes(event, log)

[
{
PrioritySampledBuffer::PRIORITY_KEY => priority
},
event
]
end

def add_json_event_attributes(event, log)
log_copy = log.dup
# Delete previously reported attributes
log_copy.delete('message')
log_copy.delete('level')
log_copy.delete('@timestamp')
attributes = event['attributes'] = {}

event['attributes'].merge!(log_copy)
end

def add_custom_attributes(custom_attributes)
attributes.add_custom_attributes(custom_attributes)
end
Expand Down Expand Up @@ -167,7 +233,11 @@ def reset!
end

def enabled?
@enabled && @instrumentation_logger_enabled
@enabled && instrumentation_enabled?
end

def instrumentation_enabled?
NewRelic::Agent::Instrumentation::Logger.enabled? || NewRelic::Agent::Instrumentation::LogStasher.enabled?
end

private
Expand Down
9 changes: 9 additions & 0 deletions test/multiverse/suites/logstasher/Envfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# This file is distributed under New Relic's license terms.
# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
# frozen_string_literal: true

instrumentation_methods :chain, :prepend

gemfile <<~RB
gem 'logstasher'
RB
19 changes: 19 additions & 0 deletions test/multiverse/suites/logstasher/config/newrelic.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
development:
error_collector:
enabled: true
apdex_t: 0.5
monitor_mode: true
license_key: bootstrap_newrelic_admin_license_key_000
instrumentation:
logstasher: <%= $instrumentation_method %>
app_name: test
log_level: debug
host: 127.0.0.1
api_host: 127.0.0.1
transaction_trace:
record_sql: obfuscated
enabled: true
stack_trace_threshold: 0.5
transaction_threshold: 1.0
capture_params: false
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# This file is distributed under New Relic's license terms.
# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
# frozen_string_literal: true

class LogstasherInstrumentationTest < Minitest::Test
def setup
@stats_engine = NewRelic::Agent.instance.stats_engine
end

def teardown
NewRelic::Agent.instance.stats_engine.clear_stats
end

# Add tests here
end
28 changes: 28 additions & 0 deletions test/new_relic/agent/local_log_decorator_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,34 @@ def test_safe_without_entity_name
assert_includes decorated_message, '||'
end
end

def test_decorates_json_log_hashes
canned_hostname = 'blazkowicz'
hash = {'dennis' => 'gnasher'}

in_transaction do |txn|
expected = hash.merge({'entity.name' => @enabled_config[:app_name],
'entity.type' => 'SERVICE',
'hostname' => canned_hostname,
'entity.guid' => @enabled_config[:entity_guid],
'trace.id' => txn.trace_id,
'span.id' => txn.segments.first.guid})

NewRelic::Agent::Hostname.stub(:get, canned_hostname) do
LocalLogDecorator.decorate(hash)
end

assert_equal expected, hash, "Expected hash to be decorated. Wanted >>#{expected}<<, got >>#{hash}<<"
end
end

def test_returns_early_with_frozen_hashes
hash = {'dennis' => 'gnasher'}.freeze
expected = hash.dup
LocalLogDecorator.decorate(hash)

assert_equal expected, hash, 'Expected no errors and no hash modifications for a frozen hash'
end
end
end
end
Loading