Merge pull request #2084 from newrelic/custom_log_attributes_api

Create custom_log_attributes API

CHANGELOG.md

@@ -2,7 +2,7 @@
 
 ## dev
 
-Version <dev> of the agent adds the ability to filter logs by level, expands instrumentation for Action Cable,provides bugfix for Code-Level Metrics, a bugfix for how `Fiber` args are treated, and a bugfix for `NewRelic::Agent::Logging::DecoratingFormatter#clear_tags!` being incorrectly private.
+Version <dev> of the agent adds log-level filtering, an API to add custom attributes to logs, and updated instrumentation for Action Cable. It also provides fixes for how `Fiber` args are treated, Code-Level Metrics, and `NewRelic::Agent::Logging::DecoratingFormatter#clear_tags!` being incorrectly private.
 
 - **Feature: Filter forwarded logs based on level**
 
@@ -14,6 +14,14 @@ Version <dev> of the agent adds the ability to filter logs by level, expands ins
 
   This setting uses [Ruby's Logger::Severity constants integer values](https://github.com/ruby/ruby/blob/master/lib/logger/severity.rb#L6-L17) to determine precedence.
 
+- **Feature: Custom attributes for logs API**
+
+  You can now add custom attributes to your log events using `NewRelic::Agent.add_custom_log_attributes`.
+
+  For example: `NewRelic::Agent.add_custom_log_attributes(dyno: ENV['DYNO'], pod_name: ENV['POD_NAME'])`, will add the attributes `dyno` and `pod_name` to your log events. Attributes passed to this API will be added to all log events.
+
+  Thanks to [@rajpawar02](https://github.com/rajpawar02) for raising this issue and [@askreet](https://github.com/askreet) for helping us with the solution. [Issue#1141](https://github.com/newrelic/newrelic-ruby-agent/issues/1141), [PR#2084](https://github.com/newrelic/newrelic-ruby-agent/pull/2084)
+
 - **Feature: Instrument transmit_subscription_* Action Cable actions**
 
   This change subscribes the agent to the Active Support notifications for:

lib/new_relic/agent.rb

@@ -649,6 +649,43 @@ def add_custom_span_attributes(params)
       end
     end
 
+    # Add custom attributes to log events for the current agent instance.
+    #
+    # @param [Hash] params    A Hash of attributes to attach to log
+    #                         events. The agent accepts up to 240 custom
+    #                         log event attributes.
+    #
+    #                         Keys will be coerced into Strings and must
+    #                         be less than 256 characters. Keys longer
+    #                         than 255 characters will be truncated.
+    #
+    #                         Values may be Strings, Symbols, numeric
+    #                         values or Booleans and must be less than
+    #                         4095 characters. If the value is a String
+    #                         or a Symbol, values longer than 4094
+    #                         characters will be truncated. If the value
+    #                         exceeds 4094 characters and is of a
+    #                         different class, the attribute pair will
+    #                         be dropped.
+    #
+    #                         This API can be called multiple times.
+    #                         If the same key is passed more than once,
+    #                         the value associated with the last call
+    #                         will be preserved.
+    #
+    #                         Attribute pairs with empty or nil contents
+    #                         will be dropped.
+    # @api public
+    def add_custom_log_attributes(params)
+      record_api_supportability_metric(:add_custom_log_attributes)
+
+      if params.is_a?(Hash)
+        NewRelic::Agent.agent.log_event_aggregator.add_custom_attributes(params)
+      else
+        NewRelic::Agent.logger.warn("Bad argument passed to #add_custom_log_attributes. Expected Hash but got #{params.class}.")
+      end
+    end
+
     # Set the user id for the current transaction. When present, this value will be included in the agent attributes for transaction and error events as 'enduser.id'.
     #
     # @param [String] user_id    The user id to add to the current transaction attributes

lib/new_relic/agent/log_event_aggregator.rb

@@ -4,6 +4,7 @@
 
 require 'new_relic/agent/event_aggregator'
 require 'new_relic/agent/log_priority'
+require 'new_relic/agent/log_event_attributes'
 
 module NewRelic
   module Agent
@@ -38,13 +39,16 @@ class LogEventAggregator < EventAggregator
       DECORATING_ENABLED_KEY = :'application_logging.local_decorating.enabled'
       LOG_LEVEL_KEY = :'application_logging.forwarding.log_level'
 
+      attr_reader :attributes
+
       def initialize(events)
         super(events)
         @counter_lock = Mutex.new
         @seen = 0
         @seen_by_severity = Hash.new(0)
         @high_security = NewRelic::Agent.config[:high_security]
         @instrumentation_logger_enabled = NewRelic::Agent::Instrumentation::Logger.enabled?
+        @attributes = NewRelic::Agent::LogEventAttributes.new
         register_for_done_configuring(events)
       end
 
@@ -116,6 +120,10 @@ def create_event(priority, formatted_message, severity)
         ]
       end
 
+      def add_custom_attributes(custom_attributes)
+        attributes.add_custom_attributes(custom_attributes)
+      end
+
       # Because our transmission format (MELT) is different than historical
       # agent payloads, extract the munging here to keep the service focused
       # on the general harvest + transmit instead of the format.
@@ -132,6 +140,8 @@ def self.payload_to_melt_format(data)
         # sent by classic logs-in-context
         common_attributes.delete(ENTITY_TYPE_KEY)
 
+        common_attributes.merge!(NewRelic::Agent.agent.log_event_aggregator.attributes.custom_attributes)
+
         _, items = data
         payload = [{
           common: {attributes: common_attributes},
@@ -151,6 +161,7 @@ def reset!
           @seen = 0
           @seen_by_severity.clear
         end
+
         super
       end
 

lib/new_relic/agent/log_event_attributes.rb

@@ -0,0 +1,115 @@
+# This file is distributed under New Relic's license terms.
+# See https://github.com/newrelic/newrelic-ruby-agent/blob/main/LICENSE for complete details.
+# frozen_string_literal: true
+
+module NewRelic
+  module Agent
+    class LogEventAttributes
+      MAX_ATTRIBUTE_COUNT = 240 # limit is 255, assume we send 15
+      ATTRIBUTE_KEY_CHARACTER_LIMIT = 255
+      ATTRIBUTE_VALUE_CHARACTER_LIMIT = 4094
+
+      def add_custom_attributes(attributes)
+        return if @custom_attribute_limit_reached
+
+        attributes.each do |key, value|
+          next if absent?(key) || absent?(value)
+
+          add_custom_attribute(key, value)
+        end
+      end
+
+      def custom_attributes
+        @custom_attributes ||= {}
+      end
+
+      private
+
+      class TruncationError < StandardError
+        attr_reader :attribute, :limit
+
+        def initialize(attribute, limit, msg = "Can't truncate")
+          @attribute = attribute
+          @limit = limit
+          super(msg)
+        end
+      end
+
+      class InvalidTypeError < StandardError
+        attr_reader :attribute
+
+        def initialize(attribute, msg = 'Invalid attribute type')
+          @attribute = attribute
+          super(msg)
+        end
+      end
+
+      def absent?(value)
+        value.nil? || (value.respond_to?(:empty?) && value.empty?)
+      end
+
+      def add_custom_attribute(key, value)
+        if custom_attributes.size >= MAX_ATTRIBUTE_COUNT
+          NewRelic::Agent.logger.warn(
+            'Too many custom log attributes defined. ' \
+            "Only taking the first #{MAX_ATTRIBUTE_COUNT}."
+          )
+          @custom_attribute_limit_reached = true
+          return
+        end
+
+        @custom_attributes.merge!(truncate_attributes(key_to_string(key), value))
+      end
+
+      def key_to_string(key)
+        key.is_a?(String) ? key : key.to_s
+      end
+
+      def truncate_attribute(attribute, limit)
+        case attribute
+        when Integer
+          if attribute.digits.length > limit
+            raise TruncationError.new(attribute, limit)
+          end
+        when Float
+          if attribute.to_s.length > limit
+            raise TruncationError.new(attribute, limit)
+          end
+        when String, Symbol
+          if attribute.length > limit
+            attribute = attribute.slice(0..(limit - 1))
+          end
+        when TrueClass, FalseClass
+          attribute
+        else
+          raise InvalidTypeError.new(attribute)
+        end
+
+        attribute
+      end
+
+      def truncate_attributes(key, value)
+        key = truncate_attribute(key, ATTRIBUTE_KEY_CHARACTER_LIMIT)
+        value = truncate_attribute(value, ATTRIBUTE_VALUE_CHARACTER_LIMIT)
+
+        {key => value}
+      rescue TruncationError => e
+        NewRelic::Agent.logger.warn(
+          "Dropping custom log attribute #{key} => #{value} \n" \
+          "Length exceeds character limit of #{e.limit}. " \
+          "Can't truncate: #{e.attribute}"
+        )
+
+        {}
+      rescue InvalidTypeError => e
+        NewRelic::Agent.logger.warn(
+          "Dropping custom log attribute #{key} => #{value} \n" \
+          "Invalid type of #{e.attribute.class} given. " \
+          "Can't send #{e.attribute}."
+        )
+
+        {}
+      end
+    end
+  end
+end

lib/new_relic/supportability_helper.rb

@@ -13,6 +13,7 @@ module SupportabilityHelper
       :insert_distributed_trace_headers,
       :accept_distributed_trace_headers,
       :add_custom_attributes,
+      :add_custom_log_attributes,
       :add_custom_span_attributes,
       :add_instrumentation,
       :add_method_tracer,

test/new_relic/agent/log_event_aggregator_test.rb

@@ -180,6 +180,15 @@ def test_record_applies_limits
       end
     end
 
+    def test_record_exits_if_forwarding_disabled
+      with_config(LogEventAggregator::FORWARDING_ENABLED_KEY => false) do
+        @aggregator.record('Speak friend and enter', 'DEBUG')
+        _, results = @aggregator.harvest!
+
+        assert_empty(results)
+      end
+    end
+
     def test_record_in_transaction
       max_samples = 100
       with_config(CAPACITY_KEY => max_samples) do