NEW_RELIC_LICENSE_KEY stopped taking priority over newrelic.yml

[sqbell]

After an upgrade from 8.13.1 to 8.16.0 we noticed no logs being shipped.

Description

The error was:

[2023-02-07 10:55:13 +0000 admin-deployment-557d4ffd7c-gj8dl (76)] WARN : No license key found. This often means your newrelic.yml file was not found, or it lacks a section for the running environment, 'production'. You may also want to try linting your newrelic.yml to ensure it is valid YML.

The file was present but had empty license_key field as we set it using environmental variable NEW_RELIC_LICENSE_KEY which, according to https://docs.newrelic.com/docs/apm/agents/ruby-agent/configuration/ruby-agent-configuration/, should take precedence.

Expected Behavior

As per https://docs.newrelic.com/docs/apm/agents/ruby-agent/configuration/ruby-agent-configuration/, we expect NEW_RELIC_LICENSE_KEY to take precedence over newrelic.yml setting.

Your Environment

Ruby: 3.1.2
Rails: 7.0.4.1
NR agent: 8.13.1 (works) => 8.16.0 (stopped working)

For Maintainers Only or Hero Triaging this bug

Suggested Priority (P1,P2,P3,P4,P5):
Suggested T-Shirt size (S, M, L, XL, Unknown):

[workato-integration]

https://issues.newrelic.com/browse/NEWRELIC-6702

[hannahramadan]

Hi @sqbell! Sorry to hear you're running into trouble and thanks for submitting an issue.

We've been unsuccessful in reproducing what you're seeing. From our investigation, the environment variable NEW_RELIC_LICENSE_KEY is still taking precedence over a license key set / unset / missing entirely from in newrelic.yml with the 8.16.0 release. We've also confirmed that environment variables are still given precedence when we search for configuration options.

To help us figure out what may be happening, we created a gist we'd love for you to run. This is a simple script that will help pinpoint what could be different in your application from what we're testing. Can you double-check that your license key environment variable is set and then run this script and see if we're grabbing the correct value?

[sqbell]

@hannahramadan Thank you for the time. I am sorry but I forgot to mention we also use dotenv to load the variables, so the original script would not work. I modified it like so:

#!/usr/bin/env ruby
# frozen_string_literal: true

# 1. Download which_license_key.rb and place it in the same repository as your application
# 2. Run this script with `ruby which_license_key.rb`
# 3. See the terminal output for the license key that New Relic is using

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'dotenv'
  gem 'newrelic_rpm'
end

Dotenv.load('../.env.local', "../.env.production", '../.env')

puts "License key is #{NewRelic::Agent.config[:license_key]}"
puts "ENV var: #{ENV['NEW_RELIC_LICENSE_KEY']}"

# Output:
License key is
ENV var: (our license key)

I'll try to spend more time figuring this one out this week.

[tannalynn]

Thanks for the additional information! Using dotenv is a different situation than using just environment variables as the ruby agent does not have any code specific to dotenv that would ensure environment variables loaded by dotenv are available to the agent.

This appears to be a load order issue. The agent needs to have environment variables available to it when the it is first loaded (which happens because it is also required by gem 'newrelic_rpm'), because that is when it loads all configuration. Since dotenv has not loaded the environment variables yet, they are not available to the agent.

I was able to confirm the same behavior you described using the script you provided. I have modified the script to delay the agent being loaded until after dotenv has run, and was able to see everything working as expected.

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'dotenv'
  gem 'newrelic_rpm', require: false
end

# Dotenv.load('../.env.local', "../.env.production", '../.env')
Dotenv.load('./.env')
require 'newrelic_rpm'

puts "New Relic agent version #{NewRelic::VERSION::STRING}"
puts "License key is #{NewRelic::Agent.config[:license_key]}"
puts "ENV var: #{ENV['NEW_RELIC_LICENSE_KEY']}"

# Output:
# New Relic agent version 8.16.0
# License key is my_license_key
# ENV var: my_license_key

Adding require: false to the agent in the gemfile and then later adding a require 'newrelic_rpm' after the point where dotenv is run in your application should allow the agent to work correctly with dotenv.

Alternatively, dotenv has some information in their README about how to allow dotenv to work with gems that require environment variables to be set before being loaded.

Also, I tried several ruby agent versions with your original script was not able to see any change in behavior between 8.13.1 and 8.16. I took a look, but I'm not aware of any changes we made in any of the versions (8.14, 8.15, 8.16) that would affect load order or anything that would explain what you're seeing. As far as I can tell, this seems like it should have always worked this way, so I'm not sure what could be going on with that. If you're able to narrow it down to a specific version bump where the behavior changes, I could take a closer look again.

Hopefully that helps! Please let me know if you run into any issues.

[sqbell]

Adding require: false to the agent in the gemfile and then later adding a require 'newrelic_rpm' after the point where dotenv is run in your application should allow the agent to work correctly with dotenv.

That works, however I still think something changed. I tried adding the debug statements directly into our Rails app:

require_relative 'boot'

require 'active_record/railtie'
require 'action_controller/railtie'
require 'action_view/railtie'
require 'action_mailer/railtie'
require 'active_job/railtie'
require 'action_cable/engine'
require 'rails/test_unit/railtie'
require 'sprockets/railtie'

# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)

module OurApp
  class Application < Rails::Application
    config.load_defaults 6.1

    root = OurApp::Engine.root
    Dotenv.load(root.join('../.env.local'), root.join("../.env.#{Rails.env}"), root.join('../.env'))

    puts "New Relic agent version #{NewRelic::VERSION::STRING}"
    puts "License key is #{NewRelic::Agent.config[:license_key]}"
    puts "ENV var: #{ENV['NEW_RELIC_LICENSE_KEY']}"
    # ...

I see the following:

# 8.13.1
New Relic agent version 8.13.1
License key is nr-license-key
ENV var: nr-license-key

# 8.14.0
New Relic agent version 8.14.0
License key is
ENV var: nr-license-key

# 8.16.0
New Relic agent version 8.16.0
License key is
ENV var: nr-license-key

I also looked at specific commits and pinpointed it to 1e989f1.

# e3171b4de2f3fc0d0d05fa4aa4bad74cdf108177

New Relic agent version 8.13.1
License key is nr-license-key
ENV var: nr-license-key

# 1e989f10338b802626635b83502fe319c18f0347

New Relic agent version 8.13.1
License key is
ENV var: nr-license-key

[tannalynn]

@sqbell Thank you for finding the specific version and commit that changed the behavior you're seeing! That is very helpful for me. I'll take a look at this to see why this changed things, since this change was intended to not alter the existing default behavior.

[tannalynn]

@sqbell It looks like this change in behavior was caused by the agent using configuration values earlier in the agent loading process than it did previously. Since we were using a new config for defer_rails_initialization to determine loading behavior with rails, the agent was triggering the configuration to load the environment variables earlier than before. This behavior is actually changing in #1791 to not use the agent config for this check, and will instead use an environment variable only for this configuration option. This is changing because accessing the config that early was actually before the agent had loaded the newrelic.yml.

I was able to reproduce that same output in a rails test app using your code snippet on 8.16 (thank you for providing that!), and switching to the bugfix/rails-init-env-toggle branch showed the agent behaving how it was previously in < 8.13. This PR will be included in our next release, 9.0.0.

[tannalynn]

closed by PR #1791 merge
This update will be included in our upcoming 9.0.0 release.