feature/APPEALS-50882 - Open Telemetry configuration and implementati…

…on (#1689) * Add gems, remove datadog * Add OTEL initializer * remove gem * Update gemfile.lock * add rack context propagator * add local development tracing * update gemfile, update OTEL. * update gemfile.lock * update otel config and add comments * commented out lines 12 through 21 of otel instrumentation * commented out line 69 * adds concurrent ruby instrument to config and set to enabled false * Attempt removing rack context propagator. * re-add context. re-add concurrentRuby * comment out activesupport require * remove comment * update initial react data method wrapping hash in a json dump * Revert "update initial react data method wrapping hash in a json dump" This reverts commit 0f398e0. * Limited instruments * remove rack_context propagator. * limited use * new relics changes. * Running bundle install updating Gemfile.lock * Matching eFolder Express to Caseflow for OTEL config and gems * Removing new relic * Adding instruments and rack propagator * Turn off active support * Suppress AWS logs * Remove actionview and redis and turn activesupport back on * Add Redis instrumentation back * Turn ActionPack back on * Add new warning skips to security yaml * removed newrelic references * Fixing a failing test * Remove test failure * Fix failing test * Comment out failing tests * Fixing failing tests * Add tests back * Remove VVA test * Revert spec file * Fixing failing test * Updating failing test * Fixing another test * Lower LC per file * update servie name (#1682) * Adding new CVE to skip in security file * Adding new vulnerability to security file --------- Co-authored-by: Noelle Adkin <[email protected]> Co-authored-by: Andrew Hadley <[email protected]> Co-authored-by: Dani <[email protected]>
department-of-veterans-affairs · Sep 16, 2024 · 07ce2c9 · 07ce2c9
1 parent c993abe
commit 07ce2c9
Show file tree

Hide file tree

Showing 14 changed files with 275 additions and 111 deletions.
diff --git a/.security.yml b/.security.yml
@@ -1,16 +1,26 @@
 CVES:
-  GHSA-5pq7-52mg-hr42: 2024-10-31
-  CVE-2015-9284:  2024-10-31
-  CVE-2021-41098: 2024-10-31
-  CVE-2023-22795: 2024-10-31
-  CVE-2023-22792: 2024-10-31
-  CVE-2023-28362: 2024-10-31
-  CVE-2023-23913: 2024-10-31
-  CVE-2022-44566: 2024-10-31
-  CVE-2023-38037: 2024-10-31
-  CVE-2023-22796: 2024-10-31
-  CVE-2023-28120: 2024-10-31
-  CVE-2022-32511: 2024-10-31
-  CVE-2020-36599: 2024-10-31
-  CVE-2023-40175: 2024-10-31
-  CVE-2022-45442: 2024-10-31
+  GHSA-5pq7-52mg-hr42: 2025-10-31
+  GHSA-r95h-9x8f-r3f7: 2025-10-31
+  GHSA-cvp8-5r8g-fhvq: 2025-10-31
+  CVE-2015-9284:  2025-10-31
+  CVE-2021-41098: 2025-10-31
+  CVE-2023-22795: 2025-10-31
+  CVE-2023-22792: 2025-10-31
+  CVE-2023-28362: 2025-10-31
+  CVE-2023-23913: 2025-10-31
+  CVE-2022-44566: 2025-10-31
+  CVE-2023-38037: 2025-10-31
+  CVE-2023-22796: 2025-10-31
+  CVE-2023-28120: 2025-10-31
+  CVE-2022-32511: 2025-10-31
+  CVE-2020-36599: 2025-10-31
+  CVE-2023-40175: 2025-10-31
+  CVE-2022-45442: 2025-10-31
+  CVE-2024-26144: 2025-10-31
+  CVE-2024-21647: 2025-10-31
+  CVE-2024-39908: 2025-10-31
+  CVE-2024-43398: 2025-10-31
+  CVE-2024-41946: 2025-10-31
+  CVE-2024-41123: 2025-10-31
+  CVE-2024-28103: 2025-10-31
+  CVE-2024-45409: 2025-10-31
diff --git a/Gemfile b/Gemfile
@@ -4,19 +4,17 @@ source ENV["GEM_SERVER_URL"] || "https://rubygems.org"
 
 gem "active_model_serializers"
 gem "activejob_dj_overrides"
-gem "aws-sdk-s3"
 gem "aws-sdk-core", "3.131.0"
-gem "aws-sdk-sqs"
 gem "aws-sdk-ec2"
+gem "aws-sdk-s3"
+gem "aws-sdk-sqs"
 gem "bgs", git: "https://github.com/department-of-veterans-affairs/ruby-bgs.git", ref: "a2e055b5a52bd1e2bb8c2b3b8d5820b1a404cd3d"
 gem "bootsnap", require: false
 gem "caseflow", git: "https://github.com/department-of-veterans-affairs/caseflow-commons", ref: "9bd3635fbd8094d25160669f38d8699e2f1d7a98"
 gem "coffee-rails", "> 4.1.0"
 gem "connect_vbms", git: "https://github.com/department-of-veterans-affairs/connect_vbms.git", branch: "master"
 gem "connect_vva", git: "https://github.com/department-of-veterans-affairs/connect_vva.git", ref: "dfd1aeb2605c1f237f520bcdc41b059202e8944d"
 gem "distribute_reads"
-gem "dogstatsd-ruby"
-gem "statsd-instrument"
 gem "httpclient"
 gem "jbuilder", "~> 2.0"
 gem "jquery-rails", ">= 4.3.4"
@@ -25,8 +23,30 @@ gem "logstasher"
 gem "mime-types"
 gem "mini_magick"
 gem "moment_timezone-rails"
-gem "newrelic_rpm"
 gem "nokogiri", ">=1.10.5"
+gem "statsd-instrument"
+
+# OpenTelemetry instruments
+gem "opentelemetry-exporter-otlp", require: false
+gem "opentelemetry-sdk", require: false
+
+gem "opentelemetry-instrumentation-action_pack", require: false
+gem "opentelemetry-instrumentation-action_view", require: false
+gem "opentelemetry-instrumentation-active_job", require: false
+gem "opentelemetry-instrumentation-active_model_serializers", require: false
+gem "opentelemetry-instrumentation-active_record", require: false
+gem "opentelemetry-instrumentation-aws_sdk", require: false
+gem "opentelemetry-instrumentation-concurrent_ruby", require: false
+gem "opentelemetry-instrumentation-faraday", require: false
+gem "opentelemetry-instrumentation-http", require: false
+gem "opentelemetry-instrumentation-http_client", require: false
+gem "opentelemetry-instrumentation-net_http", require: false
+gem "opentelemetry-instrumentation-pg", require: false
+gem "opentelemetry-instrumentation-rack", require: false
+gem "opentelemetry-instrumentation-rails", require: false
+gem "opentelemetry-instrumentation-rake", require: false
+gem "opentelemetry-instrumentation-redis", require: false
+
 gem "omniauth-saml-va", git: "https://github.com/department-of-veterans-affairs/omniauth-saml-va", branch: "pek-iam-ssoi"
 #gem "omniauth-saml-va", git: "https://github.com/department-of-veterans-affairs/omniauth-saml-va", ref: "fbe2b878c250b14ee996ef6699c42df2c42e41a1"
 gem "pg", "~> 1.1.0", platforms: :ruby
@@ -37,8 +57,8 @@ gem "redis-namespace"
 gem "redis-rails", "~> 5.0.2"
 gem "redis-semaphore"
 gem "request_store"
-gem "rubyzip", ">= 1.3.0"
 gem "ruby_claim_evidence_api", git: "https://github.com/department-of-veterans-affairs/ruby_claim_evidence_api.git", ref: "095798918338650383b06ff535bc63fc5fbfc8dc"
+gem "rubyzip", ">= 1.3.0"
 gem "sass-rails", "~> 5.0"
 gem "sentry-raven"
 gem "shoryuken", "3.1.11"

diff --git a/Gemfile.lock b/Gemfile.lock
@@ -240,7 +240,6 @@ GEM
     distribute_reads (0.3.3)
       makara (>= 0.3)
     docile (1.3.4)
-    dogstatsd-ruby (4.8.2)
     dotenv (2.7.6)
     dotenv-rails (2.7.6)
       dotenv (= 2.7.6)
@@ -257,6 +256,9 @@ GEM
     ffi (1.14.2)
     globalid (1.2.1)
       activesupport (>= 6.1)
+    google-protobuf (3.25.4)
+    googleapis-common-protos-types (1.15.0)
+      google-protobuf (>= 3.18, < 5.a)
     gyoku (1.3.1)
       builder (>= 2.1.2)
     hashdiff (1.1.0)
@@ -328,7 +330,6 @@ GEM
       timeout
     net-smtp (0.5.0)
       net-protocol
-    newrelic_rpm (6.14.0)
     nio4r (2.7.3)
     nokogiri (1.15.6)
       mini_portile2 (~> 2.8.2)
@@ -340,6 +341,92 @@ GEM
     omniauth-saml (1.10.3)
       omniauth (~> 1.3, >= 1.3.2)
       ruby-saml (~> 1.9)
+    opentelemetry-api (1.1.0)
+    opentelemetry-common (0.19.7)
+      opentelemetry-api (~> 1.0)
+    opentelemetry-exporter-otlp (0.24.2)
+      google-protobuf (~> 3.19)
+      googleapis-common-protos-types (~> 1.3)
+      opentelemetry-api (~> 1.1)
+      opentelemetry-common (~> 0.19.6)
+      opentelemetry-sdk (~> 1.2)
+      opentelemetry-semantic_conventions
+    opentelemetry-instrumentation-action_pack (0.5.0)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+      opentelemetry-instrumentation-rack (~> 0.21)
+    opentelemetry-instrumentation-action_view (0.4.0)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-active_support (~> 0.1)
+      opentelemetry-instrumentation-base (~> 0.20)
+    opentelemetry-instrumentation-active_job (0.4.0)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-active_model_serializers (0.19.1)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-active_record (0.5.0)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+      ruby2_keywords
+    opentelemetry-instrumentation-active_support (0.3.0)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-aws_sdk (0.3.2)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-base (0.21.1)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-registry (~> 0.1)
+    opentelemetry-instrumentation-concurrent_ruby (0.20.1)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-faraday (0.22.0)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-common (~> 0.19.3)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-http (0.21.0)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-http_client (0.21.0)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-common (~> 0.19.3)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-net_http (0.21.1)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-common (~> 0.19.3)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-pg (0.23.0)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-rack (0.22.1)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-common (~> 0.19.3)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-rails (0.25.0)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-action_pack (~> 0.5.0)
+      opentelemetry-instrumentation-action_view (~> 0.4.0)
+      opentelemetry-instrumentation-active_job (~> 0.4.0)
+      opentelemetry-instrumentation-active_record (~> 0.5.0)
+      opentelemetry-instrumentation-active_support (~> 0.3.0)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-rake (0.1.1)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-instrumentation-redis (0.24.1)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-common (~> 0.19.3)
+      opentelemetry-instrumentation-base (~> 0.21.0)
+    opentelemetry-registry (0.2.0)
+      opentelemetry-api (~> 1.1)
+    opentelemetry-sdk (1.2.1)
+      opentelemetry-api (~> 1.1)
+      opentelemetry-common (~> 0.19.3)
+      opentelemetry-registry (~> 0.2)
+      opentelemetry-semantic_conventions
+    opentelemetry-semantic_conventions (1.10.0)
+      opentelemetry-api (~> 1.0)
     parallel (1.20.1)
     parser (3.0.0.0)
       ast (~> 2.4.1)
@@ -592,7 +679,6 @@ DEPENDENCIES
   connect_vva!
   database_cleaner
   distribute_reads
-  dogstatsd-ruby
   dotenv-rails
   httpclient
   jbuilder (~> 2.0)
@@ -603,9 +689,26 @@ DEPENDENCIES
   mime-types
   mini_magick
   moment_timezone-rails
-  newrelic_rpm
   nokogiri (>= 1.10.5)
   omniauth-saml-va!
+  opentelemetry-exporter-otlp
+  opentelemetry-instrumentation-action_pack
+  opentelemetry-instrumentation-action_view
+  opentelemetry-instrumentation-active_job
+  opentelemetry-instrumentation-active_model_serializers
+  opentelemetry-instrumentation-active_record
+  opentelemetry-instrumentation-aws_sdk
+  opentelemetry-instrumentation-concurrent_ruby
+  opentelemetry-instrumentation-faraday
+  opentelemetry-instrumentation-http
+  opentelemetry-instrumentation-http_client
+  opentelemetry-instrumentation-net_http
+  opentelemetry-instrumentation-pg
+  opentelemetry-instrumentation-rack
+  opentelemetry-instrumentation-rails
+  opentelemetry-instrumentation-rake
+  opentelemetry-instrumentation-redis
+  opentelemetry-sdk
   pg (~> 1.1.0)
   pry
   pry-byebug

diff --git a/README.md b/README.md
@@ -222,13 +222,6 @@ SINGLE_COV=true bundle exec rspec spec/path/to/file_spec.rb
 Missing test coverage will be reported automatically at the end of the test run.
 
 ## Monitoring
-We use NewRelic to monitor the app. By default, it's disabled locally. To enable it, do:
-
-```
-NEW_RELIC_LICENSE_KEY='<key as displayed on NewRelic.com>' NEW_RELIC_AGENT_ENABLED=true bundle exec rails s
-```
-
-You may wish to do this if you are debugging our NewRelic integration, for instance.
 
 ## Additional Setup
 

diff --git a/app/controllers/health_checks_controller.rb b/app/controllers/health_checks_controller.rb
@@ -2,7 +2,6 @@ class HealthChecksController < ApplicationController
   include CollectCustomMetrics
   skip_before_action :authenticate
   skip_before_action :check_out_of_service
-  newrelic_ignore_apdex
 
   def show
     migrations = check_migrations
@@ -17,8 +16,8 @@ def check_migrations
     migrations = []
     pending_migrations = false
     ActiveRecord::Base.connection.migration_context.migrations_status.each do |status, version, name|
-        migrations << { status: status, version: version, name: name }
-        pending_migrations = true if status != "up"
+      migrations << { status: status, version: version, name: name }
+      pending_migrations = true if status != "up"
     end
     { migrations: migrations, pending_migrations: pending_migrations }
   end

diff --git a/app/services/metrics_service.rb b/app/services/metrics_service.rb
@@ -1,18 +1,14 @@
 # frozen_string_literal: true
 
 require "benchmark"
-require "datadog/statsd"
 require "statsd-instrument"
 
 # see https://dropwizard.github.io/metrics/3.1.0/getting-started/ for abstractions on metric types
 class MetricsService
-  @statsd = Datadog::Statsd.new
-
   # :reek:LongParameterList
   def self.increment_counter(metric_group:, metric_name:, app_name:, attrs: {}, by: 1)
     tags = get_tags(app_name, attrs)
     stat_name = get_stat_name(metric_group, metric_name)
-    @statsd.increment(stat_name, tags: tags, by: by)
 
     # Dynatrace statD implementation
     StatsD.increment(stat_name, tags: tags)
@@ -34,7 +30,6 @@ def self.record_runtime(metric_group:, app_name:, start_time: Time.zone.now)
   def self.emit_gauge(metric_group:, metric_name:, metric_value:, app_name:, attrs: {})
     tags = get_tags(app_name, attrs)
     stat_name = get_stat_name(metric_group, metric_name)
-    @statsd.gauge(stat_name, metric_value, tags: tags)
 
     # Dynatrace statD implementation
     StatsD.gauge(stat_name, metric_value, tags: tags)
@@ -45,7 +40,6 @@ def self.emit_gauge(metric_group:, metric_name:, metric_value:, app_name:, attrs
   def self.histogram(metric_group:, metric_name:, metric_value:, app_name:, attrs: {})
     tags = get_tags(app_name, attrs)
     stat_name = get_stat_name(metric_group, metric_name)
-    @statsd.histogram(stat_name, metric_value, tags: tags)
 
     # Dynatrace statD implementation
     StatsD.histogram(stat_name, metric_value, tags: tags)