Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add exception notification view for sidekiq data #1994

Merged
merged 1 commit into from
Feb 15, 2018

Conversation

jmhooper
Copy link
Member

Why: By default, exception notification dumps everything it knows
about a job in the email it sends. This often contains the data that was
enqueued with the job. For our idv jobs, this data is PII. This commit
uses a whitelist approach to cause exception notifier to only email us a
backtrace and some metadata about the job.

Hi! Before submitting your PR for review, and/or before merging it, please
go through the following checklist:

  • For DB changes, check for missing indexes, check to see if the changes
    affect other apps (such as the dashboard), make sure the DB columns in the
    various environments are properly populated, coordinate with devops, plan
    migrations in separate steps.

  • For route changes, make sure GET requests don't change state or result in
    destructive behavior. GET requests should only result in information being
    read, not written.

  • For encryption changes, make sure it is compatible with data that was
    encrypted with the old code.

  • For secrets changes, make sure to update the S3 secrets bucket with the
    new configs in all environments.

  • Do not disable Rubocop or Reek offenses unless you are absolutely sure
    they are false positives. If you're not sure how to fix the offense, please
    ask a teammate.

  • When reading data, write tests for nil values, empty strings,
    and invalid formats.

  • When calling redirect_to in a controller, use _url, not _path.

  • When adding user data to the session, use the user_session helper
    instead of the session helper so the data does not persist beyond the user's
    session.

  • When adding a new controller that requires the user to be fully
    authenticated, make sure to add before_action :confirm_two_factor_authenticated.

**Why**: By default, exception notification dumps everything it knows
about a job in the email it sends. This often contains the data that was
enqueued with the job. For our idv jobs, this data is PII. This commit
uses a whitelist approach to cause exception notifier to only email us a
backtrace and some metadata about the job.
@jmhooper
Copy link
Member Author

jmhooper commented Feb 15, 2018

Output in the email looks like this:

A RuntimeError occurred in background at 2018-02-15 21:17:22 UTC :

Failed to contact proofing vendor
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/bundler/gems/identity-proofer-gem-adcd6e33942f/lib/proofer/vendor/mock.rb:200:in `fail_resolution_with_exception'

-------------------------------
Backtrace:
-------------------------------

/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/bundler/gems/identity-proofer-gem-adcd6e33942f/lib/proofer/vendor/mock.rb:200:in `fail_resolution_with_exception'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/bundler/gems/identity-proofer-gem-adcd6e33942f/lib/proofer/vendor/mock.rb:118:in `perform_resolution'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/bundler/gems/identity-proofer-gem-adcd6e33942f/lib/proofer/vendor/vendor_base.rb:20:in `start'
/Users/jonathanhooper/Developer/identity/idp/app/services/idv/agent.rb:3:in `start'
/Users/jonathanhooper/Developer/identity/idp/app/jobs/idv/profile_job.rb:6:in `verify_identity_with_vendor'
/Users/jonathanhooper/Developer/identity/idp/app/jobs/idv/proofer_job.rb:31:in `perform_identity_proofing'
/Users/jonathanhooper/Developer/identity/idp/app/jobs/idv/proofer_job.rb:12:in `perform'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/execution.rb:37:in `block in perform_now'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:108:in `block in run_callbacks'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/newrelic_rpm-4.8.0.341/lib/new_relic/agent/instrumentation/active_job.rb:50:in `perform'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/newrelic_rpm-4.8.0.341/lib/new_relic/agent/instrumentation/active_job.rb:22:in `block (3 levels) in '
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `instance_exec'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/i18n-0.9.4/lib/i18n.rb:268:in `with_locale'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/translation.rb:7:in `block (2 levels) in '
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `instance_exec'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/logging.rb:24:in `block (4 levels) in '
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `block in instrument'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `instrument'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/logging.rb:23:in `block (3 levels) in '
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/logging.rb:44:in `block in tag_logger'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/tagged_logging.rb:69:in `block in tagged'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/tagged_logging.rb:26:in `tagged'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/tagged_logging.rb:69:in `tagged'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/logging.rb:44:in `tag_logger'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/logging.rb:20:in `block (2 levels) in '
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `instance_exec'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:135:in `run_callbacks'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/execution.rb:33:in `perform_now'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/execution.rb:22:in `block in execute'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:108:in `block in run_callbacks'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/railtie.rb:26:in `block (4 levels) in '
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/execution_wrapper.rb:85:in `wrap'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/reloader.rb:68:in `block in wrap'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/execution_wrapper.rb:81:in `wrap'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/reloader.rb:67:in `wrap'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/railtie.rb:25:in `block (3 levels) in '
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `instance_exec'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:135:in `run_callbacks'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/execution.rb:20:in `execute'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activejob-5.1.4/lib/active_job/queue_adapters/sidekiq_adapter.rb:40:in `perform'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:188:in `execute_job'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/Users/jonathanhooper/Developer/identity/idp/lib/no_retry_jobs.rb:3:in `call'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/Users/jonathanhooper/Developer/identity/idp/lib/worker_health_checker.rb:9:in `call'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/newrelic_rpm-4.8.0.341/lib/new_relic/agent/instrumentation/sidekiq.rb:33:in `block in call'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/newrelic_rpm-4.8.0.341/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:369:in `perform_action_with_newrelic_trace'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/newrelic_rpm-4.8.0.341/lib/new_relic/agent/instrumentation/sidekiq.rb:29:in `call'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:169:in `block in process'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:141:in `block (6 levels) in dispatch'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_retry.rb:97:in `local'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:140:in `block (5 levels) in dispatch'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/rails.rb:42:in `block in call'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/execution_wrapper.rb:85:in `wrap'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/reloader.rb:68:in `block in wrap'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/execution_wrapper.rb:85:in `wrap'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-5.1.4/lib/active_support/reloader.rb:67:in `wrap'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/rails.rb:41:in `call'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:136:in `block (4 levels) in dispatch'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:204:in `stats'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:131:in `block (3 levels) in dispatch'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_logger.rb:7:in `call'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:130:in `block (2 levels) in dispatch'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_retry.rb:72:in `global'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:129:in `block in dispatch'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/logging.rb:44:in `with_context'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/logging.rb:38:in `with_job_hash_context'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:128:in `dispatch'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:168:in `process'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:85:in `process_one'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
/Users/jonathanhooper/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'

-------------------------------
Data:
-------------------------------

Job class: Idv::ProfileJob
Queue: idv
Retry: false
Error class: RuntimeError
Error message: Failed to contact proofing vendor
Created at: 2018-02-15 15:09:57 -0600
Enqueued at: 2018-02-15 15:17:21 -0600
Failed at: 2018-02-15 15:09:57 -0600
Retry count: 1

Copy link
Contributor

@tbaxter-18f tbaxter-18f left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems straightforward.

@jmhooper jmhooper merged commit a8ef159 into master Feb 15, 2018
@konklone konklone deleted the jmhooper-background-job-error-pii branch August 14, 2018 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants