Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move WorkerHealthChecker from app/services to lib #1689

Merged
merged 1 commit into from
Sep 25, 2017

Conversation

monfresh
Copy link
Contributor

Why: Middleware, such as WorkerHealthChecker::Middleware can't be
reloaded, but because the WorkerHealthChecker module was placed in
app/services, it automatically gets reloaded by Rails, which results
in the following error when running Sidekiq jobs while Rails is trying
to clean up and unload code:

ArgumentError: A copy of WorkerHealthChecker::Middleware has been
removed from the module tree but is still active!

The solution is to move the module to a path that is not included in
the Rails autoload paths, such as lib.

A negative side effect of this issue is that any background job that
ran before Rails raised the error (when it was trying to clean up and
unload the code), will keep running over and over because Sidekiq
automatically retries failed jobs. I experienced this locally by
getting the same email over and over, and I'm assuming this is the
same issue that caused some of our production users to keep getting
the same SMS over and over. Jenny also saw this in INT in June.

@zachmargolis
Copy link
Contributor

Looks like we need to add a missing require now that this is outside of the autoload path.

bundler: failed to load command: rspec (/home/circleci/identity-idp/vendor/bundle/ruby/2.3.0/bin/rspec)
NameError: uninitialized constant WorkerHealthChecker
  /home/circleci/identity-idp/spec/rails_helper.rb:77:in `block in <top (required)>'
  /home/circleci/identity-idp/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.4/lib/sidekiq/testing.rb:55:in `server_middleware'
  /home/circleci/identity-idp/spec/rails_helper.rb:76:in `<top (required)>'

Also should we move the corresponding spec from spec/services to spec/lib ?

**Why**: Middleware, such as `WorkerHealthChecker::Middleware` can't be
reloaded, but because the `WorkerHealthChecker` module was placed in
`app/services`, it automatically gets reloaded by Rails, which results
in the following error when running Sidekiq jobs while Rails is trying
to clean up and unload code:

```
ArgumentError: A copy of WorkerHealthChecker::Middleware has been
removed from the module tree but is still active!
```
The solution is to move the module to a path that is not included in
the Rails autoload paths, such as `lib`.

A negative side effect of this issue is that any background job that
ran before Rails raised the error (when it was trying to clean up and
unload the code), will keep running over and over because Sidekiq
automatically retries failed jobs. I experienced this locally by
getting the same email over and over, and I'm assuming this is the
same issue that caused some of our production users to keep getting
the same SMS over and over. Jenny also saw this in INT in June.
@monfresh monfresh force-pushed the mb-fix-sidekiq-autoload-error branch from 17621cc to 89d6193 Compare September 25, 2017 15:02
@monfresh
Copy link
Contributor Author

Good catches. Fixed!

Copy link
Contributor

@zachmargolis zachmargolis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@monfresh monfresh merged commit 0e9286f into master Sep 25, 2017
@monfresh monfresh deleted the mb-fix-sidekiq-autoload-error branch September 25, 2017 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants