Let queue workers process an active message

https://bugzilla.redhat.com/show_bug.cgi?id=1481800

In e5f4bd3, we added a 10 minute
timeout that would give workers a little time to complete their work
after they exceed their memory threshold before we'd kill them.  This
causes workers to be killed prematurely before completing the work item.

What we really want is for the work item to complete but kill the worker
if the worker has exceeded memory/time thresholds and the work item hasn't
completed in a reasonable time.  This reasonable time is the msg_timeout
associated with the queue message.

app/models/miq_worker.rb

@@ -442,7 +442,7 @@ def stopping_for_too_long?
     # the database, so we can see how long it's been
     # 'stopping' by checking the last_heartbeat.
     stopping_timeout = self.class.worker_settings[:stopping_timeout] || Workers::MiqDefaults.stopping_timeout
-    status == MiqWorker::STATUS_STOPPING && last_heartbeat < stopping_timeout.seconds.ago
+    status == MiqWorker::STATUS_STOPPING && (last_heartbeat + current_timeout.to_i) < stopping_timeout.seconds.ago
   end
 
   def validate_active_messages

spec/models/miq_worker_spec.rb

@@ -385,6 +385,20 @@ def check_has_required_role(worker_role_names, expected_result)
         expect(subject).to be_truthy
       end
 
+      it "true if stopping and last heartbeat is within the queue message timeout of an active message" do
+        @worker.messages << FactoryGirl.create(:miq_queue, :msg_timeout => 60.minutes)
+        @worker.update(:status         => described_class::STATUS_STOPPING,
+                       :last_heartbeat => 90.minutes.ago)
+        expect(subject).to be_truthy
+      end
+
+      it "false if stopping and last heartbeat is older than the queue message timeout of the work item" do
+        @worker.messages << FactoryGirl.create(:miq_queue, :msg_timeout => 60.minutes, :state => "dequeue")
+        @worker.update(:status         => described_class::STATUS_STOPPING,
+                       :last_heartbeat => 30.minutes.ago)
+        expect(subject).to be_falsey
+      end
+
       it "false if stopping and heartbeated recently" do
         @worker.update(:status         => described_class::STATUS_STOPPING,
                        :last_heartbeat => 1.minute.ago)