Skip to content

Commit

Permalink
Only queue ansible runner ops when not in pods
Browse files Browse the repository at this point in the history
This commit adds an intermediate method (#route_signal) to determine
if a call should be queued or not.

When running in containers, each generic worker is a separate container
so we can't queue anything between checking out the playbook
repository and cleaning it up. If we do, it might end up executing
on a container that doesn't have the repo checked out or isn't
running the ansible runner process.

We want to continue queueing these operations on appliances as the
previous reasoning doesn't apply (we will always queue for a worker
on the same server) and we still need to handle ansible playbooks
that might run longer than the timeout for a single queue message.

For now these long-running playbooks won't have a solution on pods,
but shorter ones will work.
  • Loading branch information
carbonin committed Mar 16, 2020
1 parent fbcce99 commit 734ec1c
Showing 1 changed file with 15 additions and 6 deletions.
21 changes: 15 additions & 6 deletions app/models/manageiq/providers/ansible_runner_workflow.rb
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ def start
def pre_execute
verify_options
prepare_repository
queue_signal(:execute)
route_signal(:execute)
end

def launch_runner
Expand All @@ -40,15 +40,15 @@ def execute
response = launch_runner

if response.nil?
queue_signal(:abort, "Failed to run ansible #{execution_type}", "error")
route_signal(:abort, "Failed to run ansible #{execution_type}", "error")
else
context[:ansible_runner_response] = response.dump

started_on = Time.now.utc
update!(:context => context, :started_on => started_on)
miq_task.update!(:started_on => started_on)

queue_signal(:poll_runner)
route_signal(:poll_runner)
end
end

Expand All @@ -58,9 +58,9 @@ def poll_runner
if started_on + options[:timeout] < Time.now.utc
response.stop

queue_signal(:abort, "ansible #{execution_type} has been running longer than timeout", "error")
route_signal(:abort, "ansible #{execution_type} has been running longer than timeout", "error")
else
queue_signal(:poll_runner, :deliver_on => deliver_on)
route_signal(:poll_runner, :deliver_on => deliver_on)
end
else
result = response.response
Expand All @@ -74,7 +74,7 @@ def poll_runner
else
set_status("ansible #{execution_type} completed with no errors", "ok")
end
queue_signal(:post_execute)
route_signal(:post_execute)
end
end

Expand All @@ -91,6 +91,15 @@ def post_execute

protected

def route_signal(*args, deliver_on: nil)
if MiqEnvironment::Command.is_podified?
sleep deliver_on if deliver_on
signal(*args)
else
queue_signal(*args, deliver_on: deliver_on)
end
end

def queue_signal(*args, deliver_on: nil)
role = options[:role] || "ems_operations"
priority = options[:priority] || MiqQueue::NORMAL_PRIORITY
Expand Down

0 comments on commit 734ec1c

Please sign in to comment.