Embedded Ansible cannot be activated in MiQ

[mhalaslenovo]

When I activate the embedded Ansible by going to into Configuration and turning on the Embedded Ansible switch, the only think I get is periodic message "The role Embedded Ansible has started activation on server EVM " but the activation doesn't seem to ever complete. The evm.log shows some errors below.

It seems the two directories the error messages refer to do not exist

[root@localhost log]# cd /etc/tower
-bash: cd: /etc/tower: No such file or directory
[root@localhost log]# cat /etc/sysconfig/ansible-tower
cat: /etc/sysconfig/ansible-tower: No such file or directory
[root@localhost log]# cat /etc/sysconfig/ansible*
cat: /etc/sysconfig/ansible*: No such file or directory

tail -f evm.log | grep -i EmbeddedAnsible

[----] I, [2017-07-27T11:40:51.182582 #1567:131913c] INFO -- : MIQ(MiqPriorityWorker::Runner#get_message_via_drb) Message id: [759], MiqWorker id: [14], Zone: [default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [3600], Priority: [20], State: [dequeue], Deliver On: [], Data: [], Args: [{:object_type=>"MiqServer", :object_id=>1, :attrs=>{:event_type=>"evm_worker_start", :event_details=>"Worker started: ID [106], PID [], GUID [b1444114-3577-4dae-8411-2b6671884183]", :type=>"EmbeddedAnsibleWorker", "MiqEvent::miq_event"=>125, :miq_event_id=>125, "EventStream::event_stream"=>125, :event_stream_id=>125}, :instance_name=>"Event", :user_id=>1, :miq_group_id=>2, :tenant_id=>1, :automate_message=>nil}], Dequeued in: [3.87337859] seconds
[----] I, [2017-07-27T11:40:51.186373 #1567:131913c] INFO -- : MIQ(MiqAeEngine.deliver) Delivering {:event_type=>"evm_worker_start", :event_details=>"Worker started: ID [106], PID [], GUID [b1444114-3577-4dae-8411-2b6671884183]", :type=>"EmbeddedAnsibleWorker", "MiqEvent::miq_event"=>125, :miq_event_id=>125, "EventStream::event_stream"=>125, :event_stream_id=>125} for object [MiqServer.1] with state [] to Automate
[----] I, [2017-07-27T11:41:00.897866 #1331:131913c] INFO -- : MIQ(EmbeddedAnsibleWorker.sync_workers) Workers are being synchronized: Current #: [0], Desired #: [1]
[----] I, [2017-07-27T11:41:01.004633 #1331:131913c] INFO -- : MIQ(MiqQueue.put) Message id: [760], id: [], Zone: [default], Role: [], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqEvent.raise_evm_event], Timeout: [600], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [["MiqServer", 1], "evm_worker_start", {:event_details=>"Worker started: ID [107], PID [], GUID [e0d68719-c381-4fc9-944e-b0ec1f804636]", :type=>"EmbeddedAnsibleWorker"}]
[----] I, [2017-07-27T11:41:01.005415 #1331:131913c] INFO -- : MIQ(EmbeddedAnsibleWorker#start) Worker started: ID [107], PID [], GUID [e0d68719-c381-4fc9-944e-b0ec1f804636]
[----] I, [2017-07-27T11:41:01.183108 #1331:753f7bc] INFO -- : MIQ(EmbeddedAnsibleWorker::Runner#sync_config) ID [107], PID [], GUID [e0d68719-c381-4fc9-944e-b0ec1f804636], Zone [default], Active Roles [automate,database_operations,database_owner,embedded_ansible,ems_inventory,ems_operations,event,reporting,scheduler,smartstate,user_interface,web_services,websocket], Assigned Roles [automate,database_operations,database_owner,embedded_ansible,ems_inventory,ems_operations,event,reporting,scheduler,smartstate,user_interface,web_services,websocket], Configuration:
[----] I, [2017-07-27T11:41:01.521895 #1331:753f7bc] INFO -- : MIQ(EmbeddedAnsibleWorker::Runner#setup_ansible) calling EmbeddedAnsible.start

[----] E, [2017-07-27T11:41:01.632186 #1331:753f7bc] ERROR -- : MIQ(EmbeddedAnsibleWorker::Runner) ID [107] PID [1331] GUID [e0d68719-c381-4fc9-944e-b0ec1f804636] Error in before_exit: source /etc/sysconfig/ansible-tower; echo $TOWER_SERVICES exit code: 1

[----] I, [2017-07-27T11:41:01.648894 #1331:753f7bc] INFO -- : MIQ(EmbeddedAnsibleWorker#log_status) [Embedded Ansible Worker] Worker ID [107], PID [], GUID [e0d68719-c381-4fc9-944e-b0ec1f804636], Last Heartbeat [2017-07-27 15:41:00 UTC], Process Info: Memory Usage [], Memory Size [], Proportional Set Size: [], Memory % [], CPU Time [], CPU % [], Priority []

[----] E, [2017-07-27T11:41:01.649260 #1331:753f7bc] ERROR -- : MIQ(EmbeddedAnsibleWorker::Runner) ID [107] PID [1331] GUID [e0d68719-c381-4fc9-944e-b0ec1f804636] /usr/bin/python -c "import uuid; file('/etc/tower/SECRET_KEY', 'wb').write(uuid.uuid4().hex)" exit code: 1 Worker exiting.

[----] I, [2017-07-27T11:41:01.649436 #1331:753f7bc] INFO -- : MIQ(EmbeddedAnsibleWorker#start_monitor_thread) SystemExit received, exiting monitoring Thread
[----] I, [2017-07-27T11:41:02.628006 #1558:131913c] INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [760], MiqWorker id: [13], Zone: [default], Role: [], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqEvent.raise_evm_event], Timeout: [600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [["MiqServer", 1], "evm_worker_start", {:event_details=>"Worker started: ID [107], PID [], GUID [e0d68719-c381-4fc9-944e-b0ec1f804636]", :type=>"EmbeddedAnsibleWorker"}], Dequeued in: [1.634094195] seconds

[carbonin]

This thread has some more information on how to set up embedded ansible in ManageIQ http://talk.manageiq.org/t/howto-setup-embedded-ansible/2291

[carbonin]

This should be fixed by #16205

[psachin]

@carbonin

On gaprindashvili-1, when I enable Embeded ansible role I see below errors in evm.log. If fix #16205 should be include in gaprindashvili-1, I think it fails to download docker image. Is it related to this issue or should I report new one?

[----] E, [2018-02-13T11:29:13.817306 #1709:c4bb728] ERROR -- : MIQ(EmbeddedAnsibleWorker::Runner) ID [201] PID [1709] GUID [4e14f4ef-fd54-402b-9591-65ab1ab591ff] No such image: rabbitmq:3
[----] E, [2018-02-13T11:29:45.852036 #1709:a686a64] ERROR -- : MIQ(EmbeddedAnsibleWorker::Runner) ID [202] PID [1709] GUID [c1200910-13f6-479d-bb20-1a3425fc7359] No such image: rabbitmq:3
[----] E, [2018-02-13T11:30:16.893742 #1709:7d58174] ERROR -- : MIQ(EmbeddedAnsibleWorker::Runner) ID [203] PID [1709] GUID [f30e12b6-4c3d-458d-8a18-cca365101259] No such image: rabbitmq:3
[----] E, [2018-02-13T11:30:48.836597 #1709:a67ead0] ERROR -- : MIQ(EmbeddedAnsibleWorker::Runner) ID [204] PID [1709] GUID [9418b190-be7b-46c3-96b1-260bf676ba22] No such image: rabbitmq:3
[----] E, [2018-02-13T11:31:20.301330 #1709:7d6afa4] ERROR -- : MIQ(EmbeddedAnsibleWorker::Runner) ID [205] PID [1709] GUID [cd214a7b-1773-45ba-88d0-362c7ad6753e] No such image: rabbitmq:3

[carbonin]

@psachin That looks like a separate issue, can you open a new issue and mention me?

This was originally opened because awx was not supported so this was solved when we added the ability to run awx containers for embedded ansible.