Wait condition timeout encountered when instantiating cluster template.

[vishwanathj]

When attempting to instantiate the https://github.com/F5Networks/f5-openstack-hot/blob/master/experimental/templates/cluster/nnic/prod_stack/static/f5_bigip_cluster_n_nic.yaml template, at the end, the Heat Stack fails due to WaitConditionTimeout, please see the output below for details.

2018-06-25 04:41:45Z [activeactive.wait_handle_static_mgmt_config_bigip1]: SIGNAL_COMPLETE Signal: status:SUCCESS reason:Setup-staticMgmt command exited without error.
2018-06-25 04:41:46Z [activeactive.wait_condition_static_mgmt_config_complete_bigip1]: CREATE_COMPLETE state changed
2018-06-25 04:41:51Z [activeactive.wait_handle_static_mgmt_config_bigip2]: SIGNAL_COMPLETE Signal: status:SUCCESS reason:Setup-staticMgmt command exited without error.
2018-06-25 04:41:52Z [activeactive.wait_condition_static_mgmt_config_complete_bigip2]: CREATE_COMPLETE state changed
2018-06-25 04:43:35Z [activeactive.wait_handle_onboard_bigip2]: SIGNAL_COMPLETE Signal: status:SUCCESS reason:Pre-onboard completed without error.
2018-06-25 04:43:36Z [activeactive.wait_handle_onboard_bigip1]: SIGNAL_COMPLETE Signal: status:SUCCESS reason:Pre-onboard completed without error.
2018-06-25 04:48:48Z [activeactive.wait_handle_onboard_network_bigip2]: SIGNAL_COMPLETE Signal: status:SUCCESS reason:Onboard-network command exited without error.
2018-06-25 04:48:49Z [activeactive.wait_condition_onboard_network_complete_bigip2]: CREATE_COMPLETE state changed
2018-06-25 04:49:09Z [activeactive.wait_handle_onboard_network_bigip1]: SIGNAL_COMPLETE Signal: status:SUCCESS reason:Onboard-network command exited without error.
2018-06-25 04:49:10Z [activeactive.wait_condition_onboard_network_complete_bigip1]: CREATE_COMPLETE state changed
2018-06-25 04:49:47Z [activeactive.wait_handle_onboard_cluster_bigip1]: SIGNAL_COMPLETE Signal: status:SUCCESS reason:Onboard-cluster command exited without error. *** Instance: host-df9586ad-3dd5-4ca7-8ec6-effb9833133f.localhost
2018-06-25 04:49:48Z [activeactive.wait_condition_onboard_cluster_complete_bigip1]: CREATE_COMPLETE state changed
2018-06-25 04:49:53Z [activeactive.wait_handle_onboard_bigip1]: SIGNAL_COMPLETE Signal: status:SUCCESS reason:Post-onboard completed without error.
2018-06-25 04:51:06Z [activeactive.wait_handle_onboard_cluster_bigip2]: SIGNAL_COMPLETE Signal: status:SUCCESS reason:Onboard-cluster command exited without error. *** Instance: host-a46ec1c9-0e8e-453b-8e55-a4e9ca6a64a4.localhost
2018-06-25 04:51:06Z [activeactive.wait_condition_onboard_cluster_complete_bigip2]: CREATE_COMPLETE state changed
2018-06-25 04:51:20Z [activeactive.wait_handle_onboard_bigip2]: SIGNAL_COMPLETE Signal: status:SUCCESS reason:Post-onboard completed without error.
2018-06-25 05:23:00Z [activeactive.wait_condition_onboard_complete_bigip1]: CREATE_FAILED WaitConditionTimeout: resources.wait_condition_onboard_complete_bigip1: 2 of 3 received - Pre-onboard completed without error.;Post-onboard completed without error.
2018-06-25 05:23:00Z [activeactive]: CREATE_FAILED Resource CREATE failed: WaitConditionTimeout: resources.wait_condition_onboard_complete_bigip1: 2 of 3 received - Pre-onboard completed without error.;Post-onboard completed without error.
2018-06-25 05:23:14Z [activeactive.wait_condition_onboard_complete_bigip2]: CREATE_FAILED WaitConditionTimeout: resources.wait_condition_onboard_complete_bigip2: 2 of 3 received - Pre-onboard completed without error.;Post-onboard completed without error.
2018-06-25 05:23:15Z [activeactive]: CREATE_FAILED Resource CREATE failed: WaitConditionTimeout: resources.wait_condition_onboard_complete_bigip2: 2 of 3 received - Pre-onboard completed without error.;Post-onboard completed without error.

[kaustriaf5]

What does openstack stack output {stack_name} show return? You should be able to to get the log file for which script heat signal was not received, which looks like it might be from onboard.sh. Also try looking at /var/log/cloud/openstack/runScript.log.

[vishwanathj]

Find attached the log file locations for onboard.log and runScript.log for BIG IP mgmt IP ending with 83 below:
http://paste.openstack.org/show/724252/ --->onboard.log
http://paste.openstack.org/show/724253/ ---->runScript.log

Find attached the log file locations for onboard.log and runScript.log for BIG IP mgmt IP ending with 86 below:
http://paste.openstack.org/show/724254/ --> onboard.log
http://paste.openstack.org/show/724255/ --> runScript.log

The output of "openstack stack output show activeactive --all" is at http://paste.openstack.org/show/724256/

[kaustriaf5]

Thanks @vishwanathj
What does openstack stack {stack_name} show?

I'm not sure if the onboard.log was cut off, but it looks like it didn't completely finish licensing - did the license activation go through?

[vishwanathj]

I did not supply license keys, so license activation would have failed

[kaustriaf5]

@vishwanathj, per the README you should be specifying NO_LIC as the bigip_licensing_type. You had specified BYOL.

Closing this issue - feel free to reopen if you see the same issue once you make that change.

[vishwanathj]

After having specified 2 license keys in the heat environment files, I still that the heat stack has failed.

[kaustriaf5]

What's in onboard.log?