On 1st chef-server-ctl reconfigure execute[bootstrap-platform] fails

[poliva83]

For some reason it can't yet connect to bifrost, but then on 2nd reconfigure it succeeds. Not sure what actually starts bifrost component (maybe its too slow to start up) but thinking about adding retries to execute block to see if it fixes my problem.

[2015-10-27T17:46:48+00:00] INFO: Processing template[/opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/config.rb] action create (private-chef::bootstrap line 23)
[2015-10-27T17:46:48+00:00] INFO: template[/opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/config.rb] created file /opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/config.rb
[2015-10-27T17:46:48+00:00] INFO: template[/opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/config.rb] updated file contents /opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/config.rb
[2015-10-27T17:46:48+00:00] INFO: template[/opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/config.rb] owner changed to 0
[2015-10-27T17:46:48+00:00] INFO: template[/opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/config.rb] group changed to 0
[2015-10-27T17:46:48+00:00] INFO: template[/opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/config.rb] mode changed to 600
[2015-10-27T17:46:48+00:00] INFO: Processing template[/opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/pivotal.yml] action create (private-chef::bootstrap line 32)
[2015-10-27T17:46:48+00:00] INFO: template[/opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/pivotal.yml] created file /opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/pivotal.yml
[2015-10-27T17:46:48+00:00] INFO: template[/opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/pivotal.yml] updated file contents /opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/pivotal.yml
[2015-10-27T17:46:48+00:00] INFO: template[/opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/pivotal.yml] owner changed to 0
[2015-10-27T17:46:48+00:00] INFO: template[/opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/pivotal.yml] group changed to 0
[2015-10-27T17:46:48+00:00] INFO: template[/opt/opscode/embedded/service/chef-server-bootstrap/bootstrapper-config/pivotal.yml] mode changed to 600
[2015-10-27T17:46:48+00:00] INFO: Processing execute[/opt/opscode/bin/private-chef-ctl start] action run (private-chef::bootstrap line 40)
[2015-10-27T17:46:54+00:00] INFO: execute[/opt/opscode/bin/private-chef-ctl start] ran successfully
[2015-10-27T17:46:54+00:00] INFO: Processing execute[bootstrap-platform] action run (private-chef::bootstrap line 45)

================================================================================
Error executing action `run` on resource 'execute[bootstrap-platform]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/opscode/embedded/bin/bundle exec ./bin/bootstrap-platform ./bootstrapper-config/config.rb ./bootstrapper-config/pivotal.yml ----
STDOUT:
STDERR: /opt/opscode/embedded/lib/ruby/2.1.0/net/http.rb:879:in `initialize': Connection refused - connect(2) for "127.0.0.1" port 9463 (Errno::ECONNREFUSED)
from /opt/opscode/embedded/lib/ruby/2.1.0/net/http.rb:879:in `open'
from /opt/opscode/embedded/lib/ruby/2.1.0/net/http.rb:879:in `block in connect'
from /opt/opscode/embedded/lib/ruby/2.1.0/timeout.rb:76:in `timeout'
from /opt/opscode/embedded/lib/ruby/2.1.0/net/http.rb:878:in `connect'
from /opt/opscode/embedded/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
from /opt/opscode/embedded/lib/ruby/2.1.0/net/http.rb:852:in `start'
from /opt/opscode/embedded/service/gem/ruby/2.1.0/gems/rest-client-1.6.8/lib/restclient/request.rb:206:in `transmit'
from /opt/opscode/embedded/service/gem/ruby/2.1.0/gems/rest-client-1.6.8/lib/restclient/request.rb:68:in `execute'
from /opt/opscode/embedded/service/gem/ruby/2.1.0/gems/rest-client-1.6.8/lib/restclient/request.rb:35:in `execute'
from /opt/opscode/embedded/service/gem/ruby/2.1.0/gems/rest-client-1.6.8/lib/restclient.rb:74:in `post'
from ./bin/bootstrap-platform:101:in `create_object_in_authz'
from ./bin/bootstrap-platform:47:in `bootstrap'
from ./bin/bootstrap-platform:120:in `<main>'
---- End output of /opt/opscode/embedded/bin/bundle exec ./bin/bootstrap-platform ./bootstrapper-config/config.rb ./bootstrapper-config/pivotal.yml ----
Ran /opt/opscode/embedded/bin/bundle exec ./bin/bootstrap-platform ./bootstrapper-config/config.rb ./bootstrapper-config/pivotal.yml returned 1

Resource Declaration:
---------------------
# In /opt/opscode/embedded/cookbooks/cache/cookbooks/private-chef/recipes/bootstrap.rb

45: execute "bootstrap-platform" do
46:   command "/opt/opscode/embedded/bin/bundle exec ./bin/bootstrap-platform ./bootstrapper-config/config.rb ./bootstrapper-config/pivotal.yml"
47:   cwd opscode_test_dir
48:   not_if { OmnibusHelper.has_been_bootstrapped? }
49:   notifies :restart, 'service[opscode-erchef]'
50: end
51:

Compiled Resource:
------------------
# Declared in /opt/opscode/embedded/cookbooks/cache/cookbooks/private-chef/recipes/bootstrap.rb:45:in `from_file'

execute("bootstrap-platform") do
 action [:run]
 retries 0
 retry_delay 2
 default_guard_interpreter :execute
 command "/opt/opscode/embedded/bin/bundle exec ./bin/bootstrap-platform ./bootstrapper-config/config.rb ./bootstrapper-config/pivotal.yml"
 backup 5
 cwd "/opt/opscode/embedded/service/chef-server-bootstrap"
 returns 0
 declared_type :execute
 cookbook_name "private-chef"
 recipe_name "bootstrap"
 not_if { #code block }
end

[meson10]

I am experiencing the same error.

[poliva83]

@meson10 Adding the retries on execute block fixed this issue for me. I'll send out a PR for it soon.

[marcparadise]

Do you have the original reconfigure output available? Bifrost startup is
pretty fast , so while a retry may get past the issue, there might be
something else we should be looking at.

Could you also let me know what version you're installing and on what
platform?

On Mon, Nov 2, 2015, 11:01 AM Phil Oliva [email protected] wrote:

@meson10 https://github.com/meson10 Adding the retries on execute block
fixed this issue for me. I'll send out a PR for it soon.

—
Reply to this email directly or view it on GitHub
#584 (comment).

[poliva83]

@marcparadise Unfortunately I have the output any more other then what I posted above. It was an intermittent issue for me. Didn't see it happen though after doing 3 retries on execute block.

[marcparadise]

@meson10 since you're able to reproduce this consistently - I am wondering what the specs are for memory and CPU on the machine/VM you're installing to?

[meson10]

@marcparadise Have tried it on smaller Virtual machines (2 core + 4GB RAM) as well as EC2 instance (r3.large) with 2 Core + 15.25G RAM.

[meson10]

@marcparadise I have figured the problem. So, accidentally, I was supplying

oc_bifrost['db_pool_size'] = ""

(observe the empty quotes)

[marcparadise]

The 2core+4gb should be good enough, so it's definitely not a resource constraint. Tonight or tomorrow morning I will try to reproduce in the same environment - though that's similar to what our CI pipleine uses, iirc, so it will be interesting to see how it behaves.

In the interim to get you unstuck, you could manually modify the file
`/opt/opscode/embedded/cookbooks/private-chef/recipes/bootstrap.rb prior t o running reconfigure, and make the change shown in @poliva83's pull request:

https://github.com/chef/chef-server/pull/597/files#diff-1f3de27173d5e39c86242170696a4f19L45

[marcparadise]

Aha! That'd definitely do it :D It would prevent bifrost from starting up.

[meson10]

Should I do a PR to have that check in place? To have a less cryptic error being reported.

[marcparadise]

While we definitely appreciate the offer, this is one of many places chef-server would fall down if a value was improperly configured so it would be a drop in the bucket. For almost any of the tunables, a bad value would cause a failure either in the startup of a service, or a relatively obscure chef run failure in reconfigure.

I think until somehwat recently, the idea was that for the most part, people wouldn't be touching these values without guidance from Chef Software support. However, over time this is just proving to be incorrect.

We've been talking about how we can improve that - one thing we've been considering is that we add validation parameters to everything that we expose as tunables so that we can comprehensively validate instead of selectively.

[meson10]

Got that. Fair point.

[satishpatro44]

What I didn't get is you download deb file, then dpkg, then
sudo chef-server-ctl reconfigure
as per official website

oc_bifrost['db_pool_size'] = ""
Then, in which file you have written/modified the above line?