Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chef_client provisioner fails to cleanup node on chef server #2476

Closed
pharaujo opened this issue Nov 8, 2013 · 37 comments
Closed

chef_client provisioner fails to cleanup node on chef server #2476

pharaujo opened this issue Nov 8, 2013 · 37 comments

Comments

@pharaujo
Copy link

pharaujo commented Nov 8, 2013

I am having trouble cleaning up the node on chef server when destroying a vagrant instance.

My config:

  config.vm.provision :chef_client do |chef|
    chef.chef_server_url = "https://internalchefserver"
    chef.validation_key_path = "./pedro.pem"
    chef.validation_client_name = "pedro"
    chef.delete_node = "true"

    chef.add_recipe "base"
  end

Destroying gives a stack trace:

$ vagrant -v
Vagrant 1.3.5
$ vagrant destroy -f
[default] Forcing shutdown of VM...
[default] Destroying VM and associated drives...
[default] Running cleanup tasks for 'chef_client' provisioner...
Deleting node "#<Vagrant::Config::V2::DummyConfig:0x00000100ce87f8>" from Chef server...
/Applications/Vagrant/embedded/gems/gems/childprocess-0.3.9/lib/childprocess/abstract_process.rb:36:in `initialize': all arguments must be String: ["/Users/pedroaraujo/.rvm/gems/ruby-1.9.3-p448/bin/knife", "node", "delete", "--yes", #<Vagrant::Config::V2::DummyConfig:0x00000100ce87f8>] (ArgumentError)
<omitted stack trace>

I had a look at the source code and it seems node_name is not defined, so I added the following to my config:

    chef.node_name = config.vm.hostname

This almost worked, but it seems knife tries to use the embedded GEM_PATH and fails to load its core dependencies:

$ vagrant destroy -f
[default] Forcing shutdown of VM...
[default] Destroying VM and associated drives...
[default] Running cleanup tasks for 'chef_client' provisioner...
Deleting node "vagranttest" from Chef server...
Cleaning up the 'node' for Chef failed. The stdout and
stderr are shown below. Vagrant will continue destroying the machine,
so please clean up these resources manually.

stdout:

stderr: /Applications/Vagrant/embedded/lib/ruby/1.9.1/rubygems/dependency.rb:247:in `to_specs': Could not find chef (>= 0) amongst [builder-3.2.2, childprocess-0.3.9, erubis-2.7.0, ffi-1.9.0, i18n-0.6.5, i18n-0.6.4, log4r-1.1.10, mini_portile-0.5.1, net-scp-1.1.2, net-ssh-2.6.8, nokogiri-1.6.0, rbvmomi-1.6.0, trollop-2.0, vagrant-1.3.5, vagrant-1.2.3, vagrant-cachier-0.4.1, vagrant-vsphere-0.0.1] (Gem::LoadError)
    from /Applications/Vagrant/embedded/lib/ruby/1.9.1/rubygems/dependency.rb:256:in `to_spec'
    from /Applications/Vagrant/embedded/lib/ruby/1.9.1/rubygems.rb:1231:in `gem'
    from /Users/pedro/.rvm/gems/ruby-1.9.3-p448/bin/knife:22:in `<main>'
    from /Users/pedro/.rvm/gems/ruby-1.9.3-p448/bin/ruby_noexec_wrapper:14:in `eval'
    from /Users/pedro/.rvm/gems/ruby-1.9.3-p448/bin/ruby_noexec_wrapper:14:in `<main>'

@mitchellh
Copy link
Contributor

The first issue you saw should be fixed. The second is a bit more involved... I've tagged this as a bug.

@scalp42
Copy link
Contributor

scalp42 commented Dec 10, 2013

I have the same issue with latest 1.4.0:

Cleaning up the 'node' for Chef failed. The stdout and
stderr are shown below. Vagrant will continue destroying the machine,
so please clean up these resources manually.

stdout:

stderr: /Applications/Vagrant/embedded/lib/ruby/2.0.0/rubygems/dependency.rb:296:in `to_specs': Could not find 'chef' (>= 0) among 79 total gem(s) (Gem::LoadError)
  from /Applications/Vagrant/embedded/lib/ruby/2.0.0/rubygems/dependency.rb:307:in `to_spec'
  from /Applications/Vagrant/embedded/lib/ruby/2.0.0/rubygems/core_ext/kernel_gem.rb:47:in `gem'
  from /Users/scalp/.rvm/gems/ruby-1.9.3-p484@chef/bin/knife:22:in `<main>'
  from /Users/scalp/.rvm/gems/ruby-1.9.3-p484@chef/bin/ruby_executable_hooks:15:in `eval'
  from /Users/scalp/.rvm/gems/ruby-1.9.3-p484@chef/bin/ruby_executable_hooks:15:in `<main>'

@tmatilai
Copy link
Contributor

Both stack traces include RVM ruby and Vagrant's embedded ruby libraries/gems. Feels like Vagrant sets the gem environment so that when it executes knife, it won't find the RVM gems but uses the Vagrant's embedded ones.

@tmatilai
Copy link
Contributor

A dirty workaround that might work would be to install chef to Vagrant's gem/plugin store. Something like:

/Applications/Vagrant/embedded/bin/gem install -i ~/.vagrant.d/gems chef

Note that it probably gets uninstalled with next vagrant plugin operation.

@scalp42
Copy link
Contributor

scalp42 commented Dec 10, 2013

Thanks @tmatilai I got around the gem issue, but the node/client stay on the Chef server unfortunately.

I have chef.node_name matching exactly but it doesn't work.

@tmatilai
Copy link
Contributor

@scalp42 Could you run with --debug and gist the output? You can also try running the executed knife command manually with -VV to see if it prints something useful.

@scalp42
Copy link
Contributor

scalp42 commented Dec 10, 2013

My node/client name is zs-common-scalp-vagrant-vbox.

The knife commands run fine and delete the node:

knife node delete zs-common-scalp-vagrant-vbox -y
knife client delete zs-common-scalp-vagrant-vbox -y

My Vagrantfile looks like this:

config.vm.define :zs1 do |zs|
    zs.vm.hostname = 'zs-common-scalp-vagrant-vbox'

    zs.vm.provision :chef_client do |chef|
      chef.chef_server_url = ENV['KNIFE_CHEF_SERVER']
      chef.validation_client_name = ENV['KNIFE_VALIDATION_CLIENT_NAME']
      chef.validation_key_path = ENV['KNIFE_VALIDATION_CLIENT_KEY']
      chef.log_level = 'info'
      chef.node_name = 'zs-common-scalp-vagrant-vbox'
    end
end

Here the gist on vagrant destroy -f --debug: https://gist.github.com/scalp42/783533c48d57a9e9bc83

@tmatilai
Copy link
Contributor

@scalp42 You need to explicitly enable Chef client and node deletion:

  chef.delete_client = true
  chef.delete_node = true

@scalp42
Copy link
Contributor

scalp42 commented Dec 10, 2013

Derp. Sorry reverted my changes. Let me re-add it.

@tmatilai
Copy link
Contributor

Btw, this seems to be totally undocumented feature. I'll offer a 🍌 for a PR! Anyone?

@mitchellh
Copy link
Contributor

@tmatilai Keep it undocumented because it doesn't work at the moment due to various path issues... hence the issue. :(

@tmatilai
Copy link
Contributor

@mitchellh Ah, gotcha. I don't use the chef-client provider so I had no idea of its state. Just remember when it was mentioned in some release notes.

@scalp42
Copy link
Contributor

scalp42 commented Dec 10, 2013

opened this, feel free to close it #2627

@scalp42
Copy link
Contributor

scalp42 commented Dec 10, 2013

@tmatilai

/Applications/Vagrant/embedded/bin/gem install -i ~/.vagrant.d/gems chef did the trick for me.

Thanks!

@scalp42
Copy link
Contributor

scalp42 commented Dec 10, 2013

The behavior I was seeing was with vagrant-aws actually not cleaning up if on EC2. Will open a issue there mitchellh/vagrant-aws#170

@dgm
Copy link

dgm commented Dec 12, 2013

I too hit this snag, trying to use vagrant with chef.delete_node ... The above hack works, but it hardly looks like the correct fix. My host machine is using RVM ...

@pharaujo
Copy link
Author

For working around this issue I patch the chef_client provisioner to force knife to use them correct GEM_PATH, but it's hardly a fix:

diff --git a/plugins/provisioners/chef/provisioner/chef_client.rb b/plugins/provisioners/chef/provisioner/chef_client.rb
index 30913b0..b894f84 100644
--- a/plugins/provisioners/chef/provisioner/chef_client.rb
+++ b/plugins/provisioners/chef/provisioner/chef_client.rb
@@ -118,7 +118,7 @@ module VagrantPlugins
             deletable: deletable, name: node_name))

           command = ["knife", deletable, "delete", "--yes", node_name]
-          r = Vagrant::Util::Subprocess.execute(*command)
+          r = Vagrant::Util::Subprocess.execute(*command, {:env => {'GEM_PATH' => '/Users/pedro/.rvm/gems/ruby-1.9.3-p448:/Users/pedro/.rvm/gems/ruby-1.9.3-p448@global'}})
           if r.exit_code != 0
             @machine.env.ui.error(I18n.t(
               "vagrant.chef_client_cleanup_failed",

If only the original GEM_PATH was exposed to plugins, this would be a lot cleaner. Maybe by creating a new environment variable ORIGINAL_GEM_PATH in with_environment and passing that to the knife subprocess as GEM_PATH? Feels like a hack as well...

@ostap36
Copy link

ostap36 commented Feb 12, 2014

Still getting the same error with Vagrant 1.4.3 (running on windows 7). Is the work around ^ the only "fix" for now?

stderr: C:/opscode/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/dependency.rb:247:in `to_specs': Could not find chef (>= 0) amongst [childprocess-0.3.9, erubis-2.7.0, ffi-1.9.3-x86-mingw32, i18n-0.6.9, log4r-1.1.10, net-scp-1.1.2, net-ssh-2.7.0, vagrant-1.4.3] (Gem::LoadError)
        from C:/opscode/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/dependency.rb:256:in `to_spec'
        from C:/opscode/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems.rb:1231:in `gem'
        from C:/opscode/chef/bin/knife:22:in `<main>'

@thmsobrmlr
Copy link

I have the same issue and could get it running with the workaround mentioned by @tmatilai.

@tmatilai
Copy link
Contributor

Fyi, last night I spiked to replace the current shell-out implementation with using ChefAPI gem and stealing ideas from vagrant-butcher. It looks promising, and I try to get a preliminary PR out soon.

@christinedraper
Copy link

I have identical error to ostap36, running Vagrant 1.4.3 on Windows 8.1. I tried the workaround but it didnt work.

@nefilim
Copy link

nefilim commented Feb 26, 2014

+1 on fixing this

@ju2wheels
Copy link

Still having this problem with 1.5.2 on Ubuntu 12.04 x86_64, but the error for me looks a bit different than the above:

Cleaning up the 'client' for Chef failed. The stdout and
stderr are shown below. Vagrant will continue destroying the machine,
so please clean up these resources manually.

stdout: 

stderr: /opt/vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler/rubygems_integration.rb:240:in `block in replace_gem': chef is not part of the bundle. Add it to Gemfile. (Gem::LoadError)
    from /usr/bin/knife:22:in `<main>'

@whiteadam
Copy link

Same here with 1.5.2 on Mavericks

Deleting client "webserver01" from Chef server...
Cleaning up the 'client' for Chef failed. The stdout and
stderr are shown below. Vagrant will continue destroying the machine,
so please clean up these resources manually.

stdout:

stderr: /Applications/Vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler/rubygems_integration.rb:240:in `block in replace_gem': chef is not part of the bundle. Add it to Gemfile. (Gem::LoadError)
    from /Users/adam/.rbenv/versions/1.9.3-p545/bin/knife:22:in `<main>'

Deleting node "webserver01" from Chef server...
Cleaning up the 'node' for Chef failed. The stdout and
stderr are shown below. Vagrant will continue destroying the machine,
so please clean up these resources manually.

stdout:

stderr: /Applications/Vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler/rubygems_integration.rb:240:in `block in replace_gem': chef is not part of the bundle. Add it to Gemfile. (Gem::LoadError)
    from /Users/adam/.rbenv/versions/1.9.3-p545/bin/knife:22:in `<main>'

@chrissnell
Copy link

+1 Hope to see a fix. The fix from @tmatilai did not work for me:

Deleting client "mynode" from Chef server...
Cleaning up the 'client' for Chef failed. The stdout and
stderr are shown below. Vagrant will continue destroying the machine,
so please clean up these resources manually.

stdout:

stderr: /Applications/Vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler/spec_set.rb:92:in block in materialize': Could not find json-1.7.7 in any of the sources (Bundler::GemNotFound) from /Applications/Vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler/spec_set.rb:85:inmap!'
from /Applications/Vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler/spec_set.rb:85:in materialize' from /Applications/Vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler/definition.rb:133:inspecs'
from /Applications/Vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler/definition.rb:178:in specs_for' from /Applications/Vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler/definition.rb:167:inrequested_specs'
from /Applications/Vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler/environment.rb:18:in requested_specs' from /Applications/Vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler/runtime.rb:13:insetup'
from /Applications/Vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler.rb:119:in setup' from /Applications/Vagrant/embedded/gems/gems/bundler-1.5.3/lib/bundler/setup.rb:17:in<top (required)>'
from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb:55:in require' from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb:55:inrequire'

@nshenry03
Copy link

Has anyone discovered a workaround for Ubuntu Users? I tried the following

/opt/vagrant/embedded/bin/gem install --no-ri --no-rdoc --install-dir ~/.vagrant.d/gems chef

however, I still seem to be having the same problem

==> default: Running cleanup tasks for 'chef_client' provisioner...
==> default: Deleting client "nickh-devbox" from Chef server...
==> default: Cleaning up the 'client' for Chef failed. The stdout and
==> default: stderr are shown below. Vagrant will continue destroying the machine,
==> default: so please clean up these resources manually.
==> default: 
==> default: stdout: 
==> default: 
==> default: stderr: /opt/vagrant/embedded/gems/gems/bundler-1.6.2/lib/bundler/rubygems_integration.rb:252:in `block in replace_gem': chef is not part of the bundle. Add it to Gemfile. (Gem::LoadError)
==> default:    from /home/nickh/.rvm/gems/ruby-2.1.0/bin/knife:22:in `<main>'
==> default:    from /home/nickh/.rvm/gems/ruby-2.1.0/bin/ruby_executable_hooks:15:in `eval'
==> default:    from /home/nickh/.rvm/gems/ruby-2.1.0/bin/ruby_executable_hooks:15:in `<main>'
==> default: Deleting node "nickh-devbox" from Chef server...
==> default: Cleaning up the 'node' for Chef failed. The stdout and
==> default: stderr are shown below. Vagrant will continue destroying the machine,
==> default: so please clean up these resources manually.
==> default: 
==> default: stdout: 
==> default: 
==> default: stderr: /opt/vagrant/embedded/gems/gems/bundler-1.6.2/lib/bundler/rubygems_integration.rb:252:in `block in replace_gem': chef is not part of the bundle. Add it to Gemfile. (Gem::LoadError)
==> default:    from /home/nickh/.rvm/gems/ruby-2.1.0/bin/knife:22:in `<main>'
==> default:    from /home/nickh/.rvm/gems/ruby-2.1.0/bin/ruby_executable_hooks:15:in `eval'
==> default:    from /home/nickh/.rvm/gems/ruby-2.1.0/bin/ruby_executable_hooks:15:in `<main>'

Any suggestions would be appreciated!

@seburban
Copy link

I was having the same issue - I installed the vagrant-butcher plugin and was able to work around the problem (https://github.com/cassianoleal/vagrant-butcher/). I've temporarily disabled the chef.delete_client and chef.delete_node properties until a stable fix is released.

@medyagh
Copy link

medyagh commented Jun 9, 2014

any update on this issue ? is it gonna be fixed or do we need to use workarounds?

@frizop
Copy link

frizop commented Nov 4, 2014

@mitchellh are we sure this is related to the provisioner? I get a very similar error with puppet and the r10k-vagrant plugin and rbenv on a mac.

Bringing machine 'build' up with 'virtualbox' provider...
==> build: vagrant-r10k: Beginning r10k deploy of puppet modules into /Users/frizop/Documents/vagrant-windows/modules using /Users/frizop/Documents/vagrant-windows/Puppetfile
[R10K::TaskRunner - ERROR] Task #<R10K::Task::Module::Sync:0x00000100b29408> failed while running: Command puppet module --modulepath /Users/frizop/Documents/vagrant-windows/modules install --force rismoney/chocolatey exited with 1: /Applications/Vagrant/embedded/gems/gems/bundler-1.6.6/lib/bundler/rubygems_integration.rb:256:in `block in replace_gem': puppet is not part of the bundle. Add it to Gemfile. (Gem::LoadError)
    from /usr/local/var/rbenv/versions/2.0.0-p247/bin/puppet:22:in `<main>'

RuntimeError: Command puppet module --modulepath /Users/frizop/Documents/vagrant-windows/modules install --force rismoney/chocolatey exited with 1: /Applications/Vagrant/embedded/gems/gems/bundler-1.6.6/lib/bundler/rubygems_integration.rb:256:in `block in replace_gem': puppet is not part of the bundle. Add it to Gemfile. (Gem::LoadError)
    from /usr/local/var/rbenv/versions/2.0.0-p247/bin/puppet:22:in `<main>'

I've posted a full debug output to the vagrant plugin developer who seems fairly stumped: jantman/vagrant-r10k#2

@sethvargo
Copy link
Contributor

@fpletz the issue is generally "shelling out from one Ruby environment into another Ruby environment is almost 100% impossible", so yes, puppet would be affected as well if you have puppet installed like that.

@rnelson0
Copy link

rnelson0 commented Nov 4, 2014

@sethvargo As a bystander, I'm curious - WHICH Gemfile would need updated to resolve the issue?

@sethvargo
Copy link
Contributor

@rnelson0 it's not a Gemfile issue, but a "shelling out" issue. Some people are using Ruby versions managers (like rbenv, chruby), which mangle themselves into the $PATH when a new shell is spawned. We can unset all the bundler-environment variables we want, but that's not going to fix this issue sadly 😦.

@pharaujo
Copy link
Author

pharaujo commented Nov 4, 2014

Wouldn't saving the original GEMPATH on a global variable and have that available for the shell out work? (as I suggested on #2476 (comment))

@sethvargo
Copy link
Contributor

@pharaujo no, because Vagrant runs in an entirely different Ruby than these other gemsets

@pharaujo
Copy link
Author

pharaujo commented Nov 5, 2014

@sethvargo I know, but that should be irrelevant for the shell out. The problem here is that the GEMPATH set by Vagrant - which contains the embedded gems - does not have knife or its required dependencies and so it fails. If you make available the system GEMPATH to the Vagrant plugins, they can use that in their shell outs.
E.g:

Vagrant::Util::Subprocess.execute(*command, {:env => { 'GEM_PATH' => ORIGINAL_GEM_PATH }})

@frizop
Copy link

frizop commented Nov 5, 2014

Just for clarity sake, running OS X 10.10's system ruby yields the same error. Gist found here: https://gist.github.com/4c8c7cb523eeda3f9816

@sethvargo
Copy link
Contributor

I think, for once and for all, this is fixed by #5912 😄.

Unfortunately this is a generally unsolved problem and each Ruby version manager behaves slightly differently. With the complex behavior of Bundler, individual system settings, and star/moon alignment, it is possible that whatever incantation of Ruby + Chef + other rubies on the system that this might still persist for a small number of users. We have chosen to optimize and test for the "common" use case, which is ChefDK, using Vagrant in the official installer and using Vagrant as a bundled gem. I also tested rbenv and chruby (for the Chef installation) and was able to get things working.

I am marking this issue as fixed, while recognizing that certain situations may not be accounted for. Unfortunately we just can't make this work perfectly for everyone. If you are experiencing issues, please open a new issue and include your Ruby setup, Chef installation, Vagrant version, and the debug output. Moving forward, we will likely only support bug reports as they relate to the official installation processes for both tools (ChefDK and Vagrant's official installer).

This will be released in Vagrant 1.7.3

@hashicorp hashicorp locked and limited conversation to collaborators Jul 9, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.