chef-server-ctl cannot load such file -- chef/key (LoadError)

[dgoradia]

Running just chef-server-ctl or chef-server-ctl help produces the error:

[root@chefserver ~]# chef-server-ctl 
/opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/omnibus-ctl-0.4.2/lib/omnibus-ctl.rb:173:in `eval': cannot load such file -- chef/key (LoadError)
    from /opt/opscode/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
    from (eval):20:in `load_file'
    from /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/omnibus-ctl-0.4.2/lib/omnibus-ctl.rb:173:in `eval'
    from /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/omnibus-ctl-0.4.2/lib/omnibus-ctl.rb:173:in `load_file'
    from /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/omnibus-ctl-0.4.2/lib/omnibus-ctl.rb:168:in `block in load_files'
    from /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/omnibus-ctl-0.4.2/lib/omnibus-ctl.rb:167:in `each'
    from /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/omnibus-ctl-0.4.2/lib/omnibus-ctl.rb:167:in `load_files'
    from /opt/opscode/embedded/service/omnibus-ctl/chef-server-ctl:190:in `<main>'

I need to reconfigure to update an SSL certificate but I cannot do that as it gives the same error as above.

[sdelano]

@dgoradia - is this a new install of the Chef Server? Additionally , what version are you using.

[dgoradia]

No it's not a new install.
Version 12.2.0.

[sdelano]

It appears that something has changed on the server since the initial install / configuration. Does your PATH have another ruby,bundler, or gem in it? My hunch is that chef-server-ctl may be trying to require files from an older version of Chef.

[sdelano]

Since it appears that chef-server-ctl makes a good effort to shield you from environment-inflicted errors here, can you provide the output of /opt/opscode/embedded/bin/gem list chef?

[dgoradia]

No other ruby/bundle/gem installed.

# echo $PATH
/opt/opscode/embedded/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

# /opt/opscode/embedded/bin/gem list chef

*** LOCAL GEMS ***

chef (12.3.0)
chef-vault (2.6.1, 2.6.0, 2.5.0)
chef-vault-testfixtures (0.5.0)
chef-zero (4.3.0, 4.2.3)
chef_backup (0.0.1.dev.4)
cheffish (1.4.2, 1.3.1)

[sdelano]

That's an interesting output. This is what I get on a fresh install of Chef Server 12.2.0:

root@default-ubuntu-1204:~/chef-server/omnibus/pkg# /opt/opscode/embedded/bin/gem list chef

*** LOCAL GEMS ***

chef (12.5.0.current.0, 12.4.1)
chef-config (12.5.0.current.0, 12.5.0.alpha.1, 12.4.1)
chef-zero (4.2.3)
chef_backup (0.0.1.dev.4)
cheffish (1.3.1)

It's also interesting to see chef-vault installed there. A few more questions:

• Is /opt/opscode/embedded/bin always on your path?
• Is this machine configured with Chef?
• I'm concerned that something may have polluted the gems embedded within Chef Server, especially since chef-vault ended up there somehow.

[dgoradia]

/opt/opscode/embedded/bin is always in my path.
This machine is dedicated to Chef Server, nothing else is on it.
Looking at my bash history I may have accidentally typed it into a terminal logged into chefserver. I have gem install chef-vault in my history so I'm pretty sure I did that and not another application.

[dgoradia]

chef/key didn't exist in chef 12.2.0 so I installed the chef-12.4.0 gem and ran reconfigure which worked. I was able to upgrade to 12.4.1 as shown in the documentation.