Installation fails: Error executing action start on resource 'service[elasticsearch]'

[liquid1982]

I'm trying to install the elasticsearch cookbook using Chef Solo (Ubuntu 12.04 LTS 64bit) and it is failing during the execution this resource declaration (in the default recipe):

  * service[elasticsearch] action start
================================================================================
Error executing action `start` on resource 'service[elasticsearch]'
================================================================================


Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '127'
---- Begin output of /etc/init.d/elasticsearch start ----
STDOUT: Starting elasticsearch...
STDERR: bash: /usr/local/bin/elasticsearch: No such file or directory
---- End output of /etc/init.d/elasticsearch start ----
Ran /etc/init.d/elasticsearch start returned 127


Resource Declaration:
---------------------
# In /home/vincenzo/chef-solo/cookbooks-1/elasticsearch/recipes/default.rb

 61: service "elasticsearch" do
 62:   supports :status => true, :restart => true
 63:   action [ :enable ]
 64: end
 65:

because the path to the binary is wrong.
I'm stuck. Any ideas?

Thanks,
Vincenzo

[Pneumatus]

This bug appears to be due to a regression in the 'ark' cookbook (http://tickets.opscode.com/browse/COOK-3053).

We end up with /usr/local/bin/elasticsearch and /usr/local/bin/plugin being created as broken symlinks.

[leknarf]

I had the same problem. I worked around it by reverting the 'ark' cookbook to version 0.1.0. It's commit "084aef1".

[liquid1982]

You did that using Berkshelf or Librarian? Or you simply copied the files inside the ark cookbook directory?

[av-ast]

@leknarf thank you so much! It works for me.

@liquid1982 I simply added version 0.1.0 into my Berksfile:

cookbook 'ark', github: 'opscode-cookbooks/ark', ref: '0.1.0'

[cgravier]

Thanks a lot !
Using ark 0.1.0 does solve this issue that I also had.

[gwhitelaw]

Cheers @av-ast!

[karmi]

@liquid1982 @leknarf Can you provide more info here, if you're using any configuration different from the defaults? I'm looking into the issue right now, and using bryanwb/chef-ark@991613e, the installation in Vagrant is fine:

vagrant@precise64:~$ ls -la /usr/local/bin/elasticsearch
lrwxrwxrwx 1 root root 49 Jun 10 13:30 /usr/local/bin/elasticsearch -> /usr/local/elasticsearch-0.20.6/bin/elasticsearch

vagrant@precise64:~$ ls -la /usr/local/elasticsearch
lrwxrwxrwx 1 root root 31 Jun 10 13:30 /usr/local/elasticsearch -> /usr/local/elasticsearch-0.20.6

[leknarf]

@karmi, You're correct, the installation runs successfully using the bryanwb/chef-ark cookbook.

But it fails if you try to use https://github.com/opscode-cookbooks/ark, which is the default.

cookbook 'ark', github: 'opscode-cookbooks/ark', ref: '0.2.4' # this will fail
cookbook 'ark' # this fails too

cookbook 'ark', github: 'bryanwb/chef-ark' # this will work

[karmi]

@leknarf Aaargh, sure, I always had Bryan's version in the Berksfile. Seems like opscode-cookbooks/ark superseded that original repository. I think they have fixed the issue in sous-chefs/ark@0ddcd48, will try it out.

[karmi]

@leknarf This is weird, just tried it out in a fresh Vagrant with:

# Berksfile
cookbook 'ark',  git: 'git://github.com/opscode-cookbooks/ark.git', ref: '0.2.4'

and the Chef run goes fine, and the bin seems to be fine:

vagrant@precise64:~$ which elasticsearch
/usr/local/bin/elasticsearch
vagrant@precise64:~$ ll /usr/local/bin/elasticsearch
lrwxrwxrwx 1 root root 49 Jun 10 15:14 /usr/local/bin/elasticsearch -> /usr/local/elasticsearch-0.20.6/bin/elasticsearch*

Can you maybe try it once again with ark 0.2.4 from opscode-cookbooks?

[liquid1982]

@karmi I also tried with Vagrant and everything is ok.

I had this issue launching a bundle exec knife solo bootstrap <vm> on a fresh virtual machine, and if you don't change that ark dependency, elasticsearch is not correctly symlinked.

[karmi]

@liquid1982 Ah, OK, how does knife solo resolve the cookbook versions? Does it use Berksfile / Cheffile?

[liquid1982]

knife solo does not resolve the cookbook versions itself, this is left to Librarian or Berkshelf. I'm using Librarian-chef and I have a Cheffile setup in my project, where I specify:

cookbook 'ark', git: 'git://github.com/opscode-cookbooks/ark.git', ref: '0.1.0'

[liquid1982]

This is my complete Cheffile

site 'http://community.opscode.com/api/v1'

cookbook 'apt'
cookbook 'rbenv', git: 'git://github.com/fnichol/chef-rbenv.git', ref: 'v0.7.2'
cookbook 'ruby_build'
cookbook 'nginx'
cookbook 'xml'
cookbook 'nodejs'
cookbook 'memcached'
cookbook 'imagemagick'
cookbook 'redisio'
cookbook 'database'
cookbook 'java'
cookbook 'ssh_known_hosts'
cookbook 'application'
cookbook 'application_ruby'

# Vecchia versione di `ark` per bug
# https://github.com/elasticsearch/cookbook-elasticsearch/issues/96#issuecomment-18604638
cookbook 'ark', git: 'git://github.com/opscode-cookbooks/ark.git', ref: '0.1.0'
cookbook 'elasticsearch'

and this is my Cheffile.lock

SITE
  remote: http://community.opscode.com/api/v1
  specs:
    apache2 (1.6.2)
    application (2.0.4)
    application_ruby (1.1.4)
      apache2 (>= 0.0.0)
      application (>= 0.0.0)
      passenger_apache2 (>= 0.0.0)
      runit (~> 1.0)
      unicorn (>= 0.0.0)
    apt (1.10.0)
    aws (0.101.0)
    build-essential (1.4.0)
    chef_handler (1.1.4)
    database (1.4.0)
      aws (>= 0.0.0)
      mysql (>= 1.3.0)
      postgresql (>= 1.0.0)
      xfs (>= 0.0.0)
    elasticsearch (0.2.7)
      ark (>= 0.0.0)
    imagemagick (0.2.2)
    java (1.11.4)
      windows (>= 0.0.0)
    memcached (1.4.0)
      runit (~> 1.0)
      yum (>= 0.0.0)
    mysql (3.0.0)
      build-essential (>= 0.0.0)
      openssl (>= 0.0.0)
    nginx (1.6.0)
      apt (>= 0.0.0)
      build-essential (>= 0.0.0)
      ohai (>= 1.1.4)
      runit (>= 0.0.0)
      yum (>= 0.0.0)
    nodejs (1.1.2)
      apt (>= 0.0.0)
      build-essential (>= 0.0.0)
    ohai (1.1.8)
    openssl (1.0.2)
    partial_search (1.0.0)
    passenger_apache2 (2.0.0)
      apache2 (>= 1.0.4)
      build-essential (>= 0.0.0)
    postgresql (3.0.0)
      apt (>= 0.0.0)
      build-essential (>= 0.0.0)
      openssl (>= 0.0.0)
    redisio (1.4.1)
      ulimit (>= 0.1.2)
    ruby_build (0.8.0)
    runit (1.1.4)
      build-essential (>= 0.0.0)
      yum (>= 0.0.0)
    ssh_known_hosts (1.0.0)
      partial_search (>= 0.0.0)
    ulimit (0.2.0)
    unicorn (1.2.2)
    windows (1.9.0)
      chef_handler (>= 0.0.0)
    xfs (1.1.0)
    xml (1.1.2)
      build-essential (>= 0.0.0)
    yum (2.2.2)

GIT
  remote: git://github.com/fnichol/chef-rbenv.git
  ref: v0.7.2
  sha: f2b53292e810dd2b43f6121f9958f5f29979dcb1
  specs:
    rbenv (0.7.2)

GIT
  remote: git://github.com/opscode-cookbooks/ark.git
  ref: 0.1.0
  sha: 084aef10c0797d6f9ac13de668804cf78d6353dc
  specs:
    ark (0.1.0)

DEPENDENCIES
  application (>= 0)
  application_ruby (>= 0)
  apt (>= 0)
  ark (>= 0)
  database (>= 0)
  elasticsearch (>= 0)
  imagemagick (>= 0)
  java (>= 0)
  memcached (>= 0)
  nginx (>= 0)
  nodejs (>= 0)
  rbenv (>= 0)
  redisio (>= 0)
  ruby_build (>= 0)
  ssh_known_hosts (>= 0)
  xml (>= 0)

[karmi]

So when you specify the '0.2.4' version, does the run fail? I'm trying to figure out what we can do as part of the cookbook...

[liquid1982]

I'm setting up a new VM. I'll let you know soon.

[liquid1982]

Yeah, it seems that the last commit in opscode-cookbooks/ark solved the issue. All runs smooth now.
No need to specify old versions.

Thank you very much!

PS: At the end of the provisioning process, if I login in the vm and issue the service elasticsearch status command (as root), it always reports elasticsearch not running, even if I specify:

service 'elasticsearch' do
  action :start
end

in my recipe. It is a very strange behaviour. I also notice from the Chef log the following records:

  * service[elasticsearch] action start (up to date)
  * service[elasticsearch] action restart
    - restart service service[elasticsearch]

Can anyone figure out what's going on? Should I open a new ticket?

[karmi]

Hmm, it should be definitely running, but a) it needs couple of seconds to be up, b) notice the fixes in #91

[karmi]

Still seeing:

ark[elasticsearch] sending start action to service[elasticsearch] (delayed)

on the current master and latest Ark.

[karmi]

More info: on each Chef run, chef-ark will remove the /usr/local/elasticsearch link, making a solution such as:

ark "elasticsearch" do
  # ...

  not_if "test -x #{node.elasticsearch[:dir]}/elasticsearch-#{node.elasticsearch[:version]}/bin/elasticsearch"
end

impossible.

EDIT: This is actually the bash "remove the elasticsearch user home" do statement in recipes/default.rb, as stated in #104.

[karmi]

Should be solved on the current master, closing -- please feel free to reopen or ping me if you still have problems.

[manojkumarburra]

I am seeing similar error w.r.t logstash

    [execute] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

    ================================================================================
    Error executing action `run` on resource 'bash[shutdown logstash]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  "/tmp/chef-script20200409-7602-t9khsp" ----
    STDOUT:
    STDERR: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
    ---- End output of "bash"  "/tmp/chef-script20200409-7602-t9khsp" ----
    Ran "bash"  "/tmp/chef-script20200409-7602-t9khsp" returned 1

    Cookbook Trace:
    ---------------
    /var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:78:in `run_action'
    /var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:106:in `block (2 levels) in converge'
    /var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:106:in `each'
    /var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:106:in `block in converge'
    /var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:105:in `converge'

    Resource Declaration:
    ---------------------
    # In /var/chef/cache/cookbooks/tmo-logstash/recipes/default.rb

     36: bash 'shutdown logstash' do
     37:   action :run
     38:   code "kill -9 $(pgrep -u logstash)"
     39: end
     40:

    Compiled Resource:
    ------------------
    # Declared in /var/chef/cache/cookbooks/tmo-logstash/recipes/default.rb:36:in `from_file'

    bash("shutdown logstash") do
      action [:run]
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      command "shutdown logstash"
      backup 5
      returns 0
      code "kill -9 $(pgrep -u logstash)"
      interpreter "bash"
      declared_type :bash
      cookbook_name "tmo-logstash"
      recipe_name "default"
    end

    Platform:
    ---------
    x86_64-linux

Recipe: filebeat::config
  * service[filebeat] action start (up to date)

Running handlers:
Running handlers complete
Chef Client failed. 4 resources updated in 39 seconds```

Can someone tell me how should I fix this error.?