The version of elasticsearch can only be set via elasticsearch/settings databag

[jakelandis]

While trying to set the (non-default) version via a chef environment [1] attribute it does not actually install that version. It will install the default version, unless you set the version via the elasticsearch/settings.

To reproduce:

1. If using chef-server, navigate to the node, edit, attributes (bottom left), select elasticsearch, delete (red minus button), save node. (this is to get the attributes to a clean state)
2. Delete the version from the elasticsearch/settings data bag (or the entire data bag).
3. Set the version to a non-default value via an environment (a role would probably show the same behavior).
4. Run chef-client (with the -l debug to see the version)- it will install the default version, not the version specified by the environment.

This was reproduced with the 0.20.0 version of the cookbook.

[1]
"elasticsearch" => {
"version" => "0.20.5"

[karmi]

Hi, Chef attribute behaviour is notoriously tricky, if memory serves well. The Chef 11 changelog hints at some bugfixes. Are you 100% sure that you're setting the attribute in environment properly? Can you test this with Chef 11?

[karmi]

OK, spent couple of hours hunting this. Notes:

• There were fatal errors regarding setting attributes on Chef 11, which are now fixed (see attached commit)
• I was able to successfully and consistently upgrade/downgrade Elasticsearch in Chef Server and Chef Solo

With Chef Solo, the game is all the same: change configurations in Vagrant file or data bag, run vagrant provision

With Chef Server (hosted at Opscode), this are the steps I took:

First, upload all the cookbooks to Chef Server:

berks upload

Create a new virtual machine in Vagrant:

bundle exec vagrant up precise64 --no-provision

Create a knife.rb configuration file for Chef (use bundle exec vagrant ssh-config precise64 on Vagrant),
specifying correct Chef API endpoint, SSH users, ports, identity files, etc.

Then, bootstrap the machine via Chef:

knife bootstrap 127.0.0.1 --run-list 'role[elasticsearch-vagrant]' --config ./tmp/knife-vagrant.rb --sudo

Now, the important part, to change the version with

a) environment,
b) node attributes

To change the version with environment, simply create it with Chef web console or knife,
and set overrides attribute to a different version. Run Chef on the machine:

sudo chef-client

I believe this will work the same with roles.

To change the version by editing node attributes, edit it with:

knife node edit precise64 --config ./tmp/knife-vagrant.rb

and run chef.

In all cases, the correct version is downloaded, symlinked, and Elasticsearch itself is restarted.

Verify this workflow on your end, please, and get back to me.

[jakelandis]

I was able to verify the latest code + chef 11 (client) resolves this issue.

Using the latest code revision (ac4bb01) , Chef Server 10.? , and changing the version of chef on the client node.

Chef: 10.24.0 (incorrectly downloads and installs the default version in the specified version directory)

---

New install of 0.20.2 (version set via environment) notice the jar version in the lib directory:

   [2013-02-27T09:58:52-06:00] INFO: *** version: 0.20.5
   [2013-02-27T09:58:52-06:00] INFO: *** download_url: http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.5.tar.gz

  ls -l /usr/local/elasticsearch
  /usr/local/elasticsearch -> /usr/local/elasticsearch-0.20.2
 ls  /usr/local/elasticsearch-0.20.2/lib
 elasticsearch-0.20.5.jar  ....

Update to 0.20.3 (version set via environment) notice the jar version in the lib directory:

 [2013-02-27T10:03:02-06:00] INFO: *** version: 0.20.5
 [2013-02-27T10:03:02-06:00] INFO: *** download_url: http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.5.tar.gz

 ls -l /usr/local/elasticsearch
 /usr/local/elasticsearch -> /usr/local/elasticsearch-0.20.3
 ls /usr/local/elasticsearch-0.20.3/lib
 elasticsearch-0.20.5.jar  .....

Chef: 11.4.0 (downloads and installs the correct version)

---

New install of 0.20.2 (version set via environment):

      [2013-02-27T09:48:08-06:00] INFO: *** version: 0.20.2
      [2013-02-27T09:48:08-06:00] INFO: *** download_url: http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz

Update to 0.20.3 (version set via environment):

      [2013-02-27T09:51:24-06:00] INFO: *** version: 0.20.3
      [2013-02-27T09:51:24-06:00] INFO: *** download_url: http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.3.tar.gz

Note:
I added the following lines to attributes/default.rb for debugging:
Chef::Log.info "* version: #{node.elasticsearch[:version]}"
Chef::Log.info "* download_url: " + default.elasticsearch[:download_url]

[karmi]

So, on Chef 10.24.0 it still downloads the incorrect version? Trying it with Chef Server now.

Note: The ark cookbook should print a line with the download URL.

[jakelandis]

Hmm... trying to repeat the test above, but specifying the version via the elasticsearch/settings data bag. It appears in both 10.x and 11.x then version is now being ignored:

[2013-02-27T11:08:41-06:00] INFO: Loaded settings: data_bag_item["elasticsearch", "settings", {"id"=>"settings", "version"=>"0.20.2"}]
[2013-02-27T11:08:41-06:00] INFO: *** version: 0.20.5
[2013-02-27T11:08:41-06:00] INFO: *** download_url: http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.5.tar.gz

Should i log a separate issue?

[karmi]

I'll try also the data_bag way with Chef Solo and Vagrant, still debugging Chef 10.22.0

[karmi]

Added a guard against overwriting the version by using the normal precedence, should be working now either with Chef Solo or Chef Server.

[karmi]

@jakelandis Could you verify the current master, please?

[jakelandis]

Testing against 2.0.4 tag, with chef server (10.x)

Chef: 11.4.0

• Install version 0.20.2
  • version configured via databag - GOOD
  • version configured via environment - GOOD
• Update to version 0.20.3
  • version configured via databag - GOOD
  • version configured via environment - GOOD

Chef: 10.24.0

• Install version 0.20.2
  • version configured via databag - GOOD
  • version configured via environment - BAD [2]
• Update to version 0.20.3
  • version configured via databag - BAD [1]
  • version configured via environment - BAD [3]

[1] It is downloading the prior version ... but there are stale values in the node (it would work if you delete and re-add back the node, or delete the node settings prior to running chef-client)
ls elasticsearch-0.20.3/lib/
elasticsearch-0.20.2.jar
From chef server node JSON:
{"version":"0.20.3","host":"http://download.elasticsearch.org","repository":"elasticsearch/elasticsearch","filename":"elasticsearch-0.20.2.tar.gz","download_url":"http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.2.tar.gz",.....

[2] It is downloading the default version in the specified version's path
ls elasticsearch-0.20.2/lib
elasticsearch-0.20.5.jar

[3] It is downloading the default version in the specified version's path
ls elasticsearch-0.20.3/lib
elasticsearch-0.20.5.jar

[karmi]

Thanks for all this testing! I think what we're hitting here is this issue, where the node attributes are indeed “stale”. I think it's fair to advice using Chef 11 if anybody needs to configure the version attribute from role/environment...