Issues configuring unicast cluster

[Rockstar04]

I feel really bad for opening this because I am sure its my fault, since I'm really new to Chef (Puppet convert), but I need some help.

I am setting up a 3 node ElasticSearch cluster in our new data center, and since our blade chassis's switch doesn't support multicast I need to get the unicast discovery working with Chef.

This is my elasticsearch role config.

{
  "name": "Elasticsearch",
  "description": "Configure an ElasticEearch",
  "default_attributes": {
    "java": {
      "install_flavor": "openjdk",
      "jdk_version": "7"
    },
    "elasticsearch": {
      "cluster_name" : "production_cluster",
      "discovery.search_query": "role:Elasticsearch AND chef_environment:#{node.chef_environment}"
    }
  },
  "run_list": [
    "recipe[java]",
    "recipe[elasticsearch]",
    "recipe[elasticsearch::search_discovery]"
  ]
}

When I use knife search with the search query above it finds all three of my ES nodes, but the Chef run does not add them to the elasticsearch.yml

Like I said, I am 99.9% sure I am just missing something, but I spend almost 4 hours yesterday combing through the source here on GitHub, and between being new with Chef and not being fluent in Ruby, I am really stumped.

[karmi]

First, it's perfectly fine to open an issue just for a question like this.

Second, it's weird, can you run Chef with debug level logging, so the "Found elasticsearch nodes at …" line is printed? What gets added to elasticserch.yml, is the hosts value blank? What if you leave out environment:…? What if you downcase the role?

[Rockstar04]

Thanks for the lead, it looks like the search query isn't getting set. Am I correct to use "discovery.search_query" dot notation or do I need to use a json object? I have tried both but the json object crashed the chef run iirc.

Ad far as my elasticsearch.yml, it looks great, all the other options to do with discovery look correct, but hosts is indeed completely blank.

Here is the query and return from the chef run (with a lot of stuff omitted)

DEBUG: Initiating GET to https://chef.<company>.com/search/node?q=roles:elasticsearch%20AND%20chef_environment:Production%20AND%20elasticsearch_cluster_name:elasticsearch&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000
DEBUG: ---- HTTP Request Header Data: ----
<Omit>
DEBUG: ---- HTTP Status and Header Data: ----
DEBUG: HTTP 1.1 200 OK
<Omit>
DEBUG: ---- End HTTP Status/Header Data ----
DEBUG: decompressing gzip response
DEBUG: Found elasticsearch nodes at ""

Id it considered best practice to use lowercase for role names? I haven't found (haven't looked very hard) any community agreed upon standards for naming and such.

[Rockstar04]

Looks like I guessed correctly, this WAS a case of user error.

First issue was that the dot notation was causing the default search query to be used.

This is what I should have been using (and what the config looks like now):

    "elasticsearch": {
      "cluster_name" : "production_cluster",
      "discovery": {
        "search_query": "role:Elasticsearch AND chef_environment:Production"
      }
    }

The second issue was that I was trying to do this, which was causing the Chef run to crash, making me think the json was incorrectly formatted.

    "elasticsearch": {
      "cluster_name" : "production_cluster",
      "discovery": {
        "search_query": "role:Elasticsearch AND chef_environment:#{node.chef_environment}"
      }
    }

After a Chef run on all three of my nodes, and a quick reboot, I have a working ElasticSearch clsuter! Thanks for the pointer about debug, that's what got me pointed in the right direction!

[karmi]

Aye, the "dot notation" is quite specific to Elasticsearch (and the Chef cookbook can play with it), but when you're setting up your node attributes, environments, etc etc., you have to use the common & valid Ruby Hash / JSON. Glad it's working.