beats 6.0 tries to use previous template and fails with '[_all] is disabled in 6.0'

[LeeDr]

For confirmed bugs, please report:

• Version: 6.0.0-beta2
• Operating System: Ubuntu
• Steps to Reproduce:

1. Install 5.6.0 Elasticsearch and *beat
2. let *beat create its template and an index
3. stop *beat and upgrade to 6.0
4. start *beat
   Starting the beat will create a new template like this;

{
  "packetbeat-6.0.0-beta1": {
    "order": 1,
    "index_patterns": [
      "packetbeat-6.0.0-beta1-*"
    ],

but there is still the old template;

  "packetbeat": {
    "order": 0,
    "index_patterns": [
      "packetbeat-*"
    ],

and since the new index that gets created by the 6. beat is named packetbeat-6.0.0-beta1-2017.08.10 it will match either template, and apparently pickets the original one.

Then you get this in your log;

2017-08-10T18:46:20Z WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_default_]: [_all] is disabled in 6.0. As a replacemen
t, you can use an [copy_to] on mapping fields to create your own catch all field.","caused_by":{"type":"illegal_argument_exception","reason":"[_all] is disabled in 6.0. As a replacemen
t, you can use an [copy_to] on mapping fields to create your own catch all field."}}

[LeeDr]

cc @Rasroh - I think I could temporarily script a work-around to delete the old template before starting up the 6.0 beats.

[tsg]

@ruflin I think I also hit this one the other day, but then couldn't reproduce it. If the issue happens on ES upgrade, then it's understandable why I couldn't reproduce it.

@LeeDr The 6.0 template uses a higher order so it should override the 5.x one. Somehow, though, this doesn't seem to happen in this case, or the way that works is not what we expect.

[tsg]

Hmm, right, this makes sense to me know. In my previous tests, I ran metricbeat 6.x and 5.x in parallel against ES 6.0. This works fine, because metricbeat 5.x will load a different template when it sees ES 6.0.

But if you run metricbeat against ES 5.x, then upgrade to 6.x, Metricbeat can't know that it needs to use a new template. Then any index that matches the old template is going to fail.

@LeeDr I think a better workaround would be to delete the Beats templates immediately before or after the Elasticsearch upgrade.

I wonder if this is a scenario in which Elasticsearch should automatically "upgrade" the template to remove the _all setting? It sounds to me like this could simplify migration for more users than Beats.

@dakrone does the above explanation of the issue makes sense? Any comments on what we could do?

[dakrone]

@tsg I'm not sure what the scope of the upgrade tool is for automatically removing the _all setting, @imotov may know though

[LeeDr]

@tsg the problem doesn't start when I upgrade elasticsearch to 6.0 (without stopping the beats). At that point the beats are all still writing data to elasticsearch OK.
And I just tested that I can restart packetbeat 5.6 and it continues to write to elasticsearch 6.0.
It could still be a problem when the beat goes to create a new index?

The problem I've seen only comes in when I upgrade to beats 6.0. So my work-around to delete the old template before starting the 6.0 beat is working.

[tsg]

@LeeDr Yes, i think it will be a problem when the Beat creates a new index. When you upgrade the Beat to 6.0, it starts a new index (because we changed the naming convention), which triggers the issue.

@dakrone I opened a ticket xpack to discuss that, thanks.

[imotov]

@dakrone the scope of upgrade API is deal with internal indices (not templates), so it's definitely out of the scope. But we have other mechanisms that allow plugins to deal with templates. So, theoretically we could add this functionality to x-pack, but it will not help users who don't install x-pack.

[LeeDr]

@tsg who's working on this? Can we get the issue assigned?

[dakrone]

@LeeDr see: elastic/elasticsearch#26259

[tsg]

On the Beats side, this is being addressed in: #4901

[tsg]

#4901 is done, so I think we can consider this closed. Note that in the upgrading guide, we will ask people to make sure the 5.6 (or 6.0) template is loaded (via template.overwrite=true) before doing the ES template.