[Filebeat] filbeat setup -e overwrites ILM policy ignoring setup.ilm.overwrite: false

[sovcharenko]

• Version: 7.4.2 (amd64), libbeat 7.4.2 [1507515 built 2019-10-28 19:46:13 +0000 UTC]
• Operating System: Centos 7
• Discuss Forum URL: https://discuss.elastic.co/t/ilm-policy-being-over-written/188582/7
• Steps to Reproduce:

1. Create custom ILM policy with name test_policy
2. Configure filebeat

...
setup.ilm.policy_name: "test_policy"
setup.ilm.overwrite: false

3. Run filebeat setup -e

Actual: filebeat test_policy is overwritten with filebeat's default configuration
Expected: test_policy should not be changed

[MrBones757]

I can confirm i am seeing this issue too. have seen it in 7.3.2, 7.4.2 and 7.5.0 using the same flags as you have specified.

[ycombinator]

I'm not seeing this behavior on master (Filebeat built from master running against Elasticsearch built from master).

First I created an ILM policy named test_policy like so:

PUT _ilm/policy/test_policy   
{
  "policy": {                       
    "phases": {
      "delete": {
        "min_age": "7d",           
        "actions": {
          "delete": {}              
        }
      }
    }
  }
}

Then I configured filebeat.yml like so:

setup.ilm.policy_name: "test_policy"
setup.ilm.overwrite: false

Then I ran filebeat setup -e.

Finally, I checked the test_policy ILM policy in Elasticsearch and the Filebeat template like so:

GET _ilm/policy/test_policy
GET _template/filebeat-*?filter_path=*.settings.index.lifecycle

The test_policy policy was not altered, it was the same as the one I created in the first step. And
the filebeat-8.0.0 template was referencing the test_policy ILM policy as expected.

I'm going to try and reproduce on 7.x (targeting 7.6.0 release) next.

[ycombinator]

I was not able to reproduce this using Filebeat built from the 7.x branch (running against ES built from master).

Going to try and reproduce on 7.5 branch (targeting 7.5.1 release) next.

[ycombinator]

I was able to reproduce this using Filebeat built from the 7.5 branch (running against ES built from master). So it appears this is a bug that's been fixed in master (targeting 8.0.0 release) and 7.x (targeting 7.6.0 release) branches, but the bug fix hasn't been backported to the 7.5 branch (targeting 7.5.1 release).

Looks like @simitt fixed the bug in #14741. I've put up a backport of her PR to the 7.5 branch, in case there is a 7.5.1 release in the future: #15039.

[simitt]

This is a fix that breaks current behavior. That's why I listed it in the breaking changes section. Therefore I don't think we should backport it to a patch release, but only to the next minor (it is already in 7.x)

[ycombinator]

I see. But since the current behavior is broken (it's a bug). I think its okay to break the current behavior (i.e. fix the bug) in a patch release, no?

[ycombinator]

After some discussion amongst the Beats core developers, we decided not to backport the fix to the 7.5 branch, meaning the fix will not be available in 7.5.1. The bug fix is a breaking change in behavior and we don't want to introduce a breaking change in a patch release.

The fix will be available starting 7.6.0, which is the next minor version in the 7.x series.