[BUG] ISM policies and index templates are constantly updated

[lpeter91]

Operator version: 2.6.0

The following very simple ISM template and index template get constantly updated by the operator. (The operator logs "policy exists and requires update" and "Index template requires update" respectively.)

ISM template:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchISMPolicy
metadata:
  name: soc-policy
spec:
  opensearchCluster:
    name: opensearch
  policyId: soc-policy
  description: "SOC ISM policy"
  defaultState: hot
  states:
    - name: hot
      actions:
        - rollover:
            minIndexAge: "14d"
            minPrimaryShardSize: 10gb
          retry:
            count: 3
            backoff: exponential
            delay: 1m
      transitions:
        - stateName: delete
          conditions:
            minRolloverAge: "365d"
    - name: delete
      actions:
        - delete: { }
          retry:
            count: 3
            backoff: exponential
            delay: 1m
  ismTemplate:
    indexPatterns:
      - "logs-*"
      - "probe-*"
    priority: 1000

Index template:

apiVersion: opensearch.opster.io/v1
kind: OpensearchIndexTemplate
metadata:
  name: logs-generic
spec:
  composedOf:
    - soc-ecs-mappings
    - soc-settings
    - soc-mappings
  dataStream: {}
  indexPatterns:
    - logs-*
  name: logs-generic
  opensearchCluster:
    name: opensearch
  priority: 300

Now these are just examples, in reality most (maybe even all) of my ISM policies and index templates are constantly updated. In fact, my component templates also get updated all the time, but that might be fixed by the unreleased #809. (Btw please do a release, without the fix for #801 the operator isn't really usable in a conventional way.)

For the index templates the issue might be with the data stream, to which the default timestamp field is added. But in general it seems to me that this "diffing" approach is very fragile and destined to break over time. Maybe an other way to handle this should be considered, some very vague ideas:

• Storing (writing back to the K8s resource?) the API representation of the resource
• Only keeping this diffing as a performance optimization, reducing the log level to debug and increasing the time between reconcile iterations (could be configurable)

[prudhvigodithi]

[Triage]
I see there is a PR created for this issue #846, thanks @rkthtrifork.
Adding @swoehrl-mw @salyh
Thank you
@getsaurabh02

[lpeter91]

Nice, but please note that #846 only partially addresses this issue. It (supposedly) fixes the issue with ISM templates, but not for index templates.

[cthtrifork]

@lpeter91 We will look at index templates once the approach have been approved.

[lpeter91]

@swoehrl-mw I think the closing of this issue was premature, since #846 only partially addresses the problems reported, as discussed above.

[dblock]

[Catch All Triage - 1, 2, 3, 4, 5]