Rollover and alias for stream indices

[AlexRuiz7]

Description

Related issues:

• Index State Management default policy wazuh#18999
• Explore how to use the ISM features from another plugin wazuh-indexer-plugins#12

One of the main requirements of the Data Persistence Model Redesign project is to include aliases and rollover policies to stream indices by default, as Index Management related features.

For Wazuh 5, we have identified 2 stream indices:

• wazuh-alerts data stream.
• wazuh-commands data stream.

The setup plugin (see wazuh/wazuh-indexer-plugins#9) generates indices for both data streams at startup, wazuh-alerts-5.x-0001 and .commands respectively.

On this issue, we are going to create aliases and rollover policies for both data streams, defining the rollover criteria.

We have not yet found a simple way of interacting with the OpenSearch's Indexer Management plugin, which is responsible for these things. As part of this issue, we will investigate how to implement these features within our setup plugin.

Functional requirements

• The wazuh-alerts data stream is associated to an alias.
• The wazuh-alerts data stream is managed by an active rollover policy.
• The wazuh-commands data stream is associated to an alias.
• The wazuh-commands data stream is managed by an active rollover policy.
• Aliases and rollover policies are generated automatically.

Implementation restrictions

• The initialization of the index aliases and the rollover policies are the responsibility of the setup plugin.

Plan

• Spike. Investigate how the IM plugin persists such data.
• Spike. Reproduce the IM creation of policies.
• Define aliases names.
• Define rollover policies.
• Checkpoint
• Apply changes.

[mcasas993]

Investigation of how the ISM plugin persists such data

• The ismPlugin function extends the OpenSearch client by adding a namespace for managing index policies, allowing for easy interactions with the API to get, create, update, and delete policies, as well as handle associated operations like rollups and transforms.
  https://github.com/opensearch-project/index-management-dashboards-plugin/blob/main/server/clusters/ism/ismPlugin.ts

• In the ismPlugin they use API constant from "../../utils/constants" in which exists definition of parameter of Index to ISM configuration
  

export enum INDEX {
  OPENDISTRO_ISM_CONFIG = ".opendistro-ism-config",
}

• This class document and control the mapping of policy schema in ISM Plugin.

• Effectivily, after create a policy, I can search it in the .opendistro-ism-config index:

GET /.opendistro-ism-config/_search

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".opendistro-ism-config",
        "_id": "first_test",
        "_score": 1,
        "_source": {
          "policy": {
            "policy_id": "first_test",
            "description": "A first test of creation of an policy",
            "last_updated_time": 1736967823609,
            "schema_version": 21,
            "error_notification": null,
            "default_state": "pre_alias_removed",
            "states": [
              {
                "name": "pre_alias_removed",
                "actions": [
                  {
                    "retry": {
                      "count": 3,
                      "backoff": "exponential",
                      "delay": "1h"
                    },
                    "alias": {
                      "actions": [
                        {
                          "remove": {
                            "aliases": [
                              "commands"
                            ]
                          }
                        }
                      ]
                    }
                  }
                ],
                "transitions": []
              }
            ],
            "ism_template": [],
            "user": {
              "name": "admin",
              "backend_roles": [
                "admin"
              ],
              "roles": [
                "own_index",
                "all_access"
              ],
              "custom_attribute_names": [],
              "user_requested_tenant": null
            }
          }
        }
      }
    ]
  }
}

UPDATE

• The alias to do the rollover is applied in the settings of index that we want to manage with the policy

PUT /our_index/_settings
{
  "index": {
    "plugins": {
      "index_state_management": {
        "rollover_alias": "name_of_alias_to_rollover"
      }
    }
  }
}

[mcasas993]

Complete test of policy to rollover

1. Test in dashboard

### 1. Test in dashboard

Policy created to rollover alias

GET /.opendistro-ism-config/_search/
{
  "query": {
    "match": {
      "_id":"policy_rollover_6"
    }
  }
}

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".opendistro-ism-config",
        "_id": "policy_rollover_6",
        "_score": 1,
        "_source": {
          "policy": {
            "policy_id": "policy_rollover_6",
            "description": "Created first the policy",
            "last_updated_time": 1737399485357,
            "schema_version": 21,
            "error_notification": null,
            "default_state": "created",
            "states": [
              {
                "name": "created",
                "actions": [
                  {
                    "retry": {
                      "count": 3,
                      "backoff": "exponential",
                      "delay": "1m"
                    },
                    "rollover": {
                      "min_doc_count": 3,
                      "copy_alias": true
                    }
                  }
                ],
                "transitions": [
                  {
                    "state_name": "rollovered",
                    "conditions": {
                      "min_doc_count": 3
                    }
                  }
                ]
              },
              {
                "name": "rollovered",
                "actions": [],
                "transitions": []
              }
            ],
            "ism_template": [
              {
                "index_patterns": [
                  "another-*"
                ],
                "priority": 1,
                "last_updated_time": 1737382578280
              }
            ],
            "user": {
              "name": "admin",
              "backend_roles": [
                "admin"
              ],
              "roles": [
                "own_index",
                "all_access"
              ],
              "custom_attribute_names": [],
              "user_requested_tenant": null
            }
          }
        }
      }
    ]
  }
}

Index created and with an alias associated

Policy applied with the alias specified

GET /another-rollover/_settings

{
  "another-rollover-index-00001": {
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "refresh_interval": "1s",
        "number_of_shards": "1",
        "plugins": {
          "index_state_management": {
            "rollover_alias": "another-rollover",
            "auto_manage": "false"
          }
        },
        "provided_name": "another-rollover-index-00001",
        "creation_date": "1737399216111",
        "number_of_replicas": "0",
        "uuid": "WSeQeDH7S7a9LQTs0qqTSw",
        "version": {
          "created": "136347827"
        }
      }
    }
  }
}

Create documents in the test index

Create five documents like this:

POST /another-rollover/_doc
{
  "name": "Another Example",
  "price": 19.99,
  "description": "We are such stuff as dreams are made on"
}

Results of the policy run

Result in another-rollover-index-00001 index

Result in another-rollover-index-00002 index

2. Test in manually in DevOp

### 2. Test in manually in DevOp

Policy created to rollover alias

POST .opendistro-ism-config/_doc/manual_policy_rollover_3
     {
      "policy": {
          "description": "Created third manual rollover the policy",
          "last_updated_time": 1737399485357,
          "schema_version": 21,
          "error_notification": null,
          "default_state": "Initial",
          "states": [
            {
              "name": "Initial",
              "actions": [
                {
                  "retry": {
                    "count": 3,
                    "backoff": "exponential",
                    "delay": "1m"
                  },
                  "rollover": {
                    "min_doc_count": 3,
                    "copy_alias": true
                  }
                }
              ],
              "transitions": [
                {
                  "state_name": "Rollovered",
                  "conditions": {
                    "min_doc_count": 3
                  }
                }
              ]
            },
            {
              "name": "Rollovered",
              "actions": [],
              "transitions": []
            }
          ],
          "ism_template": [
            {
              "index_patterns": [
                "manual-*"
              ],
              "priority": 1,
              "last_updated_time": 1737382578280
            }
          ],
          "user": {
            "name": "admin",
            "backend_roles": [
              "admin"
            ],
            "roles": [
              "own_index",
              "all_access"
            ],
            "custom_attribute_names": [],
            "user_requested_tenant": null
          }
        }
      }

Index created and with an alias associated

Policy applied with the alias specified

PUT /manual-0001/_settings
{
  "index": {
    "plugins": {
      "index_state_management": {
        "rollover_alias": "manual"
      }
    }
  }
}

Policy applied with the alias specified

POST .opendistro-ism-config/_doc/
{
    "managed_index": {
      "name": "manual-0001",
      "enabled": false,
      "index": "manual-0001",
      "index_uuid": "OG3zRya0RoiKpjbxyJTbbA",
      "schedule": {
        "interval": {
          "start_time": 1737494828282,
          "period": 5,
          "unit": "Minutes"
        }
      },
      "last_updated_time": 1737496659085,
      "enabled_time": null,
      "policy_id": "manual_policy_rollover_3",
      "policy_seq_no": 2510,
      "policy_primary_term": 3,
      "policy": {
            "description": "Created third manual rollover the policy",
            "last_updated_time": 1737399485357,
            "schema_version": 21,
            "error_notification": null,
            "default_state": "Initial",
            "states": [
              {
                "name": "Initial",
                "actions": [
                  {
                    "retry": {
                      "count": 3,
                      "backoff": "exponential",
                      "delay": "1m"
                    },
                    "rollover": {
                      "min_doc_count": 3,
                      "copy_alias": true
                    }
                  }
                ],
                "transitions": [
                  {
                    "state_name": "Rollovered",
                    "conditions": {
                      "min_doc_count": 3
                    }
                  }
                ]
              },
              {
                "name": "Rollovered",
                "actions": [],
                "transitions": []
              }
            ],
            "ism_template": [
              {
                "index_patterns": [
                  "manual-*"
                ],
                "priority": 1,
                "last_updated_time": 1737382578280
              }
            ],
            "user": {
              "name": "admin",
              "backend_roles": [
                "admin"
              ],
              "roles": [
                "own_index",
                "all_access"
              ],
              "custom_attribute_names": [],
              "user_requested_tenant": null
            }
          },
      "change_policy": null,
      "jitter": 0.6
    }
}

Results of the policy run

Results of the policy run in manual-0001

[mcasas993]

Complete test of policy to rollover based on the previous issue

Test the issue steps

Test the issue steps

Applied an ISM policy for rollover as follows:

1. Template modification

• Edit /etc/filebeat/wazuh-template.json and add the following line inside the settings block:

  ```json
  "index.plugins.index_state_management.rollover_alias": "test"
  ```
  

• Restart wazuh-manager

  systemctl restart wazuh-manager.service

2. ISM rollover and alias policy

   • Push ISM policy to the Wazuh indexer cluster:

     "min_size": "250mb" for testing purposes only

curl -XPOST  -k -u admin:$admin_pass "https://127.0.0.1:9200/.opendistro-ism-config/_doc/MANUAL_wazuh_rollover_policy" -H 'Content-Type: application/json' -d'
{
    "policy": {
      "policy_id": "MANUAL_wazuh_rollover_policy",
      "description": "Wazuh rollover and alias policy created directly on index .opendistro-ism-config",
      "last_updated_time": 1737572429671,
      "schema_version": 21,
      "error_notification": null,
      "default_state": "active",
      "states": [
        {
          "name": "active",
          "actions": [
            {
              "retry": {
                "count": 3,
                "backoff": "exponential",
                "delay": "1m"
              },
              "rollover": {
                "min_size": "250mb",
                "copy_alias": false
              }
            }
          ],
          "transitions": []
        }
      ],
      "ism_template": [
        {
          "index_patterns": [
            "wazuh-alerts-*"
          ],
          "priority": 50,
          "last_updated_time": 1737572429671
        }
      ],
      "user": {
        "name": "admin",
        "backend_roles": [
          "admin"
        ],
        "roles": [
          "own_index",
          "all_access"
        ],
        "custom_attribute_names": [],
        "user_requested_tenant": null
      }
    }
}'

* Create initial index and quick-start the rolling process:
```bash
curl -k -u admin:$admin_pass -X PUT "https://127.0.0.1:9200/%3Ctest-1.x-%7Bnow%2Fd%7D-000001%3E?pretty" -H 'Content-Type: application/json' -d'

{
"aliases": {
"test": {
"is_write_index": true
}
}
}'

**RESULT**
Error: ""message": "Missing rollover_alias index setting [index=test-1.x-2025.01.22-000001]""

![Image](https://github.com/user-attachments/assets/3656f2b7-477a-49a9-88b7-05473217eaac)

3. Try with a the template to the index
   * Create a template to the index
    ![Image](https://github.com/user-attachments/assets/f731e246-fe23-4b65-8b86-31639e0d027f)
   
   *Delete the manage of the policy in the index
    ![Image](https://github.com/user-attachments/assets/dcaf651d-3970-4190-8c9d-2564b24eda91)

   *Delete the index
    In Dev Tools: DELETE /test-1.x-2025.01.22-000001

    * Create initial index and quick-start the rolling process:
    ```bash
    curl -k -u admin:$admin_pass -X PUT "https://127.0.0.1:9200/%3Ctest-1.x-%7Bnow%2Fd%7D-000001%3E?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "test": {
      "is_write_index": true
    }
  }
}'

RESULT
Error: ""message": "Missing rollover_alias index setting [index=test-1.x-2025.01.22-000001]""

4. Put the settiing in the index

PUT /test/_settings
{
  "index": {
    "plugins": {
      "index_state_management": {
        "rollover_alias": "test"
      }
    }
  }
}

RESULT
Error: ""message": "{
"cause": "Rollover alias [test] can point to multiple indices, found duplicated alias [[test]] in index template [test]",
"message": "Failed to rollover index [index=test-1.x-2025.01.22-000001]"
}""