[ISM rollover] Policy not uploaded if there is another with the same priority #56

AlexRuiz7 · 2023-11-15T11:49:23Z

Description

Related issues:

As described in #53 (comment), the script fails to upload the rollover policy if there is an existing policy with the same ID. This is not a problem itself, but the expected behavior. However, the script continues and fails silently.

The failure is that the script checks whether the policy was created by verifying the return code of the cURL command, which return 0 (the request went fine), but the operation actually failed, returning the following message:

curl -k ${C_AUTH} -X GET "${INDEXER_URL}/_plugins/_ism/policies/${POLICY_NAME}?pretty"

{
  "error" : {
    "root_cause" : [
      {
        "type" : "status_exception",
        "reason" : "Policy not found"
      }
    ],
    "type" : "status_exception",
    "reason" : "Policy not found"
  },
  "status" : 404
}

[root@rhel7 vagrant]# echo $?
0

The script needs to be updated to check the HTTP status code, instead of cURL's return code. Probably using both is a good idea.

AlexRuiz7 · 2023-11-15T15:24:40Z

Improved logging

[vagrant@rhel7 vagrant]$ bash indexer-ism-init.sh -i 192.168.56.10 -p +YO57pQKHvrG84qg5kFI9VNlD+Qqmmi3
Will create index templates to configure the alias
 SUCC: 'wazuh-alerts' template created or updated
 SUCC: 'wazuh-archives' template created or updated
Will create the 'rollover_policy' policy
  INFO: policy 'rollover_policy' already exists. Skipping policy creation
Will create initial indices for the aliases
  INFO: 'wazuh-alerts' write index already exists. Skipping write index creation
  INFO: 'wazuh-archives' write index already exists. Skipping write index creation
SUCC: Indexer ISM initialization finished successfully

AlexRuiz7 · 2023-11-15T16:06:10Z

Environment setup

Using the same Vagrantfile as in the tests, remove eveything added by the indexer-ism-init.sh script:

C_AUTH="-u admin:+YO57pQKHvrG84qg5kFI9VNlD+Qqmmi3"
INDEXER_URL="https://192.168.56.10:9200"

curl -k ${C_AUTH} -X DELETE "${INDEXER_URL}/_plugins/_ism/policies/rollover_policy?pretty"
curl -k ${C_AUTH} -X DELETE "${INDEXER_URL}/_template/wazuh-a*?pretty"
curl -k ${C_AUTH} -X DELETE "${INDEXER_URL}/wazuh-a*-0000*?pretty"

Scenario A) Lower prio 🟢

Succesfully created. 🟢

Details

bash indexer-ism-init.sh -i 192.168.56.10 -p +YO57pQKHvrG84qg5kFI9VNlD+Qqmmi3 -P 25
Will create index templates to configure the alias
SUCC: 'wazuh-alerts' template created or updated
SUCC: 'wazuh-archives' template created or updated
Will create the 'rollover_policy' policy
SUCC: 'rollover_policy' policy created
Will create initial indices for the aliases
SUCC: 'wazuh-alerts' write index created
SUCC: 'wazuh-archives' write index created
SUCC: Indexer ISM initialization finished successfully

curl -k ${C_AUTH} -X GET "${INDEXER_URL}/_plugins/_ism/policies/rollover_policy?pretty"
{
"_id" : "rollover_policy",
"_version" : 5,
"_seq_no" : 211,
"_primary_term" : 4,
"policy" : {
    "policy_id" : "rollover_policy",
    "description" : "Wazuh rollover and alias policy",
    "last_updated_time" : 1700063531846,
    "schema_version" : 19,
    "error_notification" : null,
    "default_state" : "active",
    "states" : [
    {
        "name" : "active",
        "actions" : [
        {
            "retry" : {
            "count" : 3,
            "backoff" : "exponential",
            "delay" : "1m"
            },
            "rollover" : {
            "min_doc_count" : 600000000,
            "min_index_age" : "7d",
            "min_primary_shard_size" : "25gb",
            "copy_alias" : false
            }
        }
        ],
        "transitions" : [ ]
    }
    ],
    "ism_template" : [
    {
        "index_patterns" : [
        "wazuh-alerts-*",
        "wazuh-archives-*",
        "-wazuh-alerts-4.x-sample*"
        ],
        "priority" : 25,
        "last_updated_time" : 1700063531846
    }
    ]
}
}

Existing policy keeps preference 🟢

Details

curl -k ${C_AUTH} "${INDEXER_URL}/_plugins/_ism/explain?pretty"
{
"wazuh-alerts-4.x-2023.11.15-000001" : {
    "index.plugins.index_state_management.policy_id" : "test_policy",
    "index.opendistro.index_state_management.policy_id" : "test_policy",
    "index" : "wazuh-alerts-4.x-2023.11.15-000001",
    "index_uuid" : "mChoeg7EQXqOO1HxQdauRg",
    "policy_id" : "test_policy",
    "policy_seq_no" : -2,
    "policy_primary_term" : 0,
    "index_creation_date" : 1700063532078,
    "state" : {
    "name" : "active",
    "start_time" : 1700063925044
    },
    "retry_info" : {
    "failed" : false,
    "consumed_retries" : 0
    },
    "info" : {
    "message" : "Successfully initialized policy: test_policy"
    },
    "enabled" : true
},
"wazuh-archives-4.x-2023.11.15-000001" : {
    "index.plugins.index_state_management.policy_id" : "test_policy",
    "index.opendistro.index_state_management.policy_id" : "test_policy",
    "index" : "wazuh-archives-4.x-2023.11.15-000001",
    "index_uuid" : "FuaTVzCfRtiQuJIvshX48Q",
    "policy_id" : "test_policy",
    "policy_seq_no" : -2,
    "policy_primary_term" : 0,
    "index_creation_date" : 1700063532424,
    "state" : {
    "name" : "active",
    "start_time" : 1700063911158
    },
    "retry_info" : {
    "failed" : false,
    "consumed_retries" : 0
    },
    "info" : {
    "message" : "Successfully initialized policy: test_policy"
    },
    "enabled" : true
},
"total_managed_indices" : 2
}

Skipped the creation of the policy if it exists. 🟢

Details

bash indexer-ism-init.sh -i 192.168.56.10 -p +YO57pQKHvrG84qg5kFI9VNlD+Qqmmi3 -P 25
Will create index templates to configure the alias
SUCC: 'wazuh-alerts' template created or updated
SUCC: 'wazuh-archives' template created or updated
Will create the 'rollover_policy' policy
INFO: policy 'rollover_policy' already exists. Skipping policy creation
Will create initial indices for the aliases
INFO: 'wazuh-alerts' write index already exists. Skipping write index creation
INFO: 'wazuh-archives' write index already exists. Skipping write index creation
SUCC: Indexer ISM initialization finished successfully

Scenario B) Equal prio 🟢

Policy not created. The expected error is properly managed 🟢

Details

bash indexer-ism-init.sh -i 192.168.56.10 -p +YO57pQKHvrG84qg5kFI9VNlD+Qqmmi3

Will create index templates to configure the alias
    SUCC: 'wazuh-alerts' template created or updated
    SUCC: 'wazuh-archives' template created or updated
Will create 'rollover_policy' rollover policy
    ERROR: 'rollover_policy' policy not created => 400
ERROR: Indexer ISM initialization failed. Check /var/log/wazuh-indexer/ism-init.log for more information

/var/log/wazuh-indexer/ism-init.log:

{"error":{"root_cause":[{"type":"index_management_exception","reason":"New policy rollover_policy has an ISM template with index pattern [wazuh-alerts-*, wazuh-archives-*, -wazuh-alerts-4.x-sample*] matching existing policy templates, please use a different priority than 50"}],"type":"index_management_exception","reason":"New policy rollover_policy has an ISM template with index pattern [wazuh-alerts-*, wazuh-archives-*, -wazuh-alerts-4.x-sample*] matching existing policy templates, please use a different priority than 50","caused_by":{"type":"exception","reason":"java.lang.IllegalArgumentException: New policy rollover_policy has an ISM template with index pattern [wazuh-alerts-*, wazuh-archives-*, -wazuh-alerts-4.x-sample*] matching existing policy templates, please use a different priority than 50"}},"status":400}

Scenario C) Higher prio 🟢

Policy created 🟢

Details

bash indexer-ism-init.sh -i 192.168.56.10 -p +YO57pQKHvrG84qg5kFI9VNlD+Qqmmi3 -P 100

Will create index templates to configure the alias
    SUCC: 'wazuh-alerts' template created or updated
    SUCC: 'wazuh-archives' template created or updated
Will create the 'rollover_policy' policy
    SUCC: 'rollover_policy' policy created
Will create initial indices for the aliases
    SUCC: 'wazuh-alerts' write index created
    SUCC: 'wazuh-archives' write index created
SUCC: Indexer ISM initialization finished successfully

Previous policy is still in use for existing indices:

curl -k ${C_AUTH} "${INDEXER_URL}/_plugins/_ism/explain?pretty"

{
"wazuh-alerts-4.x-2023.11.15-000001" : {
    "index.plugins.index_state_management.policy_id" : "test_policy",
    "index.opendistro.index_state_management.policy_id" : "test_policy",
    "index" : "wazuh-alerts-4.x-2023.11.15-000001",
    "index_uuid" : "gq6FE9izRV-MpwCphI9CGA",
    "policy_id" : "test_policy",
    "enabled" : true
},
"wazuh-archives-4.x-2023.11.15-000001" : {
    "index.plugins.index_state_management.policy_id" : "test_policy",
    "index.opendistro.index_state_management.policy_id" : "test_policy",
    "index" : "wazuh-archives-4.x-2023.11.15-000001",
    "index_uuid" : "YQf24AS6RvSVNBO-kxSTHg",
    "policy_id" : "test_policy",
    "enabled" : true
},
"total_managed_indices" : 2
}

Perform manual rollover:

curl -k ${C_AUTH} -X POST "${INDEXER_URL}/wazuh-alerts/_rollover?pretty"
curl -k ${C_AUTH} -X POST "${INDEXER_URL}/wazuh-archives/_rollover?pretty"

{
    "acknowledged" : true,
    "shards_acknowledged" : true,
    "old_index" : "wazuh-alerts-4.x-2023.11.15-000001",
    "new_index" : "wazuh-alerts-4.x-2023.11.15-000002",
    "rolled_over" : true,
    "dry_run" : false,
    "conditions" : { }
}


{
    "acknowledged" : true,
    "shards_acknowledged" : true,
    "old_index" : "wazuh-archives-4.x-2023.11.15-000001",
    "new_index" : "wazuh-archives-4.x-2023.11.15-000002",
    "rolled_over" : true,
    "dry_run" : false,
    "conditions" : { }
}

New policy takes preference for new created indices 🟢

Details

curl -k ${C_AUTH} "${INDEXER_URL}/_plugins/_ism/explain?pretty"

{
    "wazuh-alerts-4.x-2023.11.15-000001" : {
        "index.plugins.index_state_management.policy_id" : "test_policy",
        "index.opendistro.index_state_management.policy_id" : "test_policy",
        "index" : "wazuh-alerts-4.x-2023.11.15-000001",
        "index_uuid" : "gq6FE9izRV-MpwCphI9CGA",
        "policy_id" : "test_policy",
        "policy_seq_no" : -2,
        "policy_primary_term" : 0,
        "index_creation_date" : 1700069896815,
        "state" : {
        "name" : "active",
        "start_time" : 1700070233627
        },
        "retry_info" : {
        "failed" : false,
        "consumed_retries" : 0
        },
        "info" : {
        "message" : "Successfully initialized policy: test_policy"
        },
        "enabled" : true
    },
    "wazuh-alerts-4.x-2023.11.15-000002" : {
        "index.plugins.index_state_management.policy_id" : "rollover_policy",
        "index.opendistro.index_state_management.policy_id" : "rollover_policy",
        "index" : "wazuh-alerts-4.x-2023.11.15-000002",
        "index_uuid" : "0hu4iOBTQ9a5o0ZyqIHgMQ",
        "policy_id" : "rollover_policy",
        "enabled" : true
    },
    "wazuh-archives-4.x-2023.11.15-000001" : {
        "index.plugins.index_state_management.policy_id" : "test_policy",
        "index.opendistro.index_state_management.policy_id" : "test_policy",
        "index" : "wazuh-archives-4.x-2023.11.15-000001",
        "index_uuid" : "YQf24AS6RvSVNBO-kxSTHg",
        "policy_id" : "test_policy",
        "enabled" : true
    },
    "wazuh-archives-4.x-2023.11.15-000002" : {
        "index.plugins.index_state_management.policy_id" : "rollover_policy",
        "index.opendistro.index_state_management.policy_id" : "rollover_policy",
        "index" : "wazuh-archives-4.x-2023.11.15-000002",
        "index_uuid" : "5Wp8dHe-RdORH2VUrFUaLg",
        "policy_id" : "rollover_policy",
        "enabled" : true
    },
    "total_managed_indices" : 4
}

Skipped the creation of the policy if it exists. 🟢

Details

bash indexer-ism-init.sh -i 192.168.56.10 -p +YO57pQKHvrG84qg5kFI9VNlD+Qqmmi3 -P 100

Will create index templates to configure the alias
    SUCC: 'wazuh-alerts' template created or updated
    SUCC: 'wazuh-archives' template created or updated
Will create the 'rollover_policy' policy
    INFO: policy 'rollover_policy' already exists. Skipping policy creation
Will create initial indices for the aliases
    INFO: 'wazuh-alerts' write index already exists. Skipping write index creation
    INFO: 'wazuh-archives' write index already exists. Skipping write index creation
SUCC: Indexer ISM initialization finished successfull

AlexRuiz7 added level/task Task issue type/bug Bug issue labels Nov 15, 2023

AlexRuiz7 self-assigned this Nov 15, 2023

AlexRuiz7 changed the title ~~[ISM rollover] Policy not uploaded if there is another policy with the same priority~~ [ISM rollover] Policy not uploaded if there is another with the same priority Nov 15, 2023

AlexRuiz7 mentioned this issue Nov 15, 2023

Rollover + alias E2E testing #53

Closed

16 tasks

wazuhci added this to Release 4.9.0 Nov 15, 2023

wazuhci moved this to In progress in Release 4.9.0 Nov 15, 2023

wazuhci added this to Release 4.8.0 Nov 15, 2023

wazuhci moved this to In progress in Release 4.8.0 Nov 15, 2023

wazuhci removed this from Release 4.9.0 Nov 15, 2023

This was referenced Nov 15, 2023

Improve ISM init script #57

Merged

Improve ISM init script wazuh/wazuh-packages#2606

Merged

wazuhci moved this from In progress to Pending review in Release 4.8.0 Nov 15, 2023

AlexRuiz7 closed this as completed in #57 Nov 15, 2023

wazuhci moved this from Pending review to Done in Release 4.8.0 Nov 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ISM rollover] Policy not uploaded if there is another with the same priority #56

[ISM rollover] Policy not uploaded if there is another with the same priority #56

AlexRuiz7 commented Nov 15, 2023 •

edited

Loading

AlexRuiz7 commented Nov 15, 2023

AlexRuiz7 commented Nov 15, 2023 •

edited

Loading

[ISM rollover] Policy not uploaded if there is another with the same priority #56

[ISM rollover] Policy not uploaded if there is another with the same priority #56

Comments

AlexRuiz7 commented Nov 15, 2023 • edited Loading

Description

AlexRuiz7 commented Nov 15, 2023

AlexRuiz7 commented Nov 15, 2023 • edited Loading

Environment setup

Scenario A) Lower prio 🟢

Scenario B) Equal prio 🟢

Scenario C) Higher prio 🟢

AlexRuiz7 commented Nov 15, 2023 •

edited

Loading

AlexRuiz7 commented Nov 15, 2023 •

edited

Loading