Merge pull request #6106 from ministryofjustice/elasticsearch-runbook

docs: initial commit of elasticsearch runbook

runbooks/source/elasticsearch-storage-issues.html.md.erb

@@ -0,0 +1,31 @@
+---
+title: Elasticsearch Storage Issues
+weight: 9999
+last_reviewed_on: 2024-08-29
+review_in: 6 months
+---
+
+# <%= current_page.data.title %>
+
+## Hot/Warm Node Parity
+
+A way of increasing the storage space for either Hot or Warm nodes is to increase them. If increasing one then the other must be equal as well, for example if upping the Hot nodes to 20 Warm should be upped to 20 as well.
+
+The number of shards and the nodes should be increased by a multiple of the index with the largest number of shards, further [information](https://aws.amazon.com/blogs/opensource/open-distro-elasticsearch-shard-allocation/).
+
+## Manually migrating failed indexes
+
+It may be necessary to migrate indexes manually. The process for this is:
+
+1. Navigate to "Policy managed indices"
+2. Find `Failed to start **** migration`
+3. Copy the index name
+4. Search "Warm" indexes and get index size
+5. Search "Cold" indexes and get index size
+6. If the "Cold" index size is smaller than "Warm" then it can be deleted
+7. Delete the index via the devtools `DELETE _cold/live_kubernetes_ingress-2024.07.13`
+8. Click retry policy
+
+## Further Reading
+
+* [Open search best practices](https://runbooks.cloud-platform.service.justice.gov.uk/resolve-opensearch-shard-issues.html)