Security Analytics DNS created hundreds of indexs with thousands of shards hitting limit, deleted dectors but indexes are still present #667
Comments
zephyia
changed the title
|
Issue is due to alerting. |
|
I had this problem with cloudtrail detectors, and I'm also using AWS managed OpenSearch. This occurred when I had a detector running that had some mappings it couldn't automatically map, and I didn't map them to anything. (This apparently caused something in the detector to fail, and it auto retried by creating a new .opensearch-sap-cloudtrail-detectors-queries-00NNNN file. Which then failed, and it retried and created another new one. Which then failed, and ...) To understand what's going on, and how to clean it up, I found these commands in the DEV TOOLS very helpful:
Here's the steps I followed to clean up:
If you are not on OpenSearch 2.7 or later, you need to use DEV TOOLS
|
|
Thanks for taking the time to write such a comprehensive response - in the end we punted the issue to AWS support and let them deal with it . But i am sure the below will be very helpful to future people that run into this issue.I still think that the ability for the system to get into this state in the first place is a bug that should be addressed.Thanksc.On 23 Nov 2023, at 08:44, wkirke ***@***.***> wrote:
I had this problem with cloudtrail detectors, and I'm also using AWS managed OpenSearch. This occurred when I had a detector running that had some mappings it couldn't automatically map, and I didn't map them to anything. (This apparently caused something in the detector to fail, and it auto retried by creating a new .opensearch-sap-cloudtrail-detectors-queries-00NNNN file. Which then failed, and it retried and created another new one. Which then failed, and ...)
To understand what's going on, and how to clean it up, I found these commands in the DEV TOOLS very helpful:
GET /_cat/shards
list all shards, you can see the various things that weren't cleaned up when the detectors were deleted
GET /_cat/indices?v
list all non-hidden indexes
GET /_cat/indices/.*?v
list all indexes that begin with a dot, whether they are hidden or not (this will list the problem detector queries indexes. They need deleted)
GET /_cat/aliases
list all aliases (the security analytics detector queries run against these, not against the actual indexes)
GET /_cat/templates?v
list all index templates, and what they are composed of.
Here's the steps I followed to clean up:
Delete the detector (using the Security Analytics GUI)
Delete all dns detector query indexes
in DEV TOOLS execute this:
DELETE .opensearch-sap-dns-detectors-queries-*
Run the GET /_cat/shards to verify they are now gone.
Delete all dns findings indexes and alerts indexes
in DEV TOOLS execute these:
DELETE .opensearch-sap-dns-findings*
DELETE .opensearch-sap-dns-alerts*
Sometimes aliases don't get deleted, look for them via this:
GET /_cat/aliases/dns?v
Manually cross reference the indexes against the results of GET /_cat/indices/?v and GET /_cat/indices/.?v (with and without the dot). Identify any aliases that point to indexes that don't exist. Delete them via the menu > index management > aliases GUI, or in DEV TOOLS via:
POST /_aliases?pretty
{
"actions": [
{
"remove": {
"index": "index_that_doesn't_exist",
"alias": "alias_that_points_to_it"
}
}
]
}
I found that creating a detector MODIFIES the index template that it is told to run against, and I needed to manually clean these up before creating another detector:
If you are on OpenSearch 2.7 or later you can use the GUI:
Menu > Index Management > Templates > select your template (possibly log-aws-dns_aws)
Select the "Configuration" tab > select the "Component template" box.
Find the component called .opensearch-sap-alias-mappings-component-log-aws-dns-, and delete it.
If you are not on OpenSearch 2.7 or later, you need to use DEV TOOLS
Dump your template via:
GET /_index_template/log-aws-cloudtrail_aws
Copy the result into an editor, remove the component that starts with .opensearch-sap, and push it back via the DEV TOOLS command as shown here:
https://opensearch.org/docs/latest/im-plugin/index-templates/#use-component-templates-to-create-an-index-template
Make sure you ingest a log record that has all the fields that the detector will be looking for. Then refresh your index template via:
Menu > Dashboard Management > Index patterns > Select your index
Click the "refresh field list" circular arrow at the upper right.
Create the detector again, and make sure all fields are mapped (they should all auto map if you have updated your index pattern fields with everything needed. If they don't auto map, make sure you map every one to something to avoid the problem you just cleaned up)
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>
|
What is the bug?
After deleting Security Analytics DNS dectors the internal indexes have not been removed. Also why on earth does it create so many indexes (shards). Our cluster is now unuseable as we have reached the shard limit and we don't know how to remove all these indexes. The Security Analytics service shouldnt be creating so many indexes and it really should be cleaning them up. How remove all of these and clean them up to restore function to our cluster?
Here is the shard list of all the shards Security Analytics has created (it has been enabled for about 2 months) Even after deleting the dectors these are all still present:
index shard prirep state docs store ip node
.opensearch-sap-dns-detectors-queries-000502 1 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000502 1 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000502 4 p STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000502 4 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000502 3 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000502 3 r STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000502 2 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000502 2 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000502 0 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000502 0 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000503 1 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000503 1 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000503 4 r STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000503 4 p STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000503 3 r STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000503 3 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000503 2 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000503 2 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000503 0 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000503 0 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000500 1 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000500 1 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000500 4 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000500 4 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000500 3 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000500 3 r STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000500 2 r STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000500 2 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000500 0 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000500 0 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000501 1 r STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000501 1 p STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000501 3 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000501 3 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000501 4 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000501 4 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000501 2 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000501 2 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000501 0 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000501 0 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000506 1 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000506 1 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000506 4 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000506 4 r STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000506 3 r STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000506 3 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000506 2 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000506 2 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000506 0 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000506 0 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000507 1 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000507 1 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000507 4 r STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000507 4 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000507 3 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000507 3 r STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000507 2 p STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000507 2 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000507 0 p STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000507 0 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000504 1 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000504 1 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000504 4 r STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000504 4 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000504 3 p STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000504 3 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000504 2 p STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000504 2 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000504 0 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000504 0 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000505 1 r STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000505 1 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000505 4 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000505 4 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000505 3 r STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000505 3 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000505 2 r STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000505 2 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000505 0 p STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000505 0 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000620 1 r STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000620 1 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000620 3 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000620 3 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000620 4 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000620 4 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000620 2 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000620 2 r STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000620 0 p STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000620 0 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-findings-2023.10.07-000005 1 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-findings-2023.10.07-000005 1 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-findings-2023.10.07-000005 4 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-findings-2023.10.07-000005 4 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-findings-2023.10.07-000005 3 r STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-findings-2023.10.07-000005 3 p STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-findings-2023.10.07-000005 2 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-findings-2023.10.07-000005 2 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-findings-2023.10.07-000005 0 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-findings-2023.10.07-000005 0 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000508 1 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000508 1 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000508 4 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000508 4 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000508 3 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000508 3 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000508 2 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000508 2 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000508 0 r STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000508 0 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000509 1 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000509 1 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000509 4 r STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000509 4 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000509 3 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000509 3 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000509 2 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000509 2 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000509 0 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000509 0 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000612 1 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000612 1 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000612 4 r STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000612 4 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000612 3 r STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000612 3 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000612 2 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000612 2 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000612 0 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000612 0 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000613 3 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000613 3 r STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000613 4 r STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000613 4 p STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000613 1 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000613 1 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000613 2 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000613 2 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000613 0 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000613 0 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000610 1 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000610 1 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000610 4 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000610 4 r STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000610 3 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000610 3 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000610 2 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000610 2 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000610 0 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000610 0 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000611 1 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000611 1 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000611 4 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000611 4 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000611 3 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000611 3 r STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000611 2 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000611 2 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000611 0 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000611 0 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000616 3 p STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000616 3 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000616 4 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000616 4 r STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000616 1 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000616 1 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000616 2 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000616 2 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000616 0 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000616 0 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000617 1 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000617 1 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000617 4 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000617 4 r STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000617 3 p STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000617 3 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000617 2 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000617 2 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000617 0 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000617 0 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000614 1 p STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000614 1 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000614 4 r STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000614 4 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000614 3 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000614 3 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000614 2 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000614 2 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000614 0 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000614 0 p STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000615 1 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000615 1 r STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-detectors-queries-000615 3 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-detectors-queries-000615 3 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000615 4 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-detectors-queries-000615 4 p STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-detectors-queries-000615 2 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-detectors-queries-000615 2 r STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-detectors-queries-000615 0 p STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-detectors-queries-000615 0 r STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-alerts 3 r STARTED x.x.x.x 9606a706c224bf11928d794c29d08ec7
.opensearch-sap-dns-alerts 3 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-alerts 4 p STARTED x.x.x.x 8783318a8292570385c9c6aef267f820
.opensearch-sap-dns-alerts 4 r STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-alerts 1 p STARTED x.x.x.x 36a5c6163f53c2af7747b469b952902f
.opensearch-sap-dns-alerts 1 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
.opensearch-sap-dns-alerts 2 p STARTED x.x.x.x 186cdbb8f87076ca90a8159345046b8e
.opensearch-sap-dns-alerts 2 r STARTED x.x.x.x 8c7afd08634f4daa99aa951f9830ecaa
.opensearch-sap-dns-alerts 0 p STARTED x.x.x.x a73b20f73a95ad006a43bf067b29b5b6
.opensearch-sap-dns-alerts 0 r STARTED x.x.x.x 46a4ba8860cda4594a5b818a0c8d1752
<truncate ~7 thousand more lines of shards .opensearch-sap-* shards>
All in all in about 2 months the DNS detector created enough indexs that it consume over 7000 shards on the cluster.
How can one reproduce the bug?
Steps to reproduce the behavior:
Enable Security Analytics, configure DNS detector, leave running for several weeks, then attempt to delete them.
What is the expected behavior?
When the detector is deleted it cleans up its indexes, also when a detector is in place the indexes are managed more efficiently such as not to exhaust the shard limit of the cluster
What is your host/environment?
Managed opensearch on AWS, v2.9.0
The text was updated successfully, but these errors were encountered: