separate doc-level monitor query indices created by detectors #1324

sbcd90 · 2024-09-25T03:20:15Z

Description

separate doc-level monitor query indices created by detectors

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Subhobrata Dey <[email protected]>

eirsep · 2024-09-25T06:46:12Z

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

@@ -887,7 +887,7 @@ private IndexMonitorRequest createDocLevelMonitorMatchAllRequest(

        Monitor monitor = new Monitor(monitorId, Monitor.NO_VERSION, monitorName, false, detector.getSchedule(), detector.getLastUpdateTime(), null,
                Monitor.MonitorType.DOC_LEVEL_MONITOR.getValue(), detector.getUser(), 1, docLevelMonitorInputs, triggers, Map.of(),
-                new DataSources(detector.getRuleIndex(),
+                new DataSources(detector.getRuleIndex() + "_chained_findings",


why do we need separate index?

each doc-level monitor has its own query index now. So the chained findings doc-level monitor also has its own index. The Sigma Rules doc-level monitor & the bucket-level monitor's chained monitor run asynchronously & thus cannot share the same query index as it is created & deleted in every run.

eirsep · 2024-09-25T06:46:38Z

src/main/java/org/opensearch/securityanalytics/config/monitors/DetectorMonitorConfig.java

+    }
+
+    public static String getRuleIndexOptimized(String logType) {
+        return String.format(Locale.getDefault(), ".opensearch-sap-%s-detectors-queries-optimized-%s", logType, UUID.randomUUID());


we are moving from index patttern to index?

no. this method is utilized by few integ tests & hence the change.

eirsep · 2024-09-25T06:48:18Z

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

@@ -1369,7 +1369,11 @@ void onGetResponse(Detector currentDetector, User user) {
            request.getDetector().setAlertsHistoryIndexPattern(DetectorMonitorConfig.getAlertsHistoryIndexPattern(ruleTopic));
            request.getDetector().setFindingsIndex(DetectorMonitorConfig.getFindingsIndex(ruleTopic));
            request.getDetector().setFindingsIndexPattern(DetectorMonitorConfig.getFindingsIndexPattern(ruleTopic));
-            request.getDetector().setRuleIndex(DetectorMonitorConfig.getRuleIndex(ruleTopic));
+            if (currentDetector.getRuleIndex().contains("optimized")) {


if name contains optimized shouldn't we set rule index name as getRuleIndexOptimized value?

is this if else condition flipped?

the if-else condition checks if the detector is already using a dedicated query index, then keep on using it. else, switch from shared query index to new dedicated query index as the detector is updated.

eirsep · 2024-09-25T06:49:40Z

src/test/java/org/opensearch/securityanalytics/DetectorThreatIntelIT.java

-                "     }\n" +
-                "   }\n" +
-                "}";
-        SearchResponse response = executeSearchAndGetResponse(DetectorMonitorConfig.getRuleIndex(randomDetectorType()), request, true);


why has this been removed ?

Plz dont reduce test coverage..
fix test as necessary

query indices are created & deleted now. So, it is difficult to assert number of queries in query index now.

eirsep

what happens when someone updates a detector that doesnt have exclusive query index?

eirsep · 2024-09-25T07:05:06Z

src/test/java/org/opensearch/securityanalytics/alerts/AlertsIT.java

@@ -825,21 +825,10 @@ public void testMultipleAggregationAndDocRules_alertSuccess() throws IOException

        Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector));

-
-        String request = "{\n" +


plz revert and fix test as necessary

query indices are created & deleted now. So, it is difficult to assert number of queries in query index now.

eirsep · 2024-09-25T07:05:32Z

src/test/java/org/opensearch/securityanalytics/findings/FindingIT.java

@@ -1209,21 +1209,11 @@ public void testCreateDetectorWithNotCondition_verifyFindingsAndNoFindings_succe

        Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector));

-        String request = "{\n" +


query indices are created & deleted now. So, it is difficult to assert number of queries in query index now.

Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 · 2024-09-25T11:18:59Z

what happens when someone updates a detector that doesnt have exclusive query index?

The detector is updated to use exclusive query index.

Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 · 2024-09-26T00:49:25Z

security tests fail as docker image not updated.

eirsep · 2024-09-26T01:16:55Z

looks like we create aliases not indices from 'queryIndex` field in monitor datasources

eirsep · 2024-09-26T01:24:59Z

can we add tests for scenarios

setting is off. create 2 detectors. verify query index name. turn on setting. update 1 detector, verify query index name.
setting is on. create 1 detector. verify query index name. turn off setting. update detector, verify query index name. create new detector and verify query index(should be old style since setting is off)

eirsep · 2024-09-26T00:52:29Z

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

+                request.getDetector().setRuleIndex(DetectorMonitorConfig.getRuleIndexOptimized(ruleTopic));
+            } else {
+                String ruleTopicIndex = DetectorMonitorConfig.getRuleIndex(ruleTopic);
+                request.getDetector().setRuleIndex(ruleTopicIndex.substring(0, ruleTopicIndex.length()-1));


why did we change the existing code from
request.getDetector().setRuleIndex(DetectorMonitorConfig.getRuleIndex(ruleTopic));

ruleIndex is changing to dedicated ruleIndexOptimzied now. Thats why if-else condition.

eirsep · 2024-09-26T01:36:04Z

src/main/java/org/opensearch/securityanalytics/config/monitors/DetectorMonitorConfig.java

@@ -22,7 +24,11 @@ public class DetectorMonitorConfig {
    public static final String OPENSEARCH_SAP_RULE_INDEX_TEMPLATE = ".opensearch-sap-detectors-queries-index-template";

    public static String getRuleIndex(String logType) {
-        return String.format(Locale.getDefault(), ".opensearch-sap-%s-detectors-queries", logType);
+        return String.format(Locale.getDefault(), ".opensearch-sap-%s-detectors-queries*", logType);


why *
this is an alias

eirsep · 2024-09-26T01:37:21Z

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

@@ -1369,7 +1379,16 @@ void onGetResponse(Detector currentDetector, User user) {
            request.getDetector().setAlertsHistoryIndexPattern(DetectorMonitorConfig.getAlertsHistoryIndexPattern(ruleTopic));
            request.getDetector().setFindingsIndex(DetectorMonitorConfig.getFindingsIndex(ruleTopic));
            request.getDetector().setFindingsIndexPattern(DetectorMonitorConfig.getFindingsIndexPattern(ruleTopic));
-            request.getDetector().setRuleIndex(DetectorMonitorConfig.getRuleIndex(ruleTopic));
+            if (currentDetector.getRuleIndex().contains("optimized")) {
+                request.getDetector().setRuleIndex(currentDetector.getRuleIndex());


plz add comment that if we turn OFF setting after turning on, updating detector will not change from optimized to normal and we need to re-create detector

added a comment. added a test too.

eirsep · 2024-09-26T01:37:38Z

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

+                    request.getDetector().setRuleIndex(DetectorMonitorConfig.getRuleIndexOptimized(ruleTopic));
+                } else {
+                    String ruleTopicIndex = DetectorMonitorConfig.getRuleIndex(ruleTopic);
+                    request.getDetector().setRuleIndex(ruleTopicIndex.substring(0, ruleTopicIndex.length() - 1));


this line is very confusing

eirsep · 2024-09-26T01:38:15Z

src/test/java/org/opensearch/securityanalytics/SecurityAnalyticsRestTestCase.java

@@ -266,6 +266,7 @@ void setDebugLogLevel() throws IOException, InterruptedException {


        makeRequest(client(), "PUT", "_cluster/settings", Collections.emptyMap(), se, new BasicHeader("Content-Type", "application/json"));
+        updateClusterSetting("plugins.security_analytics.enable_detectors_with_dedicated_query_indices", "true");


are there tests with setting turned off

added a test for setting turned off.

eirsep · 2024-09-26T01:39:10Z

src/test/java/org/opensearch/securityanalytics/mapper/MapperRestApiIT.java

@@ -1722,12 +1725,16 @@ public void testAzureMappings() throws IOException {
        DetectorInput input = new DetectorInput("windows detector for security analytics", List.of(indexName), List.of(),
                getPrePackagedRules("azure").stream().map(DetectorRule::new).collect(Collectors.toList()));
        Detector detector = randomDetectorWithInputs(List.of(input), "azure");
-        createDetector(detector);
-
-        List<SearchHit> hits = executeSearch(".opensearch-sap-azure-detectors-queries-000001", matchAllSearchBody);


can you retain this test and add a new one - basically test for both setting turned. on and turned off

added scenario for flag turned off in test cases. updated this test case to fetch queries from index pattern. https://github.com/sbcd90/security-analytics/blob/query_idx/src/test/java/org/opensearch/securityanalytics/mapper/MapperRestApiIT.java#L1727

eirsep · 2024-09-26T01:40:02Z

src/test/java/org/opensearch/securityanalytics/resthandler/DetectorMonitorRestApiIT.java

@@ -1104,21 +1094,11 @@ public void testCreateDetector_verifyWorkflowCreation_success_WithGroupByRulesIn

        Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector));

-        String request = "{\n" +


we should not remove this code. we lose code coverage of tests. that's an anti-pattern. let's fix these tests

Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 · 2024-09-26T08:57:48Z

can we add tests for scenarios

setting is off. create 2 detectors. verify query index name. turn on setting. update 1 detector, verify query index name. setting is on. create 1 detector. verify query index name. turn off setting. update detector, verify query index name. create new detector and verify query index(should be old style since setting is off)

added both test cases.

1st scenario: https://github.com/sbcd90/security-analytics/blob/c514a780d4b3f65ac828e3ff868c56f46f2a82b7/src/test/java/org/opensearch/securityanalytics/resthandler/DetectorRestApiIT.java#L1729
2nd scenario: https://github.com/sbcd90/security-analytics/blob/c514a780d4b3f65ac828e3ff868c56f46f2a82b7/src/test/java/org/opensearch/securityanalytics/resthandler/DetectorRestApiIT.java#L1831

eirsep · 2024-09-26T09:06:46Z

src/main/java/org/opensearch/securityanalytics/util/RuleTopicIndices.java

@@ -83,7 +83,8 @@ private void getAllRuleIndices(ActionListener<List<String>> listener) {
            listener.onResponse(
                    logTypes
                        .stream()
-                        .map(logType -> DetectorMonitorConfig.getRuleIndex(logType))
+                            // use index pattern here to define rule topic index template for all query indices which match the pattern
+                        .map(logType -> DetectorMonitorConfig.getRuleIndex(logType) + "*")


shouldn't this be detector specific?
why do we have this method? getAllRuleIndices

a single index template containing specific settings for sigma rule executions are applied to all rule indices.

eirsep · 2024-09-26T16:50:00Z

CI is failing

sbcd90 · 2024-09-26T17:50:56Z

CI is failing

ci passed now.

eirsep · 2024-09-26T19:45:37Z

Security tests failed

Signed-off-by: Subhobrata Dey <[email protected]> (cherry picked from commit 1ca4090) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

opensearch-trigger-bot · 2024-09-26T20:49:33Z

The backport to 2.15 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security-analytics/backport-2.15 2.15
# Navigate to the new working tree
pushd ../.worktrees/security-analytics/backport-2.15
# Create a new branch
git switch --create backport-1324-to-2.15
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1ca40908ac7b050cc7eb90df3108ab8a21a1f6d1
# Push it to GitHub
git push --set-upstream origin backport-1324-to-2.15
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security-analytics/backport-2.15

Then, create a pull request where the base branch is 2.15 and the compare/head branch is backport-1324-to-2.15.

opensearch-trigger-bot · 2024-09-26T20:49:34Z

The backport to 2.16 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security-analytics/backport-2.16 2.16
# Navigate to the new working tree
pushd ../.worktrees/security-analytics/backport-2.16
# Create a new branch
git switch --create backport-1324-to-2.16
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1ca40908ac7b050cc7eb90df3108ab8a21a1f6d1
# Push it to GitHub
git push --set-upstream origin backport-1324-to-2.16
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security-analytics/backport-2.16

Then, create a pull request where the base branch is 2.16 and the compare/head branch is backport-1324-to-2.16.

…arch-project#1324) Signed-off-by: Subhobrata Dey <[email protected]>

…#1330) Signed-off-by: Subhobrata Dey <[email protected]>

…#1330) Signed-off-by: Subhobrata Dey <[email protected]> (cherry picked from commit 038d60a)

…arch-project#1324) Signed-off-by: Subhobrata Dey <[email protected]>

…#1344) Signed-off-by: Subhobrata Dey <[email protected]>

…#1330) (#1343) Signed-off-by: Subhobrata Dey <[email protected]> (cherry picked from commit 038d60a) Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>

ELKHostmaster · 2024-11-21T17:05:10Z

@sbcd90 Good morning,
Regarding this change to code.... We have a large cluster running lots of SIGMA rule security detectors. We recently upgraded from 2.13 to 2.16 but the cluster and detector performance has been absolutely terrible ever since. We see two nodes from our cluster that seem to perform the doclevel fan out and these tasks run for days until nodes in the cluster start having heap issues, circuit breakers, nodes dropping out, etc.

It appears in 2.14 that the change to document level fanout opensearch-project/alerting#1521 could be the issue.

Is the change above - to the way fan outs are handled an attempt to fix the performance of the security analytics detections?

sbcd90 · 2024-11-21T18:05:04Z

hi @ELKHostmaster , please bounce(stop and start) OpenSearch process in every data node. This would cleanup those tasks.

Please delete(after deleting of detectors you may also want to delete .opensearch-sap-detectors* indices & cleanup monitors linked to the 2 detectors you mentioned from .opendistro-alerting-config index) & re-create detectors on a rolling index alias with write index pointing to the current index where data is ingested(https://opensearch.org/docs/latest/im-plugin/index-alias/#index-alias-options )

In the end, it should result in better performance.

mimicbox · 2024-11-21T18:17:46Z

Hi @sbcd90

I am having a simmilar issue to this and have been following this. I want to try your suggestion but the part I do not get it is e-

create detectors on a rolling index alias with write index pointing to the current index where data is ingested

How do I define this during detector creation? I use the API to create detectors. Are you talking about the index pattern you define in the inputs.detector_input.indices value? I currently use an alias that points to a datastream.

Thank you for any help at all!

sbcd90 · 2024-11-21T18:24:05Z

hi @mimicbox , if you're currently using an index pattern with your detector, just change it to an index alias instead. the term rolling refers to index alias with rollovers.
e.g.
we have an index,

api_output-000001 -> currently points as write index for index alias api_output
now, index rolls over after sometime to ,
api_output-000002 , write index also moves if you're using an ISM policy for rollover. https://opensearch.org/docs/latest/im-plugin/ism/policies/#rollover

provide api_output as input to the detector. if there is no auto suggestion, manually type it into the text box.
Lastly, keep index alias name api_output a prefix of the index names behind it e.g api_output-000001, api_output-000002... so on, or api_output-test-000001, api_output-test-000002... so on.

ELKHostmaster · 2024-11-21T18:40:16Z

@sbcd90 If I understand correctly, your saying point the detector to the index alias and use ISM to rollover the indices at regular intervals.

I'm already doing that. There is something related to the fan-out change introduced in 2.14 that I think has really caused the performance issues.

Is this fixed in 2.18? Is upgrading from 2.16 a possible solution?

sbcd90 · 2024-11-21T18:56:34Z

@ELKHostmaster , are you using Sigma Aggregation Rules?

ELKHostmaster · 2024-11-21T19:03:04Z

@sbcd90 do you mean - using any of the aggregation functions in sigma rules (https://help.sigmacomputing.com/docs/aggregate-functions) ?

No, I dont believe so but I am scanning through them now.

ELKHostmaster · 2024-11-21T19:04:09Z

No correlation rules either. I've stripped it down to just our sigma detections as we've been trying to get to the bottom of the issue for the last month.

sbcd90 · 2024-11-21T19:07:30Z

hi @ELKHostmaster , please bounce(stop and start) OpenSearch process in every data node. This would cleanup those tasks.
please re-create(delete & create) the detectors. This will help cleanup old metadata.
I would suggest move to an odd release. e.g, 2.15 or 2.17 before doing these steps. They will have the latest fixes.

ELKHostmaster · 2024-11-21T19:15:02Z

@sbcd90 thank you. yes, I have stopped/started every data node several times over the last month. its the only way to keep the cluster running at this point. I'm currently on 2.16

I thought this fix was added in 2.18. Are you saying it was backported to 2.17.1 ? I didn't think that backports resulted in a new release package being built. So, I didn't think the fix would be included.

Also, when you say "odd" release... is there something different about odd/even releases?

Thanks again for the responses!

sbcd90 · 2024-11-21T19:19:27Z

hi @ELKHostmaster , i dont think data nodes need to be stopped/started frequently. i would say take a thread dump of a data node when doc-level fanout tasks are stuck. That would help root cause the issue.

yes. i think all these fixes should be in 2.18. But to stay up-to-date with future fixes, following an odd release helps.

ELKHostmaster · 2024-11-22T03:14:38Z

@sbcd90 Good evening. I waited until a couple data nodes got really busy and took jstack dumps for you to look at. I also included a list of tasks running on the cluster. You can see a lot of tasks are running for 8 hours now and growing since the last reboot of the nodes. I appreciate any help you can give. Thanks!
_cat_tasks.txt
data-21.txt
data-13.txt

ELKHostmaster · 2024-11-26T15:52:33Z

@sbcd90 Good morning. Could I ask a favor of you? Could you take a look at the thread dumps above and see if the issue is related to the fanout change? Our cluster is unusable right now with detectors enabled/running. Circuit breaker issues, java heap exceptions, nodes dropping out, 99% cpu on data nodes, etc. This was an upgrade from 2.13 where everything was fine. The only change was upgrading to 2.16. Same detectors/rules, etc.

Do you think the fixes to fanout in 2.18 will help us?

Thank You in advance!

separate doc-level monitor query indices created by detectors

8e6ac76

Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 requested review from amsiglan, AWSHurneyt, getsaurabh02, lezzago, praveensameneni, eirsep, jowg-amazon, engechas, goyamegh and riysaxen-amzn as code owners September 25, 2024 03:20

eirsep reviewed Sep 25, 2024

View reviewed changes

eirsep requested changes Sep 25, 2024

View reviewed changes

eirsep reviewed Sep 25, 2024

View reviewed changes

separate doc-level monitor query indices created by detectors

b3284e7

Signed-off-by: Subhobrata Dey <[email protected]>

finalize changes

1f3eb29

Signed-off-by: Subhobrata Dey <[email protected]>

eirsep requested changes Sep 26, 2024

View reviewed changes

address review comments

c514a78

Signed-off-by: Subhobrata Dey <[email protected]>

eirsep reviewed Sep 26, 2024

View reviewed changes

AWSHurneyt approved these changes Sep 26, 2024

View reviewed changes

sbcd90 added backport 2.x backport 2.15 backport 2.16 labels Sep 26, 2024

sbcd90 merged commit 1ca4090 into opensearch-project:main Sep 26, 2024
13 of 19 checks passed

opensearch-trigger-bot bot mentioned this pull request Sep 26, 2024

[Backport 2.x] separate doc-level monitor query indices created by detectors #1330

Merged

opensearch-trigger-bot bot added the failed backport label Sep 26, 2024

sbcd90 added a commit to sbcd90/security-analytics that referenced this pull request Sep 26, 2024

separate doc-level monitor query indices created by detectors (opense…

521a424

…arch-project#1324) Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 pushed a commit that referenced this pull request Sep 30, 2024

separate doc-level monitor query indices created by detectors (#1324) (…

038d60a

…#1330) Signed-off-by: Subhobrata Dey <[email protected]>

opensearch-trigger-bot bot added a commit that referenced this pull request Oct 7, 2024

separate doc-level monitor query indices created by detectors (#1324) (…

89e03e2

…#1330) Signed-off-by: Subhobrata Dey <[email protected]> (cherry picked from commit 038d60a)

sbcd90 added a commit to sbcd90/security-analytics that referenced this pull request Oct 7, 2024

separate doc-level monitor query indices created by detectors (opense…

0ca18a9

…arch-project#1324) Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 added a commit that referenced this pull request Oct 7, 2024

separate doc-level monitor query indices created by detectors (#1324) (…

b003ebb

…#1344) Signed-off-by: Subhobrata Dey <[email protected]>

		@@ -825,21 +825,10 @@ public void testMultipleAggregationAndDocRules_alertSuccess() throws IOException

		Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector));


		String request = "{\n" +

		@@ -1209,21 +1209,11 @@ public void testCreateDetectorWithNotCondition_verifyFindingsAndNoFindings_succe

		Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector));

		String request = "{\n" +

		@@ -266,6 +266,7 @@ void setDebugLogLevel() throws IOException, InterruptedException {


		makeRequest(client(), "PUT", "_cluster/settings", Collections.emptyMap(), se, new BasicHeader("Content-Type", "application/json"));
		updateClusterSetting("plugins.security_analytics.enable_detectors_with_dedicated_query_indices", "true");

		@@ -1104,21 +1094,11 @@ public void testCreateDetector_verifyWorkflowCreation_success_WithGroupByRulesIn

		Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector));

		String request = "{\n" +

separate doc-level monitor query indices created by detectors #1324

separate doc-level monitor query indices created by detectors #1324

Conversation

sbcd90 commented Sep 25, 2024

Description

Related Issues

Check List

Choose a reason for hiding this comment

sbcd90 Sep 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eirsep Sep 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eirsep left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbcd90 commented Sep 25, 2024

sbcd90 commented Sep 26, 2024

eirsep commented Sep 26, 2024

eirsep commented Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbcd90 Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbcd90 commented Sep 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eirsep commented Sep 26, 2024

sbcd90 commented Sep 26, 2024

eirsep commented Sep 26, 2024

opensearch-trigger-bot bot commented Sep 26, 2024

opensearch-trigger-bot bot commented Sep 26, 2024

ELKHostmaster commented Nov 21, 2024

sbcd90 commented Nov 21, 2024 • edited Loading

mimicbox commented Nov 21, 2024

sbcd90 commented Nov 21, 2024

ELKHostmaster commented Nov 21, 2024

sbcd90 commented Nov 21, 2024

ELKHostmaster commented Nov 21, 2024

ELKHostmaster commented Nov 21, 2024

sbcd90 commented Nov 21, 2024

ELKHostmaster commented Nov 21, 2024

sbcd90 commented Nov 21, 2024

ELKHostmaster commented Nov 22, 2024 • edited Loading

ELKHostmaster commented Nov 26, 2024

sbcd90 Sep 25, 2024 •

edited

Loading

eirsep Sep 25, 2024 •

edited

Loading

eirsep commented Sep 26, 2024 •

edited

Loading

sbcd90 Sep 26, 2024 •

edited

Loading

sbcd90 commented Nov 21, 2024 •

edited

Loading

ELKHostmaster commented Nov 22, 2024 •

edited

Loading