[BUG] Monitors notifications channel not working With RBAC setting

[techrna]

Bug
While Creating Monitors in that Triggers actions notification channel is not working. I have created the notification channel as custom webhook. When we hit send test message over there it works .But when do send test message while adding monitor it fails .

How can one reproduce the bug?
Steps to reproduce the behavior:

1. enable backend role filtering
   PUT _cluster/settings
   {
   "transient": {
   "plugins.alerting.filter_by_backend_roles": "true"
   }
   }
2. Then Create monitor
3. Add the trigger with action as notification channel as webhook
4. Try to send the test message will throw an error.and also check the opensearch.log file
   Error screenshots and logs are attach below please check.

Problem is it fails to read the backend roles for the user.when filter_by_backend_roles is enabled

host/environment

• OS: CentOS 7.9
• Version 7.9
• Plugins Alert Plugin
• Opensearch 2.3

screenshots?

OpenSearch logs
[2022-12-09T18:59:50,893][INFO ][o.o.n.i.ConfigIndexingActions] [test2-server] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@43c2eec6
[2022-12-09T18:59:50,893][WARN ][o.o.n.a.PluginBaseAction ] [test2-server] notifications:OpenSearchStatusException:
org.opensearch.OpenSearchStatusException: User doesn't have backend roles configured. Contact administrator.
at org.opensearch.notifications.security.UserAccessManager.validateUser(UserAccessManager.kt:24) ~[opensearch-notifications-2.3.0.0.jar:2.3.0.0]
at org.opensearch.notifications.index.ConfigIndexingActions.get(ConfigIndexingActions.kt:251) ~[opensearch-notifications-2.3.0.0.jar:2.3.0.0]
at org.opensearch.notifications.action.GetNotificationConfigAction.executeRequest(GetNotificationConfigAction.kt:60) ~[opensearch-notifications-2.3.0.0.jar:2.3.0.0]
at org.opensearch.notifications.action.GetNotificationConfigAction.executeRequest(GetNotificationConfigAction.kt:26) ~[opensearch-notifications-2.3.0.0.jar:2.3.0.0]
at org.opensearch.notifications.action.PluginBaseAction$doExecute$1.invokeSuspend(PluginBaseAction.kt:63) [opensearch-notifications-2.3.0.0.jar:2.3.0.0]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
[2022-12-09T18:59:50,894][INFO ][o.o.a.m.MonitorRunResult ] [test2-server] Internal error: Unable to find a Notification Channel or Destination config with id [nsgT94QBikbo0yuk80z2]. See the opensearch.log for details
java.lang.IllegalStateException: Unable to find a Notification Channel or Destination config with id [nsgT94QBikbo0yuk80z2]
at org.opensearch.alerting.MonitorRunner.getConfigAndSendNotification(MonitorRunner.kt:93) ~[opensearch-alerting-2.3.0.0.jar:2.3.0.0]
at org.opensearch.alerting.MonitorRunner$getConfigAndSendNotification$1.invokeSuspend(MonitorRunner.kt) ~[opensearch-alerting-2.3.0.0.jar:2.3.0.0]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]

[lezzago]

@techrna, I have a few questions.

• Can you please paste the results for the getUser API for the user that is making the call?
• When testing it from the Notification side, was the setting still enabled?
• Was the setting, opensearch.notifications.general.filter_by_backend_roles, set? If so, to what value?

[techrna]

Thank you!!. Bellow are the details requested.And please let me know if anything else required.

Get user api response
GET _plugins/_security/api/internalusers/[email protected]

{
  "[email protected]" : {
    "hash" : "",
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "dgvzdf8xx2toawthlte"
    ],
    "attributes" : { },
    "opendistro_security_roles" : [ ],
    "static" : false
  }
}

Role api response
GET _plugins/_security/api/roles/dgvzdf8xx2toawthlte

{
  "dgvzdf8xx2toawthlte" : {
    "reserved" : false,
    "hidden" : false,
    "cluster_permissions" : [
      "cluster:admin/opendistro/alerting/monitor/search",
      "cluster:admin/opendistro/alerting/monitor/write",
      "cluster:admin/opendistro/alerting/alerts/get",
      "cluster:admin/opendistro/alerting/monitor/get",
      "cluster:admin/opendistro/alerting/monitor/execute",
      "cluster:admin/opendistro/alerting/destination/get",
      "cluster:admin/opensearch/notifications/*",
      "cluster:monitor/*"
    ],
    "index_permissions" : [
      {
        "index_patterns" : [
          "*-dgvzdf8xx2toawthlte-*"
        ],
        "dls" : "",
        "fls" : [ ],
        "masked_fields" : [ ],
        "allowed_actions" : [
          "unlimited"
        ]
      }
    ],
    "tenant_permissions" : [
      {
        "tenant_patterns" : [
          "dgvzdf8xx2toawthlte"
        ],
        "allowed_actions" : [
          "kibana_all_write"
        ]
      }
    ],
    "static" : false
  }
}

Cluster Settings.
GET _cluster/settings

{
  "persistent" : {
    "plugins" : {
      "index_state_management" : {
        "template_migration" : {
          "control" : "-1"
        }
      },
      "alerting" : {
        "filter_by_backend_roles" : "true"
      }
    }
  },
  "transient" : {
    "plugins" : {
      "alerting" : {
        "filter_by_backend_roles" : "true"
      }
    }
  }
}

notification.yml file setting (Hope we have added the settings in right file(path: /etc/opensearch/opensearch-notifications/notifications.yml))

opensearch.notifications:
  general:
    operation_timeout_ms: 60000 # 60 seconds, Minimum 100ms
    default_items_query_count: 100 # default number of items to query
    filter_send_by_backend_roles: true # Does sendNotification needs to validate user's backend roles
    filter_by_backend_roles: true

[lezzago]

@techrna, can you run the following: GET .opendistro-alerting-config/_doc/<monitor-id>?
The reason to do this is to get the user information stored on the monitor and to see if the backend role is correctly added there as it should be passed to the Notification plugin from there.

Additionally, can you do the [getRoleMapping](GET .opendistro-alerting-config/_doc/c3CwFYQBxyWm8H1EhccS) API for the role, dgvzdf8xx2toawthlte? Just want to make sure that role is associated to the user you shared.

One last question, if you create the monitor with that notification channel associated to the trigger, does the monitor correctly send out notifications when alerts are generated?

[techrna]

@lezzago

Monitor id : HbWuYYUBb9hzdJ2RpaJ5
monitor_type : query_level_monitor

GET .opendistro-alerting-config/_doc/HbWuYYUBb9hzdJ2RpaJ5

{
  "_index" : ".opendistro-alerting-config",
  "_id" : "HbWuYYUBb9hzdJ2RpaJ5",
  "found" : false
}

Monitor Json

GET _plugins/_alerting/monitors/HbWuYYUBb9hzdJ2RpaJ5

{
  "_id" : "HbWuYYUBb9hzdJ2RpaJ5",
  "_version" : 2,
  "_seq_no" : 833,
  "_primary_term" : 35,
  "monitor" : {
    "type" : "monitor",
    "schema_version" : 5,
    "name" : "linux_multiple_logon_failures [workstation]",
    "monitor_type" : "query_level_monitor",
    "enabled" : true,
    "enabled_time" : 1672230731677,
    "schedule" : {
      "period" : {
        "interval" : 1,
        "unit" : "MINUTES"
      }
    },
    "inputs" : [
      {
        "search" : {
          "indices" : [
            "rushab-dgvzdf8zx2toawthltg-workstation-archives-*"
          ],
          "query" : {
            "size" : 100,
            "query" : {
              "bool" : {
                "filter" : [
                  {
                    "range" : {
                      "@timestamp" : {
                        "from" : "{{period_end}}||-1h",
                        "to" : "{{period_end}}",
                        "include_lower" : true,
                        "include_upper" : true,
                        "format" : "epoch_millis",
                        "boost" : 1.0
                      }
                    }
                  },
                  {
                    "terms" : {
                      "rule.id" : [
                        "5763",
                        "5712"
                      ],
                      "boost" : 1.0
                    }
                  }
                ],
                "adjust_pure_negative" : true,
                "boost" : 1.0
              }
            },
            "aggregations" : { }
          }
        }
      }
    ],
    "triggers" : [
      {
        "query_level_trigger" : {
          "id" : "Mel7D4UBXvIZ4pdFqgDQ",
          "name" : "Test Alert trigger",
          "severity" : "2",
          "condition" : {
            "script" : {
              "source" : "ctx.results[0].hits.total.value > 0",
              "lang" : "painless"
            }
          },
          "actions" : [
            {
              "id" : "Mul7D4UBXvIZ4pdFqgDQ1",
              "name" : "dgvzdf8zx2toawthltg_channel_webhook",
              "destination_id" : "dgvzdf8zx2toawthltg_channel_webhook",
              "message_template" : {
                "source" : """
Trigger name: {{ctx.trigger.name}} 
""",
                "lang" : "mustache"
              },
              "throttle_enabled" : true,
              "subject_template" : {
                "source" : "",
                "lang" : "mustache"
              },
              "throttle" : {
                "value" : 25,
                "unit" : "MINUTES"
              }
            }
          ]
        }
      }
    ],
    "last_update_time" : 1672381433218
  }
}

@lezzago i have added rbac_role while creating but its not reflecting in it.
and also tried updating the monitor. same result its not reflecting added/updated rbac_roles.

User and roles mapping

GET _plugins/_security/api/rolesmapping/dgvzdf8zx2toawthltg

{
  "dgvzdf8zx2toawthltg" : {
    "hosts" : [ ],
    "users" : [
      "[email protected]"
    ],
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [ ],
    "and_backend_roles" : [ ]
  }
}

GET _plugins/_security/api/internalusers/[email protected]

{
  "[email protected]" : {
    "hash" : "",
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "dgvzdf8zx2toawthltg"
    ],
    "attributes" : { },
    "opendistro_security_roles" : [ ],
    "static" : false
  }
}

[lezzago]

@techrna, sorry, I forgot to share the correct method to query .opendistro-alerting-config. It is a system index, so to query it, follow this guide.
The command to query it should look like curl -k --cert ./kirk.pem --key ./kirk-key.pem -XGET 'https://localhost:9200/.opendistro-alerting-config/_doc/HbWuYYUBb9hzdJ2RpaJ5'

Also the reason you do not see the rbac roles associated to the monitor is prevent leaking security credential information. If you run the command above, you should get user information tied to the monitor and that will include the rbac_roles. So when you get that, feel free to also obfuscate any important security credentials.

Also can you share the output of GET _plugins/_security/api/internalusers/[email protected] since [email protected] made the initial call?

[techrna]

@lezzago [email protected] its new user created with same settings
GET _plugins/_security/api/internalusers/[email protected]

{
  "[email protected]" : {
    "hash" : "",
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "dgvzdf8zx2toawthltg"
    ],
    "attributes" : { },
    "opendistro_security_roles" : [ ],
    "static" : false
  }
}

GET _plugins/_security/api/internalusers/[email protected]

{
  "[email protected]" : {
    "hash" : "",
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "dgvzdf8xx2toawthlte"
    ],
    "attributes" : { },
    "opendistro_security_roles" : [ ],
    "static" : false
  }
}

curl -k --cert ./kirk.pem --key ./kirk-key.pem -XGET 'https://localhost:9200/.opendistro-alerting-config/_doc/HbWuYYUBb9hzdJ2RpaJ5

{
  "_index": ".opendistro-alerting-config",
  "_id": "HbWuYYUBb9hzdJ2RpaJ5",
  "_version": 3,
  "_seq_no": 847,
  "_primary_term": 35,
  "found": true,
  "_source": {
    "monitor": {
      "type": "monitor",
      "schema_version": 5,
      "name": "linux_multiple_logon_failures [workstation]",
      "monitor_type": "query_level_monitor",
      "user": {
        "name": "[email protected]",
        "backend_roles": [
          "dgvzdf8zx2toawthltg"
        ],
        "roles": [
          "dgvzdf8zx2toawthltg",
          "own_index"
        ],
        "custom_attribute_names": [
          
        ],
        "user_requested_tenant": null
      },
      "enabled": true,
      "enabled_time": 1672230731677,
      "schedule": {
        "period": {
          "interval": 1,
          "unit": "MINUTES"
        }
      },
      "inputs": [
        {
          "search": {
            "indices": [
              "rushab-dgvzdf8zx2toawthltg-workstation-archives-*"
            ],
            "query": {
              "size": 100,
              "query": {
                "bool": {
                  "filter": [
                    {
                      "range": {
                        "@timestamp": {
                          "from": "{{period_end}}||-1h",
                          "to": "{{period_end}}",
                          "include_lower": true,
                          "include_upper": true,
                          "format": "epoch_millis",
                          "boost": 1.0
                        }
                      }
                    },
                    {
                      "terms": {
                        "rule.id": [
                          "5763",
                          "5712"
                        ],
                        "boost": 1.0
                      }
                    }
                  ],
                  "adjust_pure_negative": true,
                  "boost": 1.0
                }
              },
              "aggregations": {
                
              }
            }
          }
        }
      ],
      "triggers": [
        {
          "query_level_trigger": {
            "id": "Mel7D4UBXvIZ4pdFqgDQ",
            "name": "Test Alert trigger",
            "severity": "2",
            "condition": {
              "script": {
                "source": "ctx.results[0].hits.total.value > 0",
                "lang": "painless"
              }
            },
            "actions": [
              {
                "id": "Mul7D4UBXvIZ4pdFqgDQ1",
                "name": "dgvzdf8zx2toawthltg_channel_webhook",
                "destination_id": "dgvzdf8zx2toawthltg_channel_webhook",
                "message_template": {
                  "source": ".................",
                  "lang": "mustache"
                },
                "throttle_enabled": false,
                "subject_template": {
                  "source": "",
                  "lang": "mustache"
                }
              }
            ]
          }
        }
      ],
      "last_update_time": 1672728054262,
      "ui_metadata": {
        "schedule": {
          "cronExpression": "0 */1 * * *",
          "period": {
            "unit": "MINUTES",
            "interval": 1
          },
          "timezone": null,
          "daily": 0,
          "monthly": {
            "type": "day",
            "day": 1
          },
          "weekly": {
            "tue": false,
            "wed": false,
            "thur": false,
            "sat": false,
            "fri": false,
            "mon": false,
            "sun": false
          },
          "frequency": "interval"
        },
        "search": {
          "searchType": "query",
          "bucketValue": 1,
          "timeField": "",
          "bucketUnitOfTime": "h",
          "where": {
            "fieldName": [
              
            ],
            "fieldRangeEnd": 0,
            "fieldRangeStart": 0,
            "fieldValue": "",
            "operator": "is"
          },
          "groupBy": [
            
          ],
          "aggregations": [
            
          ]
        },
        "triggers": {
          "Brute Force": {
            "value": 10000,
            "enum": "ABOVE"
          }
        },
        "monitor_type": "query_level_monitor"
      }
    }
  }
}

[lezzago]

Can you send the test message as the [email protected] user? The monitor has the same backend roles as [email protected] and not [email protected], so that can be part of the issue.

[goelvarun1]

Hi @techrna @lezzago

Are we able to solve this issue as I am also facing the same problem, and this seems a big blocker in my project delivery as if I am disabling the configuration of filtering_by_backend_role its working but than my all tenants was able to see each other's data like their own created monitors and if I am enabling, I am not able to trigger any notification

Please help

[lezzago]

Looks like there is an issue in passing in the backend roles to the notification plugin when sending the notification.
The current workaround should be:

PUT _cluster/settings
{
  "persistent": {
    "opensearch.notifications.general.filter_by_backend_roles": "false"
  }
}

This will only disable RBAC for notification and still allow you to have it enabled on Alerting resources.

[goelvarun1]

Thanks, @lezzago

Yes, I was able to send the notification by using the cluster setting as mentioned by you, but this change introduced a big security flaw for me as my product tenants segregated based on their roles was able to see everybody else notification channels

I am able to fix the same by changing the code inside notification-plugin repo,
I commented the method which was throwing this role error inside kotlin file and built the new jar out of it, that's the workaround I did so that with the role-based isolation, notification will also work, but will upgrade with the release once you will give the fix of passing those backend roles to notification plugin.

Please do update here once you fix the issue and give the release with that fix

[lezzago]

Closing this issue as the PR has been merged and will be part of the 2.7 release