[BUG] Monitors notifications channel not working! with RBAC setting

[techrna]

Bug
While Creating Monitors in that Triggers actions notification channel is not working. I have created the notification channel as custom webhook. When we hit send test message over there it works .But when do send test message while adding monitor it fails .

How can one reproduce the bug?
Steps to reproduce the behavior:

1. enable backend role filtering
   PUT _cluster/settings
   {
   "transient": {
   "plugins.alerting.filter_by_backend_roles": "true"
   }
   }
2. Then Create monitor
3. Add the trigger with action as notification channel as webhook
4. Try to send the test message will throw an error.and also check the opensearch.log file
   Error screenshots and logs are attach below please check.

Problem is it fails to read the backend roles for the user.when filter_by_backend_roles is enabled

host/environment

• OS: CentOS 7.9
• Version 7.9
• Plugins Alert Plugin
• Opensearch 2.3

screenshots?

OpenSearch logs
[2022-12-09T18:59:50,893][INFO ][o.o.n.i.ConfigIndexingActions] [test2-server] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@43c2eec6
[2022-12-09T18:59:50,893][WARN ][o.o.n.a.PluginBaseAction ] [test2-server] notifications:OpenSearchStatusException:
org.opensearch.OpenSearchStatusException: User doesn't have backend roles configured. Contact administrator.
at org.opensearch.notifications.security.UserAccessManager.validateUser(UserAccessManager.kt:24) ~[opensearch-notifications-2.3.0.0.jar:2.3.0.0]
at org.opensearch.notifications.index.ConfigIndexingActions.get(ConfigIndexingActions.kt:251) ~[opensearch-notifications-2.3.0.0.jar:2.3.0.0]
at org.opensearch.notifications.action.GetNotificationConfigAction.executeRequest(GetNotificationConfigAction.kt:60) ~[opensearch-notifications-2.3.0.0.jar:2.3.0.0]
at org.opensearch.notifications.action.GetNotificationConfigAction.executeRequest(GetNotificationConfigAction.kt:26) ~[opensearch-notifications-2.3.0.0.jar:2.3.0.0]
at org.opensearch.notifications.action.PluginBaseAction$doExecute$1.invokeSuspend(PluginBaseAction.kt:63) [opensearch-notifications-2.3.0.0.jar:2.3.0.0]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
[2022-12-09T18:59:50,894][INFO ][o.o.a.m.MonitorRunResult ] [test2-server] Internal error: Unable to find a Notification Channel or Destination config with id [nsgT94QBikbo0yuk80z2]. See the opensearch.log for details
java.lang.IllegalStateException: Unable to find a Notification Channel or Destination config with id [nsgT94QBikbo0yuk80z2]
at org.opensearch.alerting.MonitorRunner.getConfigAndSendNotification(MonitorRunner.kt:93) ~[opensearch-alerting-2.3.0.0.jar:2.3.0.0]
at org.opensearch.alerting.MonitorRunner$getConfigAndSendNotification$1.invokeSuspend(MonitorRunner.kt) ~[opensearch-alerting-2.3.0.0.jar:2.3.0.0]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]

[lezzago]

Closing this issue in favor of this one: opensearch-project/alerting#698