[BUG]Can't add permission for Notification. Permissions [cluster:admin/opensearch/notifications/*] don't exist

[qmonitoring]

What is the bug?
It is impossible to add any cluster:admin/opensearch/notifications permission, thought it's a necessary condition for users to have access to the Channels

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to 'Home > Notifications > Channels' under any non-admin user (without full access privileges).
2. You see nothing at this page.
3. Errors in browser console are "error": "Forbidden", "message": "[security_exception] no permissions for [cluster:admin/opensearch/notifications/features] and User [name=...
"error": "Forbidden", "message": "[security_exception] no permissions for [cluster:admin/opensearch/notifications/configs/get] and User [name=...
4. Go to 'Home > Security > Roles > Create Role' under admin user and try to add permissions from the errors to any role
5. Impossible to add permission - the permission doesn't match any options

What is the expected behavior?
Worked procedure for permission providing.

What is your host/environment?

• OS: Debian 11 bullseye
• Version: opensearchproject/opensearch:2.1.0
• Plugins:
  opensearch-alerting 2.1.0.0
  opensearch-notifications 2.1.0.0
  opensearch-security 2.1.0.0

Additional info
OpenSearch security config

plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - "*******"
plugins.security.nodes_dn:
  - "********"
  - "********"
  - "********"
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

Authentication and authorization: LDAP enabled

[qmonitoring]

Hello, Team! Is there any update on this issue? This bug is very critical for us, since we can't provide appropriate permission for end customers of our OpenSearch cluster.

[qreshi]

Hi @qmonitoring,

It looks like these options are populated in part by the CLUSTER_PERMISSIONS constant in the Security OpenSearch Dashboards plugin where the Notifications cluster permissions are missing. We'll need to add them there so they can be selected when creating a Role.

[qmonitoring]

@qreshi, thank you for your reply! This is most likely the reason.
I have workarounded the issue by manual adding of missing permissions using API
PUT _plugins/_security/api/roles/alerting_opensearch

[glewis-vectra]

Hello @qreshi , I am experiencing an issue that pertains to this bug and would like to ask if has been prioritized.

[LucDesaulniers]

Hello @qreshi. We are also running into the exact same situation. Any idea if this will get addressed in the near future?

[qreshi]

Hey @glewis-vectra and @LucDesaulniers. It appears the change still hasn't been done on the Security Dashboards side. I can't say for sure if this will be backported (since that is up to the Security team) but I'll tag this issue for 2.5 to ensure it is prioritized to be part of the next upcoming release.

[LucDesaulniers]

Thank you @qreshi. At least now we know it will get looked at.

[qreshi]

The change has been merged in and should be available in 2.5.

[guillaumeldc]

@qreshi I just installed 2.5 and still getting the message "Notifications plugin is not installed" when a non-admin user is tryin to update a monitor within the action section. Please see attached. The permissions have been updated accordingly using the latest addition but no improvement. Am I missing something?
The plugin is installed and an admin user can easily update monitor without any issues.
Also the non-admin user is able to update and create new channels from the notifications plugin which leads me to believe that it might be an issue with the alerting plugin...

[LucDesaulniers]

Same here. Took a look at the 2.5 release notes and it seems that the fix was not included in this release. Unless I'm wrong of course.

[qreshi]

@guillaumeldc Just to clarify, you're not seeing this "Notifications plugin is not installed" banner as an admin user?

[qreshi]

@LucDesaulniers The discrepancy that was originally called out in this issue was missing cluster permissions in the dropdown on the OpenSearch Dashboards side. That fix was included in 2.5 of Security Dashboards.

That "Notifications plugin is not installed" banner is based on a conditional check for the backend plugin being present and shouldn't be related but could be another issue.

[guillaumeldc]

@qreshi no I'm not seeing the banner as an admin user. I've investigated a bit the issue and found out that the following two permissions must also be added to the user's permissions:

• cluster:monitor/state
• cluster:monitor/nodes/info

the error we were getting was stating: There was a problem getting plugins list with resp "[security_exception] no permissions for [cluster:monitor/state] and User [name=test_user, backend_roles=[managers], requestedTenant=my_tenant]"

this might not a notifications plugin bug in the end and more related to the alerting plugin? Definitely a separate issue.

[qreshi]

@guillaumeldc Thanks for looking into it. Yeah, this makes sense since that permission seems to be required to make the "list plugins API call" that's being used to determine whether Notifications is installed.

It seems the alerting_full_access role grants all cluster:monitor permissions. However, a non-admin user shouldn't need to manually add this because of a check that Alerting Dashboards does.

Would you mind creating an issue for this in the https://github.com/opensearch-project/alerting-dashboards-plugin repo? We can continue discussions there and possibly loop in folks from the Security plugin to understand the correct flow for Alerting Dashboards, possibly by making this an elevated call in the background outside of the user context, since it's a validation being done by the plugin.

[guillaumeldc]

@qreshi will do! thanks for your help.

[LucDesaulniers]

@LucDesaulniers The discrepancy that was originally called out in this issue was missing cluster permissions in the dropdown on the OpenSearch Dashboards side. That fix was included in 2.5 of Security Dashboards.

That "Notifications plugin is not installed" banner is based on a conditional check for the backend plugin being present and shouldn't be related but could be another issue.

Thanks @qreshi. Will go ahead with 2.5 then!