[BUG] Scroll query is broken for multi node clusters for LDAP users

[DarshitChanpura]

What is the issue?

Security plugin supports different types of user like InjectedUser and LDAPUser for various Authentication mechanism, where those various type of users are serialized and passed to different nodes after AuthN/AuthZ on co-ordinating node to avoid re-AuthN/AuthZ.

With this PR #2765 which was added to improve performance by avoiding serialization on same node.

But a corner case got missed i.e. scroll queries where user who create scroll query can only access it and it's done by security plugin by creating reader context on scroll query and store it locally which also stores user information code: https://github.com/opensearch-project/security/blob/2.9/src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java#L653,L679.

Now let consider scenario where we've two node cluster let's say Node-1 and Node-2. When create scroll request lend on Node-1 it will have readerContext with Injected/LDAP user with respective class type and Node-2 will have readerContext with user with User.class type. Now if scroll get request lend on Node-1 then it will work as both current user and readerContext user are of same type. But if request lend on Node-2 it will serialize current user and send to Node-1 now current user will of type User.class but readerContext will have user with class Injected/LDAP user which will fail as both users have different class types.

How can one reproduce the bug?

A simple scroll request on a multi node cluster should reproduce the issue.

# create scroll

GET movies/_search?scroll=10m
{
  "size": 10
}

# get response

GET _search/scroll
{
  "scroll": "10m",
  "scroll_id": "<scroll_id_from above_step>"
}

What is the expected behavior?

Scroll query should work on multi node cluster for different types of users.

What is your host/environment?

OS v2.9
Security Plugin

Do you have any additional context?

code where security plugin makes user match for scroll query: https://github.com/opensearch-project/security/blob/2.9/src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java#L692.

Exit criteria:

• Reproduce the bug
• Implement and verify the fix
• Add integration test for this scenario

[cwperks]

[Triage] Please fill this issue out with more description of the issue.

[DarshitChanpura]

[Triage] Please fill this issue out with more description of the issue.

Updated the description.

[cwperks]

I cannot reproduce this issue. When running a multi-node cluster and creating a scroll, it will create the readerContext with an instance of LdapUser regardless of the node that creates the context. (The node that creates the context will be a node containing a shard for the index). The receiving node serializes the authenticated LdapUser uses the transport layer to forward the request to create a scroll to a node with a shard where its deserialized as LdapUser and stored.

When a user uses the scroll id to then fetch more results, they authenticate as an LdapUser and it matches what is stored in the reader context.

Can you provide more information about how to reproduce this issue? There could be edge cases like an internal user and LDAP user with the same username or a user may try to user impersonation to impersonate another user in the system.

[cwperks]

I reproduced the issue using user impersonation. For my LDAP configuration I use the following ldif file

directory.ldif

# --- OUs -------------------------------------

dn: ou=Groups,dc=example,dc=org
objectClass: organizationalunit
objectClass: top
ou: Groups

dn: ou=People,dc=example,dc=org
objectClass: organizationalunit
objectClass: top
ou: People

--- People ----------------------------------

dn: cn=jsmith,ou=People,dc=example,dc=org
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: top
cn: psmith
userpassword: password
givenname: John
sn: Smith
mail: [email protected]
uid: 1003

--- Groups ----------------------------------

dn: cn=Administrator,ou=Groups,dc=example,dc=org
objectClass: groupofuniquenames
objectClass: top
ou: Groups
cn: Administrator
uniquemember: cn=jsmith,ou=People,dc=example,dc=org

and in opensearch.yml

plugins.security.authcz.rest_impersonation_user:
  admin:
    - "*"

Create the scroll query with an LDAP user and the reader context stores an instance of LDAP user.

Then use user impersonation to impersonate the LDAP user:

curl -XGET https://admin:admin@localhost:9200/_search/scroll -k -H "opendistro_security_impersonate_as: jsmith" -H "Content-Type: application/json" --data '
{
  "scroll": "10m",
  "scroll_id": "<scroll_id>"
}'

When impersonating jsmith its looking for User.class instead of LdapUser.class and produces the following error:

opensearch-ldap-node3  | [2023-12-06T19:22:13,545][ERROR][o.o.s.s.GuardedSearchOperationWrapper] [opensearch-ldap-node3] Wrong user User [name=jsmith, backend_roles=[Administrator], requestedTenant=null] in reader context, expected User [name=jsmith, backend_roles=[Administrator], requestedTenant=null]

[cwperks]

Closing this issue. #3805 has been merged and backported.