"Could not load roles" when authenticating against Authentik with read-only Active Directory

[kenodai]

Hi Team,
not sure if this is related to my configuration, the chart or ocis in general.

I'm running a Samba-based DC for multiple years. In order to get ocis running on k8s and authenticate against the DC, I've deployed an Authentik instance.

Authentik is running on a k8s, trying to run ocis in minikube.

ocis-values.yaml:

externalDomain: ocis.owncloud.test
insecure:
  oidcIdpInsecure: true
  ocisHttpApiInsecure: true

logging:
  level: "debug"
  pretty: "true"
  color: "true"

cache:
  type: "redis-sentinel"
  nodes:
    - redis.ocis-redis.svc.cluster.local:26379/mymaster
store:
  type: "redis-sentinel"
  nodes:
    - redis.ocis-redis.svc.cluster.local:26379/mymaster

services:
  nats:
    persistence:
      enabled: true
  store:
    persistence:
      enabled: true
  search:
    persistence:
      enabled: true
  storagesystem:
    persistence:
      enabled: true
  storageusers:
    persistence:
      enabled: true
  web:
    persistence:
      enabled: true


features:
  externalUserManagement:
    enabled: true
    oidc:
      issuerURI: https://sso.*/application/o/ocis-web/
      webClientID: *
      userIDClaim: username
      userIDClaimAttributeMapping: username
      roleAssignment:
        enabled: true
        claim: roles
        mapping:
          - role_name: admin
            claim_value: ocisAdmin
          - role_name: spaceadmin
            claim_value: ocisSpaceAdmin
          - role_name: user
            claim_value: ocisUser
          - role_name: guest
            claim_value: ocisGuest
    ldap:
      writeable: false
      uri: ldaps://*/
      certTrusted: true
      insecure: true
      bindDN: "CN=tu_ldap,ou=ressource,dc=*"
      useServerUUID: false
      refintEnabled: true
      user:
        objectClass: user
        baseDN: "OU=Accounts,DC=*"
        schema:
          #id: "objectGUID"
          #idIsOctetString: true
          id: "sAMAccountName"
          userName: sAMAccountName
      group:
        objectClass: group
        baseDN: "OU=Groups,DC=*"
        schema:
          #id: "objectGUID"
          #idIsOctetString: true
          id: "cn"
          groupName: cn

ingress:
  enabled: true
  #ingressClassName: private-nginx
  ingressClassName: nginx
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 1024m
  tls:
    - secretName: ocis-tls
      hosts:
        - ocis.owncloud.test

secretRefs:
  ldapSecretRef: "ldap-secret"

For some reason using objectGUID wasn't working. Switched to something easier for now.

2023-09-15T07:43:41Z DBG director found line=github.com/owncloud/ocis/v2/services/proxy/pkg/router/router.go:222 method=POST path=/api/v0/settings/values-list policy=ocis prefix=/api/v0/settings routeType=prefix service=proxy
2023-09-15T07:43:41Z DBG parsed access token access token={"aud":["wraOalEg*o3dQjhLv"],"exp":1694764118,"iat":1694763818,"iss":"https://sso.*/application/o/ocis-web/","sid":"","sub":"32"} line=github.com/owncloud/ocis/v2/ocis-pkg/oidc/client.go:305 service=proxy
2023-09-15T07:43:41Z DBG Expiration Time from access_token exp="2023-09-15 07:48:38 +0000 UTC" line=github.com/owncloud/ocis/v2/services/proxy/pkg/middleware/oidc_auth.go:133 service=proxy
2023-09-15T07:43:41Z DBG extracted claims claims={"acr":"goauthentik.io/providers/oauth2/default","aud":"wraOalEg*o3dQjhLv","auth_time":1694688163,"azp":"wraOalEgQaAKVmM*27Lo3dQjhLv","email":"<redacted>","email_verified":true,"exp":1694764118,"given_name":"M*","iat":1694763818,"iss":"https://sso.*/application/o/ocis-web/","name":"M*","roles":["ocisAdmin"],"sub":"32","uid":"DN32TuK*gsMwOoh1fL","username":"marc"} line=github.com/owncloud/ocis/v2/services/proxy/pkg/middleware/oidc_auth.go:123 service=proxy
2023-09-15T07:43:41Z DBG successfully authenticated request authenticator=oidc line=github.com/owncloud/ocis/v2/services/proxy/pkg/middleware/oidc_auth.go:171 path=/api/v0/settings/values-list service=proxy
[...]
2023-09-15T07:43:41Z DBG GetUserByClaim claim=username line=github.com/cs3org/reva/[email protected]/pkg/user/manager/ldap/ldap.go:137 pkg=rgrpc service=users traceid=00000000000000000000000000000000 value=marc
2023-09-15T07:43:41Z DBG LDAP Search backend=ldap basedn=OU=Accounts,DC=* filter=(&(objectclass=user)(sAMAccountName=marc)) line=github.com/cs3org/reva/[email protected]/pkg/utils/ldap/identity.go:217 pkg=rgrpc scope=2 service=users traceid=00000000000000000000000000000000
2023-09-15T07:43:41Z DBG entries entry={"Attributes":[{"ByteValues":["*"],"Name":"displayName","Values":["M*"]},{"ByteValues":["*=="],"Name":"sAMAccountName","Values":["marc"]},{"ByteValues":["*="],"Name":"mail","Values":["*"]},{"ByteValues":["*="],"Name":"uidNumber","Values":["123"]},{"ByteValues":["*="],"Name":"gidNumber","Values":["123"]}],"DN":"CN=Marc*,OU=Accounts,DC=*"} line=github.com/cs3org/reva/[email protected]/pkg/user/manager/ldap/ldap.go:144 pkg=rgrpc service=users traceid=00000000000000000000000000000000
2023-09-15T07:43:41Z DBG LDAP Search backend=ldap basedn=OU=Groups,DC=* filter="(&(objectclass=group)(member=CN=Marc*,OU=Accounts,DC=*))" line=github.com/cs3org/reva/[email protected]/pkg/utils/ldap/identity.go:349 pkg=rgrpc scope=2 service=users traceid=00000000000000000000000000000000
2023-09-15T07:43:41Z DBG unary code=OK end="15/Sep/2023:07:43:41 +0000" from=tcp://10.244.0.14:54884 line=github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/log/log.go:69 pkg=rgrpc service=gateway start="15/Sep/2023:07:43:41 +0000" time_ns=66366705 traceid=00000000000000000000000000000000 uri=/cs3.gateway.v1beta1.GatewayAPI/GetUserByClaim user-agent=grpc-go/1.57.0
[...]
2023-09-15T07:43:41Z DBG got roles in claim line=github.com/owncloud/ocis/v2/services/proxy/pkg/userroles/oidcroles.go:49 request-id=9ca280a0-c884-4b4e-9b5b-d554edf5ba63 rolesClaim=roles rolesInClaim=["ocisAdmin"] service=proxy userid=
2023-09-15T07:43:41Z DBG first matching role line=github.com/owncloud/ocis/v2/services/proxy/pkg/userroles/oidcroles.go:74 ocisRole=admin request-id=9ca280a0-c884-4b4e-9b5b-d554edf5ba63 role id=71881883-1768-46bd-a24d-a356a2afdf7f service=proxy userid=
2023-09-15T07:43:41Z ERR Could not load roles error="{\"id\":\"ocis-settings\",\"code\":400,\"detail\":\"cannot be blank\",\"status\":\"Bad Request\"}" line=github.com/owncloud/ocis/v2/services/proxy/pkg/userroles/oidcroles.go:88 request-id=9ca280a0-c884-4b4e-9b5b-d554edf5ba63 service=proxy userid=
2023-09-15T07:43:41Z ERR Could not get user roles error="{\"id\":\"ocis-settings\",\"code\":400,\"detail\":\"cannot be blank\",\"status\":\"Bad Request\"}" line=github.com/owncloud/ocis/v2/services/proxy/pkg/middleware/account_resolver.go:154 service=proxy

Authentication seems to work fine. LDAP Search seems to be successful as well. Role is getting extracted from the response, but then... Could not load roles

No upgrade or something like that happend. Tried to run from scratch already, completely destroying minikube installation.

I do find it a bit strange the userid is not filled on the last few log entries.

Regards,
Marc

[wkloucek]

@kenodai Could you please provide us the commit id / tag of the oCIS Chart you're using? I assume you're using something recent (master / 0.5.0) with oCIS 4.0.1 image, right?

@rhafer do you have an idea why the userid could "disappear" between the OIDC auth middleware and the OIDC roles step?

[rhafer]

@kenodai Could you please provide us the commit id / tag of the oCIS Chart you're using? I assume you're using something recent (master / 0.5.0) with oCIS 4.0.1 image, right?

Yes. I'd like to know those details as well.

@rhafer do you have an idea why the userid could "disappear" between the OIDC auth middleware and the OIDC roles step?

Not, really. But it looks a bit as if the GetUserByClaim call does not return a userid value at all in this setup. I think we need to improve the logging in that case. (Might also be some bug in the charts, though I haven't found anything on a quick glance)

For some reason using objectGUID wasn't working. Switched to something easier for now.

Note, that even if you manage to get by this specific problem, I think that with using samAccountName as the userid you'll get into different problems later on. IIRC the decomposedfs storage provider has a minimum requirement on the length of the userids. (Users with short usernames will likely run into issues)
What exactly was not working with objectGUID? Recent ocis releases should actually work fine with it.

[kenodai]

Commit ID is 1a00bb6
oCIS Image is 4.0.1

2023-09-18T12:43:13Z ERR Could not get user by claim error="could not get user by claim username with value marc : error authenticating user " line=github.com/owncloud/ocis/v2/services/proxy/pkg/middleware/account_resolver.go:146 service=proxy

Might be related to my configuration. Same as above, just using the commented parameters. As sAMAccountName gave me some easy progress, I ignored the GUID for now.

[rhafer]

I think I am able to reproduce the issue. It seems to be a bug in the users service.

As a workaround you could apply this patch to the helm chart for now, until have the fix in ocis

--- a/charts/ocis/templates/users/deployment.yaml
+++ b/charts/ocis/templates/users/deployment.yaml
@@ -77,7 +77,7 @@ spec:
             - name: USERS_LDAP_GROUP_OBJECTCLASS
               value: {{ .Values.features.externalUserManagement.ldap.group.objectClass | quote }}
 
-            - name: USERS_LDAP_USER_SCHEMA_ID
+            - name: OCIS_LDAP_USER_SCHEMA_ID
               value: {{ .Values.features.externalUserManagement.ldap.user.schema.id | quote }}
             - name: USERS_LDAP_GROUP_SCHEMA_ID
               value: {{ .Values.features.externalUserManagement.ldap.group.schema.id | quote }}

BTW, with the above workaround I also successfully tried:

     user:
        objectClass: user
        baseDN: "CN=Users,DC=owncloud,DC=test"
        schema:
          id: "objectGUID"
          idIsOctetString: true
          userName: sAMAccountName

[butonic]

running into this withk AD FS. Can confirm that changing USERS_LDAP_USER_SCHEMA_ID to OCIS_LDAP_USER_SCHEMA_ID allows building a user struct from the ldap response. But now I see

{
    "level": "debug",
    "service": "proxy",
    "claims": {
        "appid": "https://cloud.example.com/",
        "apptype": "Public",
        "aud": "microsoft:identityserver:a-u-u-i-d",
        "auth_time": "2023-09-21T09:51:48.619Z",
        "authmethod": "http://schemas.microsoft.com/ws/2008/06/identity/authenticationmethod/windows",
        "displayname": "John, Doe",
        "exp": 1695298629,
        "iat": 1695295029,
        "iss": "http://cloud.example.com/adfs/services/trust",
        "mail": "[email protected]",
        "nbf": 1695295029,
        "samaccountname": "JohnDoe",
        "scp": "email profile openid",
        "ver": "1.0"
    },
    "time": "2023-09-21T11:17:13.1156504Z",
    "line": "github.com/owncloud/ocis/v2/services/proxy/pkg/middleware/oidc_auth.go:123",
    "message": "extracted claims"
}
{
    "level": "debug",
    "service": "proxy",
    "authenticator": "oidc",
    "path": "/api/v0/settings/values-list",
    "time": "2023-09-21T11:17:13.1156602Z",
    "line": "github.com/owncloud/ocis/v2/services/proxy/pkg/middleware/oidc_auth.go:171",
    "message": "successfully authenticated request"
}
{
    "level": "error",
    "service": "proxy",
    "error": "{\"id\":\"go.micro.server\",\"code\":500,\"detail\":\"panic recovered: runtime error: invalid memory address or nil pointer dereference\",\"status\":\"Internal Server Error\"}",
    "time": "2023-09-21T11:17:13.2519563Z",
    "line": "github.com/owncloud/ocis/v2/services/proxy/pkg/userroles/defaultrole.go:38",
    "message": "Could not load roles"
}
{
    "level": "error",
    "service": "proxy",
    "error": "{\"id\":\"go.micro.server\",\"code\":500,\"detail\":\"panic recovered: runtime error: invalid memory address or nil pointer dereference\",\"status\":\"Internal Server Error\"}",
    "time": "2023-09-21T11:17:13.2519742Z",
    "line": "github.com/owncloud/ocis/v2/services/proxy/pkg/middleware/account_resolver.go:154",
    "message": "Could not get user roles"
}
{
    "level": "info",
    "service": "proxy",
    "proto": "HTTP/1.1",
    "request-id": "f9462791-4030-4fc1-99e5-f5fcbdfcb724",
    "remote-addr": "10.68.204.59",
    "method": "POST",
    "status": 500,
    "path": "/api/v0/settings/values-list",
    "duration": 137.4538,
    "bytes": 0,
    "time": "2023-09-21T11:17:13.2519951Z",
    "line": "github.com/owncloud/ocis/v2/services/proxy/pkg/middleware/accesslog.go:31",
    "message": "access-log"
}

What is annoying is the the users service never logs an error that it cannot map the id (because it is empty or malformed) ... it bubbles up the error but it is never logged ... and as a result at some point it is just swallowed leading to a 500 without any insights. Hard to debug.

Another thing that is hard to debug is that go micro seems to swallow the stacktrace and just returns a panic recovered error. I assume because either the id is still empty or the settings service is somehow misconfigured... will dig into that.

[rhafer]

@butonic Do you have the idIsOctetString: true setting in you vars file?

And yes. The error/debug logging has room for improvement.

[butonic]

yes ... and login finally worked ... was fixed by restarting the services. seems to have been a connection issue.

[kenodai]

I can confirm the above patch is working. Also the objectGUID part is working now.

Thanks for the solution, looking forward playing with oCIS.