Circles, contact groups and user groups

[MorrisJobke]

Problem

We have so many grouping mechanisms and they are confusing for people that don't know the technical background of them. We should unify the way those are handled into a single way of grouping users.

• circles
• contact groups
• (system) user groups
• federated groups
• ...

Idea

Idea is to have some grouping entity that is the only way of organizing a group of users and contacts.

Additionally a layer is added to the server to share to some grouping entity only and not to groups, users or federated users anymore, because those are 3 concepts that basically do the same: share something to 1...n users.

• each user is at least in one of those groupings (the grouping with only the user itself) <- that is used to replace user shares

Attributes

• this grouping entity has a name (random one by default)
• this grouping entity has a type (single user, group of users)? - can be build with permissions (no adding/removing permission for anybody for the single user type)
• this grouping entity has a permission who can manage it (joining, adding, removing users, change name)
• this grouping entity has a visibility

Steps

• discuss the idea (right here in the ticket)
• define the scope and specs
• implement this grouping mechanism (which exists in parallel to the existing ones and undisclosed to any UI)
• implement sharing part
• migrate user shares, groups, circles, contact groups
• remove existing shares, rooms, circles, ...

Note: the last two steps need to be done in one release. The other ones can be done in separate major releases to be able to move this forward and not have one huge changeset that nobody can review properly.

Problems

• the UI needs to make it easy what this grouping entity is about (admins, permissions, visibility)
• the UI needs to make it obvious where this grouping entity comes from to avoid phishing attacks by creating a similar looking entity via federated groups for example
• for migration: what about existing collisions? if there is a circle, a group and 3 contact groups named the same? - during migration, we can add something at the end of the name
• how are the groups managed? User mgmt and Contacts? And additionally inherent grouping using "Linking Collaboration Things"? (Ideally the name "Circles" does not show up – it’s just "users" and "groups")

Other approaches

Linking collaboration things™ - #11015

• links a random entity (room, file, board, ...) to another one and each user that has access to any of them then they also have access to the linked ones
• it's still different, because the goal is not to group users but to group data objects

Design forward: How we handle Users / Groups / Circles - #4493

• this one is another approach to not evolve but rethink the whole concept to not be limited by the existing structures

cc @skjnldsv @juliushaertl @rullzer @jancborchardt @ChristophWurst @danielkesselberg @blizzz @schiessle @nickvergessen @daita for feedback on this topic.

[jancborchardt]

Does it make most sense to talk about it more at the Contributor Week in person? Or is that too late, and we should rather have another (or two) calls about this?

I ask mainly because of:

Linking collaboration things™ - #11015
it's still different, because the goal is not to group users but to group data objects

Technically that’s what it does, but UX-wise it’s really similar. That’s why I think we need to coordinate this here and that one, so we don’t end up in 2 different concepts again. :)

Design forward: How we handle Users / Groups / Circles - #4493

This includes a whole lot of other things (federation of contacts, own profile etc), so this here is one part of it. Probably that issue should be renamed. :)

[juliusknorr]

Technically that’s what it does, but UX-wise it’s really similar. That’s why I think we need to coordinate this here and that one, so we don’t end up in 2 different concepts again. :)

Well, the approach this PR describes is more a generalization of our model of sharing entities (file/calendar) to a user/group/userdefined-group, while #11015 is about combining related entities. Yes, the linking will have the effect of allowing users to access those entities as well, but the main concept is about grouping related elements.

Does it make most sense to talk about it more at the Contributor Week in person? Or is that too late, and we should rather have another (or two) calls about this?

This issue is something we won't be able to tackle for 16 I think, regarding #11015 we already discussed that at the last hackweek and the concept is different from sharing, so I see no collision there.

[ArtificialOwl]

Finally, I found a way to do something nice: IEntitiesManager !

While this is a group management solution, we will take the file sharing as an example for a better overview.
When you share a file, you share to an entity, could it be:

• a single local user,
• a single mail address,
• a group of multiple local users and/or mail address.

(we can imagine a list of other types like federated cloud, federated users, ...)

Now, let's talk about the databases
Please note that this is the general structure, some fields might pop up before and during the development.

table 'entities'

This table will be the core of any request regarding a user or a group of users.

- id             string    - a unique uuid to ident the entity
- owner_id       string    - id from 'entitied_accounts' for the owner
- type           string    - type of the entity: no_member, unique, group, admin_group
- visibility     small int - visible to all, visible to owner only, visible to members
- access         small int - free, invite only, request needed
- name           string    - name of the entity
- creation       datetime

• 'no_member' means that the entity contains no members at all: the owner is the entity. As an example, when you share your file to a local user, you're sharing to the 'no_member' entity that have the local user as owner_id
• 'unique' means that the entity will have a unique member: this unique member is the entity and the owner is the one that can use this entity. As an example, when you share to a mail address, you're sharing to the 'unique' entity that have your userId as owner_id and the mail address the unique member of the entity (see the table 'entities_members') and 'name' should be the mail address of that unique member.
• 'group' means that the entity is a group of multiple accounts. works like 'unique' but is not limited to a single account and the 'name' will be editable.
• 'admin_group' is a 'group' with admin rights on the Nextcloud.

Uniqueness is not managed by the database.

table 'entities_accounts'

This table contains all 'accounts' that can be added to an entity: Local Users and Mail Addresses

- id             string    - a unique uuid to ident the account
- type           string    - local_user, mail_address, guest_user
- account        string    - account/user_id
- creation       datetime

'id' is linked to 'entities.owner_id' or to 'entities_members.account_id'.

• 'local_user' means it is a local account identified by the userId
• 'mail_address' is a mail address.
• 'federated_user' and 'federated_cloud' can be one of the possible entries as 'type'.

Uniqueness is managed by the database using 'type' and 'account', meaning that the same mail address can be used by different owner in the 'entities' table. Of course, when sharing to a mail address, we provide completion based on the 'entities.owner_id', meaning that we only returns the known mail address if it was previously used by the user.

table 'entities_members'

This table will help to link 'entities_accounts' to 'entities'

- id             string    - a unique uuid
- entity_id      string    - id from 'entities'
- account_id     string    - id from 'entities_accounts'
- status         string    - invited, requesting, member
- level          small int - 1=member, 4=moderator, 8=admin  
- creation       datetime

'entity_id' is linked to 'entities.id'
'account_id' is linked to 'entities_accounts.id'

• Status and Level will help manage the current rights in the group for each members: like in Circles, moderator can add/remove users, admin can add/remove moderators.

table 'entities_types'

This table should returns interface and class used by some of the types in the previously defined table. This looks like a descent way to have some extension to the whole system.

- id             int       - incremented and primary key, nothing more
- type           string    - string that define the type
- interface      string    - type of the type (sic)
- class          string    - class to be called to manage the service

This table needs some improvement during the developement of the project, my guess is that the 'entities_accounts.type' will be the first field to use this table with type='local_user', interface='IEntitiesAccounts', class='OC\Entities\Accounts\LocalUser::class'

Please provide feedback and comments.

[MorrisJobke]

Looks good so far. I haven't re-assembled all details in my head fully, but roughly it seem to be what we discussed in the past.

Maybe @nickvergessen @kesselb @blizzz @rullzer @ChristophWurst @skjnldsv @juliushaertl also want to check this out, because that is what we want to replace groups, circles and stuff like that with.

[nickvergessen]

My biggest question for talk is how we should use this.
This needs to be circular proof:

• A talk conversation should be an entity, because you can share files into a conversation
• We want to be able to add entities to a conversation, so you can invite e.g. a group, email, circle, ... into a conversation.
• But at the same time we need to be able to detect if a user is removed from a conversation (while staying in the original entity that was invited) etc.

Hope you can put those thoughts into the thingy as well :)

[MorrisJobke]

• But at the same time we need to be able to detect if a user is removed from a conversation (while staying in the original entity that was invited) etc.

That's quite interesting ... because it would mean that it changes either the original entity or creates a copy without this one user. 🤔

[ArtificialOwl]

@nickvergessen : About adding entites, this could be done in 2 ways:

• In a sharing approach, I was thinking to add a 2-level (and non-recursive) way to group groups. The idea is that you could add an slave entity as part of a master entity, without being too heavy on resources.

• In a grouping approach, a better approach would be to create 1-way link between entities.

Not sure there is a real difference from a user point of view, I need to think about the back-end perspective.

@MorrisJobke : We could use the status field in the entities_members database, even better if a sql request on bit flag value is doable. If not, we can have a text field to store that kind of data in a json that will be stored by the EntityManager, but used by the Talk app. A third solution would be that the Talk app manage the in/out conversation flag using the uuid of the entities_member (which is unique per user per group)

Both are just quick answers!

[blizzz]

GroupAdmins

If I understand it correctly and want to apply it to group admins:

• the target entity is of type group
• regular admins have ownership built-in, so they don't need to be in owner_id
• a "virtual" entity of type admin_group is created and all group admins (each entity no_member) are members of it
• this virtual groupadmin entity is the owner of the target entity

That's a bit cumbersome. To have n owners another relational table could be used.

External users and groups

If I understand correctly, backends that might bring users and groups from somewhere else like LDAP and SAML do register theirs as a type in "entities_types"?

Since, right now there is one service for all users and another for all groups, so duplicating them with "local_users", "ldap_users", "saml_users", …, is perhaps not necessary, unless there is a reason for having it like that?

[ArtificialOwl]

@blizzz

GroupAdmins are just a group member with Admin rights on the current group, right ? This should be managed by the IEntityManager itself when the admin level is assigned to a member.
The 'admin_group' type of an entity are groups that provide Nextcloud Admin rights to its members. Like the 'admin' group in today's Nextcloud.
Users within an admin_group also have Admin Level in all 'group' entities.

External users and groups wont need any specific stuff.

Edit: I now understand the misunderstood: 'local_user' is 'local' as from the Nextcloud. the way it authed is not important.

[skjnldsv]

Can we do a table with different examples so I can properly get a grasp on it? It's a tad complex! :)
Please edit the following one ?
@daita

Shares	entity_type	account_type	...	...
User share	unique	local_user
Mail share	unique	mail_address
Talk file share	unique?	??
Talk conversation share	unique?	??
Circle share	group?	??
Public share	unique?	guest_user
Group share	group?	??
(Contact share)?
...

[nickvergessen]

Mind explaining Talk thread share ? Do you mean a file shared into a Talk conversation? Or a Talk conversation in itself?

[skjnldsv]

Yeah, conversation was too long, I was mixed between lazyness and it being too wide for the table 😉
I added the differentiation of the two in the table

[blizzz]

@daita you're right, i missed that field in the members table, sorry.

External users and groups wont need any specific stuff.

But then 'local_users' is also unnecessary?

[ArtificialOwl]

* We want to be able to add entities to a conversation, so you can invite e.g. a group, email, circle, ... into a conversation.

* But at the same time we need to be able to detect if a user is removed from a conversation (while staying in the original entity that was invited) etc.

So !

First, on install of the Talk app, we have a new entry in the entities_types:

type='room'
interface='IEntity'
class='\OCA\Talk\Entity'

When you create a new talk room, it creates a new entity in the 'entities' table :

id='1a2b3c'
owner_id='9a8b7c'
type='room'
visibility=(not relevant right now
access='invite only'
name='your room name'

For each entity (user or group) you want to include in your room

entity_id='1a2b3c'
account_id='[a...f0-9]' (in case of an account)
slave_entity_id=[a...f0-9]' (in case of an group)
status='member'
level=1

The Talk app would also need its own table (talk_left_convo) to manage people leaving the conversation, like this:

entity_id='1a2b3c'
account_id='[a...f0-9]'
left_conversation=true

Now, IEntity would be structured with methods that would returns members but also IQueryBuilder.

protected method getMembersQueryBuilder() {
    $qb = $this->dbConnection->getQueryBuilder();
    $qb->select()->from();

[...]

    return $qb;
}

public method getMembers() {
    $qb = $this->getMembersQueryBuilder();
    $data = $qb->execute();
    
[...]

    return $users;
}

extending it would allow to rewrite the getMembers() method:

public method getMembers() {
    $qb = $this->getMembersQueryBuilder();
    $qb->select('tlc.left_conversation');
    $qb->leftJoin(
        'entities_members, 'talk_left_convo', 'tlc',
	$qb->expr()->eq('account_id', 'tlc.account_id')
    );
    $data = $qb->execute();
    
[...]   // for each entries, check the value of 'left_conversation'

    return $users;
}

Note: adding 'slave_entity_id' in entities_members to link group_entity within a group_entity (with a limit to a certain depth)

[ArtificialOwl]

@blizzz : yes, sorry, I edited my answer half an hour ago about it:

I think I understand the misunderstood: 'local_user' is 'local' as from the instance of Nextcloud. the way it authed is not important. We could name it 'user'.

[ArtificialOwl]

@skjnldsv you're right, let's do some case studies !

• User shares :
  you are sharing to an entity with:

(entities) type='no_member'
(entities) owner_id='user account'
(entities) name='user display name'

• Mail shares
  You are sharing to an entity with:

(entities) type='unique' :
(entities) owner_id='your account'
(entities) name='recipient mail address'
(entities_accounts) type='mail_address'
(entities_accounts) account '[email protected]'
(entities_members) entity_id=(entities) id
(entities_members) account_id=(entities_accounts) id

• Group shares
  You are sharing to an entity with:

(entities) type='group'
(entities) owner_id='owner of the group'
(entities) visibility='all'
(entities) access='limited'
(entities) name='name of the group'
(entities_accounts) type='local_user'
(entities_accounts {entry_1}) account 'userA'
(entities_accounts {entry_2}) account 'userB'
(entities_accounts {entry_3}) account 'userC'
(entities_members) entity_id=(entities) id
(entities_members {entry_1}) account_id=(entities_accounts) id
(entities_members {entry_2}) account_id=(entities_accounts) id
(entities_members {entry_3}) account_id=(entities_accounts) id

• Personal Circles
  You are sharing to an entity with:

(entities) type='group'
(entities) owner_id='your account'
(entities) visibility='hidden'
(entities) access='limited'
(entities) name='name of the personal circle'
(entities_accounts {entry_1}) type='local_user'
(entities_accounts {entry_1}) account 'userA'
(entities_accounts {entry_2}) type='local_user'
(entities_accounts {entry_2}) account 'userB'
(entities_accounts {entry_3}) type='mail_address'
(entities_accounts {entry_3}) account '[email protected]'
(entities_members) entity_id=(entities) id
(entities_members {entry_1}) account_id=(entities_accounts) id
(entities_members {entry_2}) account_id=(entities_accounts) id
(entities_members {entry_3}) account_id=(entities_accounts) id

• Public Circles
  You are sharing to an entity with:

(entities) type='group'
(entities) owner_id='owner of the circle'
(entities) visibility='all'
(entities) access='free'
(entities) name='name of the circle'
(entities_accounts {entry_1}) type='local_user'
(entities_accounts {entry_1}) account 'userA'
(entities_accounts {entry_2}) type='local_user'
(entities_accounts {entry_2}) account 'userB'
(entities_accounts {entry_3}) type='mail_address'
(entities_accounts {entry_3}) account '[email protected]'
(entities_members) entity_id=(entities) id
(entities_members {entry_1}) account_id=(entities_accounts) id
(entities_members {entry_2}) account_id=(entities_accounts) id
(entities_members {entry_3}) account_id=(entities_accounts) id

• Private Circles
  You are sharing to an entity with:

(entities) type='group'
(entities) owner_id='owner of the circle'
(entities) visibility='all'
(entities) access='request_needed'
(entities) name='name of the circle'
(entities_accounts {entry_1}) type='local_user'
(entities_accounts {entry_1}) account 'userA'
(entities_accounts {entry_2}) type='local_user'
(entities_accounts {entry_2}) account 'userB'
(entities_accounts {entry_3}) type='mail_address'
(entities_accounts {entry_3}) account '[email protected]'
(entities_members) entity_id=(entities) id
(entities_members {entry_1}) account_id=(entities_accounts) id
(entities_members {entry_2}) account_id=(entities_accounts) id
(entities_members {entry_3}) account_id=(entities_accounts) id

• Hidden Circles
  You are sharing to an entity with:

(entities) type='group'
(entities) owner_id='owner of the circle'
(entities) visibility='members_only'
(entities) access='all'
(entities) name='name of the circle'
(entities_accounts {entry_1}) type='local_user'
(entities_accounts {entry_1}) account 'userA'
(entities_accounts {entry_2}) type='local_user'
(entities_accounts {entry_2}) account 'userB'
(entities_accounts {entry_3}) type='mail_address'
(entities_accounts {entry_3}) account '[email protected]'
(entities_members) entity_id=(entities) id
(entities_members {entry_1}) account_id=(entities_accounts) id
(entities_members {entry_2}) account_id=(entities_accounts) id
(entities_members {entry_3}) account_id=(entities_accounts) id

• Talk Discussion
  You are sharing to an entity with:

(entities) type='room'
(entities) owner_id='owner of the room'
(entities) visibility='members_only'
(entities) access='invite_only'
(entities) name='name of the room'
(entities_accounts {entry_1}) type='local_user'
(entities_accounts {entry_1}) account 'userA'
(entities_accounts {entry_2}) account 'userB'
(entities_accounts {entry_3}) account 'userC'
(entities_members) entity_id=(entities) id
(entities_members {entry_1}) account_id=(entities_accounts) id
(entities_members {entry_2}) account_id=(entities_accounts) id
(entities_members {entry_3}) account_id=(entities_accounts) id

Note: visibility/access can be changed if you want anyone to be able to join your conversation.

• Public Share (wip)
• Contact Share (wip)

[skjnldsv]

@daita can't you use a table please? This is not very much understandable ^^

[ArtificialOwl]

it isunderstandable from the sql point of view !

anyway userA is the user sharing stuff:

Type of Shares	entity.type	entity.owner_id	entity.visibility	entity.access	entity_accounts.type	entity_accounts.account
User - Sharing to userB	no_member	userB
Mail - Sharing to [email protected]	unique	userA			mail_address	[email protected]
Group - Sharing to GroupA	group	OwnerA	all	limited	local_user	userId
Personal Circle - Sharing to CircleA	group	UserA	hidden	limited	local_user / mail_address	userId or mail address
Public Circle - Sharing to CircleB	group	UserB	all	free	local_user / mail_address	userId or mail address
Private Circle - Sharing to CircleC	group	UserC	all	request_needed	local_user / mail_address	userId or mail address
Hidden Circle - Sharing to CircleD	group	UserD	members_only	free	local_user / mail_address	userId or mail address
Talk - Sharing/Talking to RoomA	room (extends group)	OwnerA	*	*	local_user	userId

[skjnldsv]

it isunderstandable from the sql point of view !

But I'm not a sql expert ;)
Thanks for the tabel!! 🤗

[sunjam]

• Comments. I would love file and folder Comments to be treated as part of a project. This would greatly increase their usefulness, and would probably rely on ironing out comment federation as well.
• Social ActivityPub would also be useful if you incorporate your Mastodon comments or add followers into a project.

[fwolfst]

Just a very quick feedback from an outsider perspective. We all know, naming is hard. To me it seems that Entity might not be a good name (it is as good as Object would be). I dont want to propose something better, as I am not involved and definately do not want to inspire to move away from the focused constructive discussion in this issue, but after reading this, I couldnt refrain from raising concerns about the name (Entity).

[ArtificialOwl]

@fwolfst you'll have to come with some better names :-)

[fwolfst]

some better names :-)

ShareEntity ShareGroup SharingEntity Recipients RecipientEntity SharePool - but probably there are even better ones :)

[ArtificialOwl]

rebased, squashed, documented

@MorrisJobke any chance you would have a look to the current status of the project ?

[chikamichi]

Hi, this feature looks interesting!

It's been a year in the making though, what is its current status?

[sunjam]

Also mentioned in the integration of Project and Teams into Dashboard.

Could be worth involving the new Collectives app, which is an extension of Circles.
Documentation for users, admins, and developers
Here is their Gitlab repo
Here is a request for developing a Collectives Dashboard widget

    Collective and non-hierarchical workflow by heart: Collectives are tied to a Nextcloud Circle and owned by the collective.
    Collaborative page editing like known from Etherpad thanks to the Text app
    Well-known Markdown syntax for page formatting

[ArtificialOwl]

Quick post to explain why Circles might be a descent option to this issue.

Membership

Should be ready for NC22, the app will not define its Circles by 'type' but using options (set by bit flag). Instead of being Personal, Public, Private or Secret a Circle will always be fully lock by default, but configurable to unlock features:

• Single user (Circle contains only one user)
• Personal (Only the owner can see the Circle)
• Visible (Visible to everyone. f not visible, people have to know its name to be able to find it)
• Open (People can join)
• Invite (Adding a member generate an invitation that needs to be accepted by the recipient)
• Request (Request to join a Circle needs to be confirmed by a moderator)
• Friends (Members of the Circle can invite their friends)
• Protected (Password protected)
• No Owner (no owner, only basic members)
• Hidden (fully hidden, fully managed by the backend)
• Root (Circle cannot be a member of another Circle)
• Federated (Federated)

Important notes:

• Single and Personal are really specific Circles, and when selected, no other options can be added. While Personal Circle are managed by a user, Single are only managed by the backend based on the creation/suppression of users (local, remote)

• Open/Invite/Request/Friends can be mixed for multiple results:

 * OPEN                            => everyone can enter. moderator can add members.
 * OPEN | REQUEST               => anyone can initiate a request to join the circle, moderator can
 *                                           add members
 * OPEN | INVITE                => every one can enter, moderator must send invitation.
 * OPEN | INVITE | REQUEST  => every one send a request, moderator must send invitation.
 * OPEN | FRIEND                => useless
 * OPEN | FRIEND | *            => useless
 *
 * INVITE                           => no one can enter, moderator must send invitation.
 * FRIEND                           => no one can enter, but all members can add new member.
 * REQUEST                          => useless (use OPEN | REQUEST)
 * FRIEND | REQUEST             => no one can join the circle, but all members can request a
 *                                           moderator to accept new member
 * FRIEND | INVITE              => no one can join the circle, but all members can add new member.
 *                                           An invitation will be generated
 * FRIEND | INVITE | REQUEST  => no one can join the circle, but all members can request a
 *                                             moderator to accept new member. An invitation will be generated

• The last option No_owner, Hidden and Root can be used by 3rd party app that want to generate their own groups using a provided API.

Federated

I am currently re-writing from scratch the concept of Federated Circles. Based this time on the work already done to have the Circles app and group files sharing compatible with Global Scale.

If the Circles app is [fully] integrated to the Core, any app can share its data with any external instance of Nextcloud.

Sharing

Sharing will be improved a lot, using a dynamic table that list all Unique Id available for each user. Instead of checking all different membership. That will helps managing groups within groups.