Rework sync user handling and metadata storage #7300

tgoyne · 2024-01-29T16:32:54Z

This makes a bunch of changes required by both the multiprocess sync and app services split projects because they sort of collided and had to change the same types in conflicting ways.

For the app services split, the core idea is that SyncUser is now an abstract virtual class which has just the interface required by the sync components. SyncManager is no longer involved in user management, but is now responsible for session management. When using sync without the core app services implementation, the entire app namespace will be disabled. The SDK will implement SyncUser to expose the SDK's users to core/sync, and instantiate a SyncManager directly.

The user management functionaly that used to be split between SyncManager and App is now entirely on App. App vends app::User instances, which implement the new SyncUser interface and are roughly a drop-in replacement for the old SyncUser type. App continues to instantiate and own a SyncManager. Unlike before, App's cache of users is now a weak one, and instead app::User instances retain the parent App, which makes it so that a SyncConfig remains valid without something else keeping the App alive.

To support sharing users between processes, the sync metadata store has been entirely rewritten and renamed to the app metadata store to reflect that it's now part of App rather than SyncManager.

Unlike the old SyncUser, app::User is never the source of truth for the state of a user, and instead just holds a cached copy of the last-read user data. Instead there is a unidirectional data flow where all updates are made by writing to the metadata store, which then publishes updates to all live app::User instances. This simplifies the implementation and will make it so that writes to the metadata realm can update users in other processes via normal Realm change notifications. This is not yet fully implemented in this PR.

The metadata store is no longer optional, and instead there are two implementations of it: the persisted store which is backed by a Realm file, and a basic in-memory store suitable for running most tests. The in-memory store should be indistinguishable from the persisted store in any scenario that doesn't involve multiple App instances, unlike the old no metadata mode. The persisted store does not yet have the actual multi-process functionality, but it does make atomic updates to the metadata realm, unlike the old implementation which would split some updates over multiple write transactions. This may fix some bugs resulting from crashing at exactly the incorrect time. The new persisted store is also more consistent about how logged-out users are handled (they are now always in App::all_users(), rather than only until the next launch of the process) and fixes some problems around cleaning up removed users.

coveralls-official · 2024-02-13T21:07:38Z

Pull Request Test Coverage Report for Build thomas.goyne_275

Details

3165 of 3247 (97.47%) changed or added relevant lines in 46 files are covered.
34 unchanged lines in 9 files lost coverage.
Overall coverage increased (+0.5%) to 92.608%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/realm/object-store/sync/app.cpp	399	401	99.5%
src/realm/object-store/sync/generic_network_transport.cpp	4	6	66.67%
test/object-store/util/sync/baas_admin_api.cpp	1	3	33.33%
src/realm/object-store/sync/app_user.hpp	51	54	94.44%
src/realm/sync/network/default_socket.cpp	0	3	0.0%
src/realm/sync/network/websocket.cpp	15	18	83.33%
test/object-store/sync/app.cpp	679	685	99.12%
src/realm/object-store/sync/impl/app_metadata.cpp	643	661	97.28%
src/realm/object-store/sync/app_user.cpp	185	205	90.24%
src/realm/object-store/c_api/app.cpp	64	87	73.56%

Files with Coverage Reduction	New Missed Lines	%
src/realm/object-store/c_api/app.cpp	1	54.06%
src/realm/object-store/impl/realm_coordinator.cpp	1	97.96%
test/object-store/util/sync/baas_admin_api.cpp	1	87.66%
src/realm/util/file.cpp	2	92.75%
test/object-store/c_api/c_api.cpp	2	99.79%
src/realm/sync/noinst/server/server.cpp	3	82.25%
src/realm/sync/network/websocket.cpp	4	82.01%
src/realm/object-store/sync/sync_manager.cpp	5	98.82%
src/realm/object-store/c_api/sync.cpp	15	93.1%

Totals
Change from base Build 2207:	0.5%
Covered Lines:	249420
Relevant Lines:	269329

💛 - Coveralls

tgoyne · 2024-03-08T22:34:28Z

For reference realm/realm-swift@04af045 is the changes required to update Swift for the API changes here.

src/realm/object-store/sync/push_client.cpp

src/realm/object-store/sync/app.cpp

std::string_view can already contain a null pointer and there's no reason to have a pointer to a string view.

We already have a more robust implementation of Uri splitting that should be used instead of a second implementation. Putting the redirect count on Request is more complicated than passing it directly, and testing that there are not unexpected additional requests made via checking the request count directly caught a bug which validating the redirect count did not. A few places used std::optional<std::string> where none and empty were treated identically, which means that it should be just an unwrapped string. The wrapper to heap-copy Requests did not do anything useful.

…he token fields

This introduces the beginning of a split between sync and app services. Object store Sync types (almost) don't depend on the `app` namespace, and can be used independently of it. The SyncUser type now implements only the functionality required internally for sync, and is backed by a UserProvider interface which allows it to request things like access token refreshes. All user management has been removed from SyncManager, and it now only owns the sync client and active sync sessions. SyncSession is mostly unchanged. `app::User` is the new equivalent of the old SyncUser type. The user management which used to be in SyncManager is now in App, which implements the UserProvider interface. Metadata storage for sync and App has been completely redesigned. The metadata store is no longer optional, and instead has an in-memory implementation that should work identically to the persistent store other than not being persistent. The interface has been reworked to enable atomic updates to the metadata store rather than relying on the filesystem mutex in SyncManager, which will be required for multiprocess sync. This required pushing significantly more logic into the metadata storage, which fortunately turned out to also simplify things in the process. The ownership relationship between `App` and `User` has been inverted, with `App` now holding a weak cache of users and `User` strongly retaining the `App`. This ensures that a `SyncConfig` now retains everything it depends on even when app caching is not used. Co-authored-by: James Stone <[email protected]>

…ta realm

…n keys

…al handler

We sometimes need to retry requests after doing things like fetching a new access token, so the response handlers need access to the request. This requires either copying the request or heap allocating it, and as the request body can be quite large heap allocating it is better. Previously we created them on the stack and then moved them to the heap, but as every request goes through the path where they're moved to the heap they should just start there.

kraenhansen · 2024-04-25T12:38:56Z

src/realm/object-store/sync/app_user.hpp

+    // ------------------------------------------------------------------------
+    // SyncUser implementation
+
+    std::string user_id() const noexcept override;


Why not simply id ... it's already namespaced by User?

Most of the places where user_id is used isn't in User.

tgoyne self-assigned this Jan 29, 2024

cla-bot bot added the cla: yes label Jan 29, 2024

tgoyne force-pushed the tg/rework-metadata-storage branch 7 times, most recently from 5778fd4 to 07f0ac9 Compare February 1, 2024 17:20

tgoyne mentioned this pull request Feb 8, 2024

Assorted sync metadata storage refactoring #7329

Merged

tgoyne changed the base branch from master to tg/file-copy-race February 13, 2024 20:21

tgoyne force-pushed the tg/rework-metadata-storage branch from 6e8a96c to 519849b Compare February 13, 2024 20:21

tgoyne force-pushed the tg/file-copy-race branch from d09e596 to e31fc3b Compare February 14, 2024 19:06

Base automatically changed from tg/file-copy-race to master February 14, 2024 21:17

tgoyne mentioned this pull request Feb 16, 2024

Begin decoupling sync tests from App #7351

Merged

tgoyne force-pushed the tg/rework-metadata-storage branch 4 times, most recently from c579d7b to faeb506 Compare February 23, 2024 21:24

tgoyne force-pushed the tg/rework-metadata-storage branch from faeb506 to 7bb0c3c Compare February 27, 2024 22:52

tgoyne force-pushed the tg/rework-metadata-storage branch 4 times, most recently from e7b82a0 to a381d9b Compare March 8, 2024 19:43

tgoyne marked this pull request as ready for review March 8, 2024 22:34

tgoyne requested review from michael-wb and danieltabacaru March 8, 2024 22:35

tgoyne added the no-jira-ticket Skip checking the PR title for Jira reference label Mar 8, 2024

tgoyne force-pushed the tg/rework-metadata-storage branch 3 times, most recently from 94c7d43 to 9489195 Compare April 8, 2024 17:20

michael-wb reviewed Apr 8, 2024

View reviewed changes

src/realm/object-store/sync/push_client.cpp Outdated Show resolved Hide resolved

src/realm/object-store/sync/app.cpp Show resolved Hide resolved

tgoyne added 3 commits April 8, 2024 17:01

Remove a pointless layer of indirection

612363a

std::string_view can already contain a null pointer and there's no reason to have a pointer to a string view.

Update the old metadata realms to have syntatically valid tokens in t…

87d280d

…he token fields

tgoyne force-pushed the tg/rework-metadata-storage branch from 9489195 to 11a70a7 Compare April 9, 2024 00:01

tgoyne and others added 8 commits April 8, 2024 20:33

Make Uri::validate() less weird

41145a9

Add a comment noting schema changes that should be made to the metada…

8e9a2ab

…ta realm

Throw InvalidArgument rather than a std::logic_error for bad partitio…

70919ad

…n keys

Suppress tsan warnings related to stopping BaaSaaS from inside a sign…

c41c4d3

…al handler

Expose App::app_id() directly

3d2101f

Extract some duplicated code in PushClient

84a9d03

tgoyne force-pushed the tg/rework-metadata-storage branch from 11a70a7 to 84a9d03 Compare April 9, 2024 03:33

tgoyne merged commit f9212cc into master Apr 9, 2024
39 checks passed

tgoyne deleted the tg/rework-metadata-storage branch April 9, 2024 15:42

kneth added a commit that referenced this pull request Apr 25, 2024

[bindgen] follow up to #7300

2955493

This was referenced Apr 25, 2024

[bindgen] follow up to #7300 #7623

Merged

Upgrade to HEAD of Realm Core's master realm/realm-js#6637

Merged

kraenhansen reviewed Apr 25, 2024

View reviewed changes

jedelbo pushed a commit that referenced this pull request Apr 26, 2024

[bindgen] follow up to #7300 (#7623)

bec09f8

This was referenced Apr 26, 2024

RCORE-2100: Restore ability to create test users #7632

Merged

Restore ability to create fake users for tests #7633

Closed

dianaafanador3 mentioned this pull request Apr 29, 2024

Update for the app services split branch realm/realm-swift#8539

Merged

kraenhansen mentioned this pull request May 2, 2024

Deadlock when calling App::current_user from an App listener #7670

Closed

nielsenko mentioned this pull request May 3, 2024

Issue while a app is on running on background via workmanager realm/realm-dart#1511

Open

github-actions bot locked as resolved and limited conversation to collaborators May 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework sync user handling and metadata storage #7300

Rework sync user handling and metadata storage #7300

tgoyne commented Jan 29, 2024 •

edited

Loading

coveralls-official bot commented Feb 13, 2024 •

edited

Loading

tgoyne commented Mar 8, 2024

kraenhansen Apr 25, 2024 •

edited

Loading

tgoyne Apr 25, 2024

Rework sync user handling and metadata storage #7300

Rework sync user handling and metadata storage #7300

Conversation

tgoyne commented Jan 29, 2024 • edited Loading

coveralls-official bot commented Feb 13, 2024 • edited Loading

Pull Request Test Coverage Report for Build thomas.goyne_275

Details

💛 - Coveralls

tgoyne commented Mar 8, 2024

kraenhansen Apr 25, 2024 • edited Loading

Choose a reason for hiding this comment

tgoyne Apr 25, 2024

Choose a reason for hiding this comment

tgoyne commented Jan 29, 2024 •

edited

Loading

coveralls-official bot commented Feb 13, 2024 •

edited

Loading

kraenhansen Apr 25, 2024 •

edited

Loading