RealmSwift.List order not in sync on different clients

[divan84]

How frequently does the bug occur?

All the time

Description

We found the following problem with conflict resolution in RealmSwift.List: When changing the list order on several devices of which some are online and some are offline, the list order will stay different on the devices even after they were re-synced and are all back online.

We would expect the list order to be the same on all devices after sync. Right now, only if the app is deleted and reinstalled the order is the same again.

Stacktrace & log output

No response

Can you reproduce the bug?

Yes, always

Reproduction Steps

1. Create a simple app with device sync on
2. The app should display a simple data model like this:

class Project: Object, ObjectKeyIdentifiable {
    @Persisted(primaryKey: true) public var _id = ObjectId.generate()
    @Persisted public var comments: RealmSwift.List<Comment> = .init()
    @Persisted public var title: String
}
class Comment: Object, ObjectKeyIdentifiable {
    @Persisted(primaryKey: true) public var _id = ObjectId.generate()
    @Persisted public var text: String
}

3. The list items can be re-ordered
4. Log in with with at least two users on different devices
5. One user is offline and randomly changes list order several times
6. Another user does the same while online
7. The first user goes back online

Result
The list order on both devices is different and will stay so until the app is deleted and reinstalled.

Version

Realm 10.33.0, RealmDatabase 12.13.0

What SDK flavour are you using?

MongoDB Realm (i.e. Sync, auth, functions)

Are you using encryption?

No, not using encryption

Platform OS and version(s)

iOS 16.1, macOS 13.0

Build environment

Xcode version: 14.2
Dependency manager and version: SPM

[sync-by-unito]

➤ Jonathan Reams commented:

Sorry for the late response, between the holidays and a bunch of work coming back from the holidays I only got to this recently.

I tried to reproduce this in a C++ integration test in realm core and came up with the below test case, which I think does all the reproduction steps. This test case actually reproduced the issue until I added orig_realm->refresh() towards the end to force reading from the latest version of the database - which would suggest you're seeing stale data. You say in the description that you have to delete/re-install the app? Does that mean if you just close/re-open the app it doesn't update? You shouldn't have to do this for the realm to eventually update, but have you tried explicitly refreshing the realm to make sure you're seeing the latest data? Do you have any trace-level logs I could look at to get a better sense of exactly what's going on between your devices?

TEST_CASE("RCOCOA-1929 repro", "") {
    using namespace std::literals::string_literals;
    std::string base_url = get_base_url();
    REQUIRE(!base_url.empty());
    const auto partition = random_string(100);
    Schema schema = {{"Project",
                      {
                          {"_id", PropertyType::ObjectId, true},
                          {"title", PropertyType::String},
                          {"comments", PropertyType::Array | PropertyType::Object, "Comments"},
                      }},
                     {"Comments",
                      {
                          {"_id", PropertyType::Int, true},
                          {"text", PropertyType::String | PropertyType::Nullable},
                      }}};
    auto server_app_config = minimal_app_config(base_url, "rcocoa-1929", schema);
    TestAppSession test_session(create_app(server_app_config));
    std::string parition("default");
    auto project_id = ObjectId::gen();
    std::default_random_engine engine(std::random_device{}());
    const size_t expected_comments = 10;

    auto shuffle_list = [&](const SharedRealm& realm, List& list) {
        std::uniform_int_distribution<size_t> dist;
        using dist_param = decltype(dist)::param_type;
        size_t n = list.size() - 1;
        for (size_t idx = n - 1; idx > 0; --idx) {
            auto second_idx = dist(engine, dist_param(0, idx));
            if (idx == second_idx) {
                continue;
            }
            realm->begin_transaction();
            std::cerr << "=== Swapping idx " << idx << " with idx " << second_idx << " ===\n";
            list.move(idx, second_idx);
            realm->commit_transaction();
        }
    };

    auto get_comments_list = [&](const SharedRealm& realm) {
        CppContext c(realm);
        auto project_obj = Object::get_for_primary_key(c, realm, "Project", std::any(project_id));
        return List(project_obj, project_obj.get_object_schema().property_for_name("comments"));
    };

    auto comments_to_pks = [](List& list) {
        std::vector<int64_t> ret;
        for (size_t i = 0; i < list.size(); ++i) {
            ret.push_back(list.get_object(i).get_primary_key().get_int());
        }
        return ret;
    };

    create_user_and_log_in(test_session.app());
    auto user_1 = test_session.app()->current_user();
    SyncTestFile user_1_realm(user_1, partition, schema);
    auto orig_realm = Realm::get_shared_realm(user_1_realm);
    {

        std::cerr << "=== Creating initial object ===\n";
        CppContext c(orig_realm);
        orig_realm->begin_transaction();
        auto project_obj = Object::create(c, orig_realm, "Project",
                                          std::any(AnyDict{{"_id"s, project_id}, {"title"s, "test title"s}}));
        auto comments_list = get_comments_list(orig_realm);
        orig_realm->commit_transaction();

        std::cerr << "=== Creating comments ===\n";
        for (size_t i = 0; i < expected_comments; ++i) {
            orig_realm->begin_transaction();
            auto obj =
                Object::create(c, orig_realm, "Comments",
                               std::any(AnyDict{{"_id"s, int64_t(i)}, {"text"s, util::format("comment %1", i)}}));
            comments_list.add(obj.obj());
            orig_realm->commit_transaction();
        }
        wait_for_upload(*orig_realm);

        std::cerr << "=== Comments order " << ::Catch::Detail::stringify(comments_to_pks(comments_list)) << " ===\n";
        std::cerr << "=== Logging out original session ===\n";
        orig_realm->sync_session()->log_out();
    }

    create_user_and_log_in(test_session.app());
    auto user_2 = test_session.app()->current_user();
    SyncTestFile user_2_realm(user_2, partition, schema);
    {
        std::cerr << "=== Opening second realm ===\n";
        auto realm = Realm::get_shared_realm(user_2_realm);
        wait_for_download(*realm);

        std::cerr << "=== Shuffling second list while connected ===\n";
        auto comments_list = get_comments_list(realm);
        REQUIRE(comments_list.size() == expected_comments);
        shuffle_list(realm, comments_list);
        wait_for_upload(*realm);

        std::cerr << "=== Second realm is shuffled ===\n";
        std::cerr << "=== Second realm comments order after shuffle "
                  << ::Catch::Detail::stringify(comments_to_pks(comments_list)) << " ===\n";
    }

    {
        std::cerr << "=== Shuffling original list while disconnected ===\n";
        auto comments_list = get_comments_list(orig_realm);
        shuffle_list(orig_realm, comments_list);
        std::cerr << "=== Original realm comments order after shuffling "
                  << ::Catch::Detail::stringify(comments_to_pks(comments_list)) << " ===\n";
    }

    {
        std::cerr << "=== Re-opening first realm/uploading/downloading ===\n";
        orig_realm->sync_session()->revive_if_needed();
        wait_for_upload(*orig_realm);
        wait_for_download(*orig_realm);

        orig_realm->refresh();
        auto orig_comments_list = get_comments_list(orig_realm);
        std::cerr << "=== Original realm comments order after upload/download "
                  << ::Catch::Detail::stringify(comments_to_pks(orig_comments_list)) << "===\n";

        std::cerr << "=== Re-opening second realm/uploading/downloading ===\n";
        auto second_realm = Realm::get_shared_realm(user_2_realm);
        wait_for_upload(*second_realm);
        wait_for_download(*second_realm);

        auto second_comments_list = get_comments_list(second_realm);
        std::cerr << "=== Second realm comments order after upload/download "
                  << ::Catch::Detail::stringify(comments_to_pks(second_comments_list)) << "===\n";
        std::cerr << "=== Checking order ===\n";
        REQUIRE(orig_comments_list.size() == second_comments_list.size());
        REQUIRE(comments_to_pks(orig_comments_list) == comments_to_pks(second_comments_list));
    }
}

[divan84]

Refreshing the realm unfortunately doesn't help. No restart of the app either - only when the app is deleted and reinstalled, is the data correct again.

I am sending you the Xcode logs containing the realm log messages from a session where a user just goes offline and moves the list items (while another user does the same online), and goes online again - the order of the lists is different for both users and will remain so.

If one of the users then changes the list order, on the others users device it changes the element in the same place (but it is and remains the wrong element)

Xcode-log.txt

[sync-by-unito]

➤ Jonathan Reams commented:

I think I may have reproduced this, but I'm still trying to figure out exactly what's going on in this repro case. I'll update here when I know more.

[ianpward]

I don't think Sync or Core has ever guaranteed the unsorted list order to be consistent across different devices. In order for list order to be consistent you must apply the same sort method - are you sorting the same way across both devices?

[tgoyne]

Lists do guarantee that they are in the order you set them in.

[divan84]

Is there any update regarding this problem?

[danieltabacaru]

@divan84 I took over from Jonathan and I am investigating it.
I looked at the log file you attached, but the client seems to always be online since download/upload messages are flowing from/to the server.

[divan84]

About the logfile: it reflects the state when the client that was offline went back online again (couldn't log the whole session in Xcode because it aborts as soon as you go offline) - see comment above.

It seems that the error is always there when you implement the simple data scheme from above. We can invite you to our test repo that reliably reproduces the issue if it helps.

Do I understand correctly that @jbreams was already able to reproduce the issue?

[danieltabacaru]

@divan84 Yes, we can reproduce it so no need for a new logfile. We'll keep you posted once we found the issue.

[danieltabacaru]

@divan84 We identified the issue and have a fix for it.

[divan84]

I tested the behavior of the realm list again with the current Realm Swift version, but the problem persists.

You can still reproduce the error with the originally described test scenario

[danieltabacaru]

@divan84 I think we correctly identified the original issue, but to confirm whether it's the same or a new one, do you have new logs (ideally at debug level)?

[divan84]

Here are the log files from a device that was always online and from one that was online and offline at the same time:
AlwaysOnlineDevice.txt
SometimesOfflineDeviceLog.txt

[divan84]

@danieltabacaru - Could the log files help to identify the problem?

[danieltabacaru]

@divan84 I didn't get the chance to have a look, will do it this week and let you know what we find.

[danieltabacaru]

@divan84 I am still investigating the logs, it takes longer than expected.

[divan84]

@danieltabacaru - Any news regarding the problem?

[danieltabacaru]

@divan84 Sorry for the delay, we are working on a repro case.

[divan84]

@danieltabacaru - Any updates?

[danieltabacaru]

No luck so far. I have a test which replays the same set of instructions from both users, but the lists are the same at the end of the test. The conflict resolution algorithm yields different results in my test though, so that's what I'm investigating now.

[divan84]

@danieltabacaru:
Any news on the issue?

We still face the problem with the wrong order of lists on the clients...

[danieltabacaru]

@divan84 I am back from vacation and looking into it again.

It's not clear what's causing the issue, but our theory is that it involves the conflict resolution algorithm. The log files you attached have a lot of user changes so it's hard to pinpoint which sequence of instructions is the culprit. It seems you can still repro it, so if you can do it with less user changes, please send the log files.

[sync-by-unito]

➤ danieltabacaru commented:

Moving to backlog for now.