725.conflict resolution

.github/workflows/linux.yml

@@ -86,12 +86,11 @@ jobs:
         name: "unit-test"
         path: "eliot*"
 
-    - name: Upload coverage report
-      uses: codecov/codecov-action@v2
+    - name: Coveralls
+      uses: coverallsapp/github-action@v2
       with:
-        token: "322d708d-8283-4827-b605-ccf02bfecf70"
-        file: "./coverage.xml"
-
+        parallel: true
+        flag-name: "unit"
 
 
   integration-tests:
@@ -157,8 +156,9 @@ jobs:
         name: "integration"
         path: "eliot*"
 
-    - uses: codecov/codecov-action@v2
+    - name: Coveralls
+      uses: coverallsapp/github-action@v2
       with:
-        token: "322d708d-8283-4827-b605-ccf02bfecf70"
-        file: "./coverage.xml"
-        flags: "integration"
+        parallel: true
+        flag-name: "integration"
+        parallel-finished: true

docs/conflicts.rst

@@ -0,0 +1,133 @@
+.. -*- coding: utf-8 -*-
+
+.. _conflicts:
+
+Magic Folder Conflicts and Resolution
+=====================================
+
+When we have two or more participants updating a folder, it can happen that a file is modified "at the same time" by two or more participants.
+More correctly, "at the same time" here means "between communications events".
+
+Effectively, participants are speaking a protocol by posting updates to their mutable Capabilities.
+Namely, updating which Snapshot object a particular (file) name points to (including adding a new name, or removing one).
+
+We do not know *when* another Participant has updated their notion of the current state.
+However, what we *can* notice is when they "commit" to that state (that is, they update publically their current Snapshot pointer).
+
+Each file is considered individually, and is represented by a tree of Snapshots of the contents (plus metadata).
+Included in the metadata are one or more "parent" Snapshots (a brand new file has zero parents).
+
+For more on this, the :ref:`datamodel` has a section on :ref:`datamodel_conflicts`.
+
+
+Communication Between Folders
+-----------------------------
+
+Consider a single file named ``"foo"``.
+When this file is created (for example on the device called Laptop) that device uploads a Snapshot (``S0``) with no parent Snapshots.
+
+By "upload" we mean to push the content and metadata to the Tahoe-LAFS client, receiving ultimately a Capability for the Snapshot ``S0``.
+The file entry for ``"foo"`` is then updated, visualized as: ``foo -> S0``
+Here, "updated" means that we use Tahoe-LAFS to edit to contents of our Personal directory to change "foo" to point at Snapshot ``S0``.
+
+Now, all other Participants can (and, eventually, will) notice this update when they poll the magic-folder.
+
+Each of these updates to our Personal directory is a "communication event", so we talk about the regions between these as a cohesive state of that client.
+That is, anything that happens during that time is "at the same time" for the purposes of this protocol.
+
+Note that this time interval could be as short as a few seconds or as long as days (or more).
+
+Observe too that the "parents" of a particular Snapshot are a commitment to the state visible by a particular client when creating any new Snapshot.
+
+
+Detecting Conflicts
+-------------------
+
+A new update is communicated to us -- that is, we've downloaded a previously-unknown Snapshot from some other Participant's mutable Capability.
+
+We now look at our own Snapshot for the corresponding file.
+Either our Snapshot appears in some parents (including grandparents, etc) of the new Snapshot, or it doesn't.
+
+If it does appear, this is an "update" (and we simply replace the local content with the incoming content).
+
+Instead if our Snapshot does not appear as any ancestor of the incoming Snapshot, a conflict is determined.
+
+This is because when the other device created their Snapshot, they didn't know about ours (or else it would appear as an ancestor) so we have made a change "at the same time".
+
+Note that unlike tools like Git, we do not examine the contents of the file or try to produce differences -- everything is determined from the metadata in the Snapshot.
+This means that even if we happened to make the very same edits "at the same time" it would still be a conflict.
+
+
+Showing Conflicts Locally
+-------------------------
+
+Once a conflict is detected, "conflict marker" files are put into the local magic folder location (our local file remains unmodified, and something like ``foo.conflict-desktop`` will appear.
+The state database is also updated (conflicts can also be listed via the API and CLI).
+
+Although magic-folder itself doesn't try to examine the contents, you can now use any ``diff`` or similar tools you prefer to look at what is different between your copy and other participant(s) copies.
+
+
+Resolving Conflicts
+-------------------
+
+We cannot "magically" solve a conflict: two devices produced new Snapshots while not communicating with each other.
+
+Thus, it is up to the humans using these devices to determine what happens.
+Once appropriate changes are decided upon, a new Snapshot is produced with *two or more parents*: one parent for each of the Snapshots involved in the conflict (we've only talked about one other participant so far, but there could be more).
+
+Such a Snapshot (with two or more parents) indicates to the other clients a particular resoltion to the conflict has been decided.
+
+So there's actually another case when we see an incoming new Snapshot: it may in fact *resolve an existing* conflict.
+If this is the case, conflict markers are removed and the local database is updated (i.e. removing the conflict).
+
+It is a human problem if this resolution is not to your particular liking; you can produce an edit again or talk to the human who runs the other computer(s) involved.
+The history of these changes *is available* in the parent (or grandparent) Snapshots if the UX you're using can view or restore these.
+
+
+Resolving via Filesystem
+------------------------
+
+Not currently possible (see `https://github.com/tahoe-lafs/magic-folder/issues/754 <Issue 754>`_).
+
+
+Resolving via the CLI
+---------------------
+
+The subcommand ``magic-folder resolve`` may be used to specify a resolution.
+It allows you to choose ``--mine`` or ``--theirs`` (if there is only one other conflict).
+Otherwise, you must apply the ``--use <name>`` option to specify which version to keep.
+
+Currently there is no API for doing something more complex (e.g. simultaneuously replacing the latest version with new content).
+As with everything else on the CLI, the :ref:`http_api` may be used to accomplish the same simple resolution tasks as above.
+
+Complete example:
+
+.. code-block:: console
+
+    magic-folder resolve --mine ~/Documents/Magic/foo
+
+
+Resolving via the HTTP API
+--------------------------
+
+See :ref:`api_resolve_conflict`
+
+
+Future Directions
+-----------------
+
+We do not consider the current conflict functionality "done".
+There are other features required to make this more robust and have a nicer user experience.
+Some of those features are:
+
+*Viewing old data*: While it is currently possible in the datamodel to view past versions of the files, we do not know of any UI that does this (and the CLI currently cannot).
+
+*Restore old version*: Similarly, it is possible to produce a new Snapshot that effectively restores an older version of the same file.
+We do not know of any UI that can do this.
+
+*Completely new content*: As hinted above, it might be nice to be able to produce a resolution that is some combination of multiple versions (like one sometimes does with Git conflicts, for example).
+While this isn't directly possible currently, you can always take the "closest" one via the existin conflict-resolution API and then immediately produce an edit that has the desired new content.
+
+*Resolution via file manipulation*: Currently, filesystem manipulation is one API (e.g. you just change a file and new Snapshots are produced).
+Similarly, conflict-marker files are used to indicate a conflict via the filesystem.
+It would be nice if you could use a similar mechanism to *eliminate* conflicts -- one way to design this could be to notice that the user has deleted all the conflict-markers and take this as a sign that the remaining file is in fact the desired resolution.

docs/datamodel.rst

@@ -123,7 +123,8 @@ If any Snapshot is different, it is downloaded and acted upon.
 For a full discussion of this process, see :ref:`downloader`.
 
 Ultimately, for normal updates or deletes, the change will be reflected (or "acknowledged" if you prefer) by updating our own Personal folder after making local changes.
-In case of a "conflict" (e.g. two changes at "the same" time) we will not update the Personal folder until the user resolves the conflict (this part isn't possible yet, see `Issue 102 <https://github.com/LeastAuthority/magic-folder/issues/102>`_).
+In case of a "conflict" (e.g. two or more changes at "the same" time) we will not update the Personal folder until some participant resolves the conflict.
+See also :ref:`conflicts`
 
 Considered together, an abstract view of a two-Participant example:
 
@@ -140,6 +141,8 @@ There is a single file (``grumpy-cat.jpeg``) which has been changed once (the or
 We can see that both Participants are up-to-date because both Personal folders point at the latest Snapshot.
 
 
+.. _datamodel_conflicts:
+
 Conflicts
 ---------
 

docs/interface.rst

@@ -49,6 +49,17 @@ A mechanism to add deprecation of APIs will be added in a future release.
  - **Version 1 (``/v1``)**: initial version of the API (not yet considered 100% stable).
 
 
+Error Handling
+~~~~~~~~~~~~~~
+
+Various sorts of errors may occur while using the API.
+
+Most input-validation errors (e.g. nonsensical argument, missing arugments, parsing errors, etc) will be answered with a "400 Bad Request".
+Sometimes, 500-level errors may occur if something actually "internally bad" has happened.
+A "502 Bad Gateway" will result from incorrect interactions with Magic Wormhole servers.
+A "409 Conflict" results when a conflict-resolution attempt fails.
+
+
 .. _`daemon configuration`: :ref:`config`
 
 ``GET /v1/magic-folder``
@@ -262,6 +273,22 @@ Each item in the ``dict`` maps a relpath to a list of author-names.
 The author-names correspond to the device that conflicts with this file.
 There will also be a file named like ``<relpath>.conflict-<author-name>`` in the magic-folder whose contents match those of the conflicting remote file.
 
+.. _api_resolve_conflict:
+
+POST ``/v1/magic-folder/<folder-name>/resolve-conflict``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Ask for a particular resolution to a conflicted file.
+The body is a JSON object containing the keys:
+
+* ``relpath``: the file name inside the folder
+* ``take``: ``"mine"`` or ``"theirs"`` indicating whose version to take
+* ``use``:  a Participant name indicating whose version to take
+
+Note that you can use only one of ``take`` or ``use``, and that ``take="theirs"`` only works when there is exactly one other conflict.
+
+Returns a ``dict`` mapping ``relpath`` to a list of all Participant names that conflicted before the resolution.
+
 
 GET ``/v1/magic-folder/<folder-name>/scan-local``
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

integration/test_multiuser.py

@@ -206,17 +206,13 @@ async def test_conflicted_users(request, reactor, temp_filepath, alice, bob, edm
     # shouldn't _keep_ trying to download conflicted updates.
 
     # now, wait for updates
-    all_updates = await DeferredList([
-        alice.status_monitor(how_long=20),
-        bob.status_monitor(how_long=20),
-        edmond.status_monitor(how_long=20),
+    await DeferredList([
+        alice.status_monitor(how_long=5),
+        bob.status_monitor(how_long=5),
+        edmond.status_monitor(how_long=5),
     ])
-    # ensure we don't "keep downloading" when there's a conflict
-    for st, updates in all_updates:
-        assert st, "status streaming failed"
-        assert len([e for e in updates if e["kind"] == "download-queued"]) < 2, "too many downloads queued"
 
-    # everyone should have a conflict though...
+    # everyone should have a conflict...
     assert find_conflicts(magic) != [], "alice should have conflicts"
     assert find_conflicts(magic_bob) != [], "bob should have conflicts"
     assert find_conflicts(magic_ed) != [], "edmond should have conflicts"

integration/test_resolve_conflict.py

@@ -0,0 +1,101 @@
+"""
+Testing synchronizing files between 3 or more participants
+"""
+
+from eliot.twisted import (
+    inline_callbacks,
+)
+import pytest_twisted
+
+from .util import (
+    find_conflicts,
+)
+from twisted.internet.defer import DeferredList
+
+
+def non_lit_content(s):
+    # type: (str) -> bytes
+    """
+    Pad the given string so it is long enough to not fit in a tahoe literal
+    URI.
+    """
+    # The max size of data that will be stored in a literal tahoe cap is 55.
+    # See allmydata.immutable.upload.Uploader.URI_LIT_SIZE_THRESHOLD
+    # We don't need to be exactly longer than that threshold, as long as we
+    # are over it.
+    return "{} {}\n".format(s, "." * max(55 - len(s), 0)).encode("utf8")
+
+
+async def perform_invite(request, folder_name, inviter, invitee_name, invitee, invitee_magic_fp, read_only=False):
+    invitee_magic_fp.makedirs()
+
+    code, magic_proto, process_transport = await inviter.invite(folder_name, invitee_name)
+    await invitee.join(
+        code,
+        folder_name,
+        invitee_magic_fp.path,
+        invitee_name,
+        poll_interval=1,
+        scan_interval=1,
+        read_only=read_only,
+    )
+
+    def cleanup_invitee():
+        pytest_twisted.blockon(invitee.leave(folder_name))
+    request.addfinalizer(cleanup_invitee)
+
+    await magic_proto.exited
+    print(f"{invitee_name} successfully invited to {folder_name}")
+
+
+@inline_callbacks
+@pytest_twisted.ensureDeferred
+async def test_resolve_two_users(request, reactor, temp_filepath, alice, bob):
+    """
+    Two users both add the same file at the same time, producing conflicts.
+
+    One user resolves the conflict.
+    """
+
+    magic = temp_filepath.child("magic-alice")
+    magic.makedirs()
+
+    await alice.add(request, "conflict", magic.path)
+
+    # invite some friends
+    magic_bob = temp_filepath.child("magic-bob")
+    await perform_invite(request, "conflict", alice, "robert", bob, magic_bob)
+
+    # add the same file at "the same" time
+    content0 = non_lit_content("very-secret")
+    magic.child("summertime.txt").setContent(content0)
+    magic_bob.child("summertime.txt").setContent(content0)
+
+    await DeferredList([
+        alice.add_snapshot("conflict", "summertime.txt"),
+        bob.add_snapshot("conflict", "summertime.txt"),
+    ])
+    # we've added all the files on both participants
+
+    # wait for updates
+    await DeferredList([
+        alice.status_monitor(how_long=20),
+        bob.status_monitor(how_long=20),
+    ])
+
+    # everyone should have a conflict...
+    assert find_conflicts(magic) != [], "alice should have conflicts"
+    assert find_conflicts(magic_bob) != [], "bob should have conflicts"
+
+    # resolve the conflict
+    await alice.resolve("conflict", magic.child("summertime.txt").path, "theirs")
+
+    # wait for updates
+    await DeferredList([
+        alice.status_monitor(how_long=20),
+        bob.status_monitor(how_long=20),
+    ])
+
+    # no more conflicts
+    assert find_conflicts(magic) == [], "alice has conflicts"
+    assert find_conflicts(magic_bob) == [], "bob has conflicts"

integration/util.py

@@ -540,6 +540,22 @@ def add_snapshot(self, folder_name, relpath):
             ],
         )
 
+    def resolve(self, folder_name, magic_file, resolution):
+        """
+        magic-folder resolve
+        """
+        if resolution not in ["theirs", "mine"]:
+            raise ValueError("Invalid resolution: {}".format(resolution))
+        return _magic_folder_runner(
+            self.reactor, self.request, self.name,
+            [
+                "--config", self.magic_config_directory,
+                "resolve",
+                "--theirs" if resolution == "theirs" else "--mine",
+                magic_file,
+            ],
+        )
+
     def scan_folder(self, folder_name):
         """
         magic-folder-api scan-folder

newsfragments/725.feature

@@ -0,0 +1 @@
+Ability to resolve conflicted files.