Explicit sync

[Deleplace]

What

A "synchronize now" button, and an indicator of success/failure.

Why

Preamble

I've been using this nice service for several years on many devices. It is very useful for me. Thank you so much to the whole team!

Problem

Synchronization between client and server sometimes fails. Some new notes are not uploaded, some recent notes are not imported, and we have no indicator to know if the client is "currently" (i.e. reasonably recently) up-to-date with the server, or not. With 2 clients open, we don't know which one is failing.

Disclaimer

Managing a distributed state is no picnic. I totally acknowledge that it is hard to guarantee that the state of an open client (currently in use) is always the same as the state of the server.

Tradeoff

In an ideal world, we don't need any bloated UI overhead for this : sync always succeeds and client data is always up-to-date with server data (in both directions). Fewer widgets is better. And this works indeed, most of the time!

When it doesn't, it would be very useful for me to:

• have the client acknowledge that is has not been able to sync for more than one minute (e.g. an orange bullet instead of green bullet),
• let me explicitly "try again" to sync,
• if the explicit action fails, then it's the right time to display an error message with a plausible cause (e.g. "connectivity problem" is not the same as "encountered corrupt data")

I call this a difficult tradeoff as it's simultaneously about confessing that things are going wrong, and adding complexity to mitigate the problem with a "better degraded experience".

I emphasize that this is very different from "fixing all the existing synchronization bugs". Fixing the bugs is of course useful, but IMO it's important to acknowledge that sync problems will always exist (e.g. internet connectivity issues) and still make the app as useful as it can be.

[Deleplace]

This problem is the same with all client distributions, and the suggested feature is similar.

However the implementation would live in different repos for different clients, which is why I created the same issue:

• in simplenote-electron #1897
• in simplenote-ios #639
• in simplenote-android #945

[jleandroperez]

Hello there @Deleplace,

I'm afraid adding a Sync button is not possible. Why:

0. Simplenote's Sync'ing mechanism is handled by Simperium
1. The Simperium library keep track of changes performed over entities, and relay them immediately to the backend over a WebSocket
2. Such WebSocket is expected to be always open, in theory. But when it's not, local changes are just stored in a (local) database
3. If the WebSocket is closed, for whatever reason, the second it's reestablished, local pending changes are posted immediately

There is no timer that dictates when sync'ing would occur, and literally no entry point in which a Sync event could be forced: Sync is expected to always happen, whenever there's connectivity.

You can read more about the Simperium protocol here and here.

Now, all of that being said, if you're experiencing sync'ing issues, we'd very much love to get them fixed ASAP. Few questions for you!

A. Is this something you can reproduce regularly?
B. Are you using multiple devices simultaneously?
C. Can you recall specific steps?
D. Are you running the latest + greatest? (a sync'ing bug was recently patched, that involved emojis).

Thanks in advance!

[jleandroperez]

Closing this issue for now. We'd love to track (and definitely fix) scenarios in which Sync'ing breaks, you can contact us over this channel, or support (at simplenote dotcom).

Thank you!

[Deleplace]

Thanks Jorge.

Indeed the "fallback to local database" works well and I've never experienced the loss of a recent, pending note.

I have many times terminated a client only to relaunch it in hope it syncs, I think it helped (can't prove it right now) and the app launch is an example of an entry point event where the app actively tries to connect and sync, even if it has nothing to send, to retrieve new changes.

The "problem with internet connectivity" was an example of a possible cause of difficulty, but actually when I'm using the desktop (electron) app I know for sure that the internet is fine, and SimpleNote is experiencing some kind of bug.

I'd love to be able to reproduce this problem on demand, but I can't. The use cases in which it impacts my workflow are either "I need to access this specific note for reading" or "I need to modify this specific note", but it's not yet propagated to the device where I need it.

If you know a way to "dump the internal state" of the client when I observe it's obviously out-of-sync, then I'm willing to do it and send the dump.

I often "use multiple devices simultaneously" i.e. have at least 2 clients open at the same time, which has sometimes caused another category of sync problems like "my last sentence has disappeared" even if I was typing in the note text only on one device at a given time.

[jleandroperez]

cc'ing @belcherj: JB, is it there any (public) way to get a diagnose log // internal state in the Electron app?

As per iOS, there is no way to present the Debug UI (other than running a binary from the beta channel). We'll work on a mechanism to enable such UI in the AppStore build (#676), that'd give us the internal state (unsent changes // websocket state).

Is this a bug that's affecting a specific client, or all of them? (mobile / desktop). Do you usually use emojis in your notes?

Thank you @Deleplace !!!

[sshaikh]

The issue is one of information - it would be useful to know when the last sync was and a button with feedback is one way of doing that. So although it's arguable that a button would have little value a visible "last updated" would be great for those situations where you have content on your phone and now know that it hasn't yet been pushed - so that you don't go and make changes elsewhere before that push has happened.

Something like

1. edit a note (on android or whatever) at t0
2. see that the last sync was < t0
3. make a note that your "gold" source is currently only on your phone and so not to make changes elsewhere until it's sorted.
4. once last sync >= t0 then show a green light or something.

It's a shame to have to go through the same lost/diverging notes -> support -> issue -> closed dance each time something breaks, when a simple indicator would tell us there's an issue before we get into that versioning mess.

[jleandroperez]

Hi there @sshaikh,

The canonical version is always the one stored in the backend, and it's up to the backend to accept or reject changes (so you cannot really mark the phone as the "gold source", not without modifying the way everything works).

On top of this, I'm afraid that providing a last sync indicator would add little to no value, in the eventuality of a sync'ing bug. You'd probably see that everything is sync'ed, when it's really not (because of an underlying bug!).

May I ask you if you're experiencing any troubles? If so, we'd very much love to get that fixed.

• Do you use multiple devices?
• What version(s)?
• What was the flow like? Is this something you can consistently reproduce?

Thanks in advance!

[sshaikh]

The canonical version is always the one stored in the backend, and it's up to the backend to accept or reject changes (so you cannot really mark the phone as the "gold source", not without modifying the way everything works).

I meant "mentally". Eventually, yes, you would want the backend to be the truth. But I would say the /latest/ unsync'd version has value. The point is if the user knows it's not been pushed, then they are in a better position to take care of it (eg by holding off on any updates elsewhere, exporting it, copying it etc).

On top of this, I'm afraid that providing a last sync indicator would add little to no value, in the eventuality of a sync'ing bug. You'd probably see that everything is sync'ed, when it's really not (because of an underlying bug!).

It's not about fixing anything, it's about letting the user know what's going on. At the moment we are blind to /known/ or even /expected/ sync failures. Take the basic case where a phone has no network for instance. It would be handy to know not to expect the latest version of a note (made on the phone) to be available on your desktop. Right now it's a bit of a lucky dip, and requires manual visual verification to see which instance of a note should be edited next to in order to maintain a linear history. It becomes quite anxious actually.

May I ask you if you're experiencing any troubles? If so, we'd very much love to get that fixed.

And so the support -> issue -> reject cycle continues ;).

[jleandroperez]

I'll bring this discussion up with the team, thank you for your feedback @sshaikh!