rmapi out of sync? (most probably cloud problem of remarkable)

[flomlo]

Hi,

I've been experiencing heavy out-of-sync between the rm-cloud (accessed only via rmapi) and my remarkable 2. Has anyone else been noticing it? My troubles began at 2021-07-05 at around 11:00.

I'm aware that most probably rmapi is not at fault, but something else in the system (not that remarkable as a company is remarkably quick at acknowleding such problems). But I'ld like to inquire here just in case.

Problem summary: rmapi shows me a state of the cloud which is out of sync with the device. File transfer in both directions via cloud does not work. Sending files by mail does work. File transfer via USB-C does work.

I've relinked the device to the cloud and reinstalled rmapi as well as removing rmapi.cfg and thus relinking rmapi. Nothing changed.

Will remove this ticket once rmapi as an error source is ruled out.

[TaiBa131]

Same issue here.

I factory reset my Remarkable, and when resyncing, it got the latest modifications from just before I reset it.

It seems like the cloud works fine for Remarkable devices at least.

Edit: just use the USB-C web interface for the moment, it seems to work just fine. For backup-ing, maybe use SSH so you get the remarkable files instead of e.g. PDFs.

[juruen]

@flomlo @TaiBa131 could you please describe in more detail the problem you both are experiencing?

I just uploaded a PDF through rmapi and it showed up right away on my remarkable (v1) device.

[TaiBa131]

It is as if the state of the cloud (as seen in rmapi), is totally different than what the remarkable device (and clients) can access/does.

This includes files and folders:

• rmapi can download a notebook that apparently only has 23 pages, while it clearly has 24 on the remarkable.
• rmapi can download a PDF, that apparently has 0 as the lastOpenedPage in the .content file, while on the remarkable (synced with the cloud), the last opened page is 42.
• remarkable device can sync after reset, with the exact state it was before the reset (and so, sync from the remarkable cloud), this includes moved files, new files and new folders.

So I'm pretty sure the cloud works for remarkable devices/applications (it was able to re-sync everything as it was before the reset after a factory reset + official Remarkable application syncs as it should).

The cloud, as seen by rmapi is totally different though. As if stuck in time, but editable. Found out about the issue when trying to upload a PDF and the remarkable device wouldn't sync it, and then saw that moved files (from a couple of days ago) were still listed (inside of rmapi) as if not moved.

I deleted the .config file for rmapi and put in a new one-time code but it didn't change anything.

Some info that might be useful:

• Platform: Linux (NixOS stable 20.05)
• Version: 0.0.15
• Device: Remarkable 2

P.S. I also just downloaded the Remarkable Windows client (not rmapi) on a different machine to test, and it works: it has the new folder I've moved some stuff into (as well as other changes.)

[flomlo]

Yes, I've got exactly the same symptoms, down to the letter. I've also rebooted to windows (first time in two years!) just to try the official client, that works flawlessly.

It is, as @TaiBa131 describes, a time-frozen but still editable version of the cloud state.
My system specs are Archlinux instead of NixOS, otherwise the same. Fresh rmapi install, fresh .config.

@juruen: I'ld be happy to dig a bit around and search for errors, but I've never even looked at go. If you could give me a few pointers where to best inject some debug statements (e.g. dumping network messages) that would be great!

[flomlo]

Oh, additionally: I'm on the reMarkable software beta branch (2.8.0.98). Are you as well, @TaiBa131? This might be why it works for most but not for us?

[TaiBa131]

Oh, additionally: I'm on the reMarkable software beta branch (2.8.0.98). Are you as well, @TaiBa131? This might be why it works for most but not for us?

I'm using 2.8.0.98 on a remarkable 2, but I'm not enrolled in the remarkable beta program.

[juruen]

I'm also on 2.8.098 on rm v1.

It could be helpful if you could come up with a short list of actions that leaves the devices out of sync. For instance, I just tried:

• Create a dir on the device
• Move a few files to this new directory on the device
• Run rmapi and check the new directory and the files show up correctly

For the record, I just tried the above and it worked as expected.

[flomlo]

it is more like @TaiBa131 described: Once it stops syncing, it's just completely frozen in time. Nothing you do is represented in rmapi and vice versa.

Might I ask how the devicetoken is generated? Maybe, somehow, rmapi gets an old/invalid one?

[ddvk]

it is more like @TaiBa131 described: Once it stops syncing, it's just completely frozen in time. Nothing you do is represented in rmapi and vice versa.

Might I ask how the devicetoken is generated? Maybe, somehow, rmapi gets an old/invalid one?

the token's validity is 12h iirc, so it gets regenerated quite often.
where are you geographically?
can you ping/traceroute: document-storage-production-dot-remarkable-production.appspot.com

and the result of:
curl "https://service-manager-production-dot-remarkable-production.appspot.com:443/service/json/1/webapp?environment=production&group=&apiVer=3"

and

curl "https://service-manager-production-dot-remarkable-production.appspot.com:443/service/json/1/document-storage?environment=production&apiVer=2"

[ddvk]

@juruen my theory is that they've added more storage instances and are distributing the clients in some way

[flomlo]

@ddvk: I've already answered on the remapy-thread accidentally (we should focus on one of them. maybe here, cause it is primarily an API/rmapi issue?)

I'm located in Graz, Austria and connected via UPC fiber.

---

(repost of from the other thread)

That is the traceroute:

traceroute to document-storage-production-dot-remarkable-production.appspot.com (142.251.36.180), 30 hops max, 60 byte packets
 1  _gateway (192.168.0.1)  1.648 ms  2.635 ms  3.604 ms
 2  217-25-120-4.static.upcbusiness.at (217.25.120.4)  35.756 ms  36.421 ms  36.113 ms
 3  217-25-122-252.static.upcbusiness.at (217.25.122.252)  37.409 ms  36.803 ms  37.087 ms
 4  217-25-123-7.static.upcbusiness.at (217.25.123.7)  37.654 ms  37.949 ms  38.219 ms
 5  * * *
 6  * * *
 7  muc12s11-in-f20.1e100.net (142.251.36.180)  47.168 ms 209.85.252.208 (209.85.252.208)  19.953 ms 142.251.68.120 (142.251.68.120)  19.444 ms

Note that it changes a lot, occasionally 10 hops, sometimes 6.

furthermore, curl results in:
{"Status":"OK","Host":"webapp-production-dot-remarkable-production.appspot.com"} for the first (apiVer3=3) and
{"Status":"OK","Host":"document-storage-production-dot-remarkable-production.appspot.com"} for the second.

[flomlo]

I could offer investigating the network traffic of the official reMarkable windows client. I have to run it in qemu anyway, then I can probably wireshark that from my linux.

Would that help? I'm not tooo familiar with networks sniffing and that kind of stuff, so it would take some time.

[ddvk]

I could offer investigating the network traffic of the official reMarkable windows client. I have to run it in qemu anyway, then I can probably wireshark that from my linux.

Would that help? I'm not tooo familiar with networks sniffing and that kind of stuff, so it would take some time.

The traceroute is a deadend, it all goes to the google cloud.

You can use mitm proxy or Fiddler on windows to inspect the traffic of the windows client. Fiddler being quite easy to use. So far I haven't seen anything different in the requests. The curls I posted lack the group parameter, which could play a role, which you can see by snooping on the traffic

[flomlo]

@ddvk I've now captured a whole full syncing process of the windows app via mitmproxy. Will compare with the initial handshake of rmapi later.

I can share with you the handshake of the windows app, but I would prefer to not upload it on github for obvious reasons.

[ddvk]

@ddvk I've now captured a whole full syncing process of the windows app via mitmproxy. Will compare with the initial handshake of rmapi later.

I can share with you the handshake of the windows app, but I would prefer to not upload it on github for obvious reasons.

yes, dont post auth tokens.

just check if there is any difference in the url for getting the docs.

[flomlo]

So the huge difference first: The official windows client seems to use apiVer=1 exclusively. It also does not call the docs, ever, to get a document tree.
On the first sync it just downloads everything and afterwards it is informed by the server that it should request a new document.

Otherwise most of the stuff seems kinda fine. Except for the API difference o.O

All the stuff for devicetoken etc ... I've could not make out a major difference.

But I've just looked at it by hand, maybe I've missed something?

[flomlo]

Does anyone know an official apiv2-client? I could try the official android app, if it absolutely has to be, but I would have to start and android emulator for that. Which is no fun at all.

[flomlo]

Ok, I've now tried sniffing the the windows-client devicetoken and usertoken and placing it into the rmapi.conf. It is accepted as proper, but requesting the document tree still yields the frozen-in-time results.

So my guess is: Under some circumstances, the v2-api-call to
https://document-storage-production-dot-remarkable-production.appspot.com/document-storage/json/2/docs just yields the old, frozen-in-time document tree.

The windows app circumvents that by never calling for that /docs url at all, instead it calls
POST https://rm-blob-storage-prod.appspot.com/api/v1/signed-urls/downloads HTTP/2.0
and gets as a reply

{
    "expires": "2021-07-11T09:48:52.354068051Z",
    "method": "GET",
    "relative_path": "root",
    "url": "https://storage.googleapis.com/rm-blob-storage-bucket-prod/auth0%SECRET/root?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=url-signer%40rm-blob-storage-prod.iam.gserviceaccount.com%2F20210711%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20210711T094752Z&X-Goog-Expires=59&X-Goog-Signature=SECRET&X-Goog-SignedHeaders=host"
}

(I've censored it a little bit, even if it expires within a minute)

By that method the cloud server tells the client which documents to request and the client downloads them, basically by replacing root with the unique file identifier like e.g. fa587cd7ae5526a95fcf26d3862b0c00e4fc5717a594824c1b1755eb15025d2f

So yeah, for some strange reason some APIv2-calls to https://document-storage-production-dot-remarkable-production.appspot.com/document-storage/json/2/docs work, for others they don't.

Would be someone where rmapi still works would be willing to run mitmproxy and see if his rmapi behaves slightly differently?

[juruen]

@flomlo thank you for taking the time to look into this. If I remember correctly I don't think we use the https://rm-blob-storage-prod.appspot.com/api/v1/signed-urls/downloads URL.

I'll be taking a look at this today.

[flomlo]

@juruen: Where did you find the documentation for the APIv2? Did you reverse-engineer some official app? Did you just guess? Or is it standard google-blob-storage API?

It is really strange that it works for some and not for others.

[ddvk]

also can you post the scopes in the auth token

[ddvk]

latest findings:

rm have 2 sync protocols now: XoApp 1.0 ans XoApp 1.5
the auth token has a scopes field, scope:default being the old style and scope:tortoise, scope:hare scope:fox being the new ones.

some users have beein migrated to 1.5, thus they have the new scopes

if 1.5 is used then the first request is:
https://service-manager-production-dot-remarkable-production.appspot.com:443/service/json/1/blob-storage?environment=production&group=xxx&apiVer=1

which returns the url: https://rm-blob-storage-prod.appspot.com
which has a new api, which has a new backend probably

[flomlo]

Hi,

I login using email/password for my.remarkable.com
With scopes of the auth token you mean the SECRET in auth0%SECRET?

I can post that here, just was not sure how "secret" this was. Will do so once I'm back home.

Nice findings! I'm not sure if it is something in my power to change if I belong to a XoApp 1.0 or XoApp 1.5 server.

I can try whining on the support, though. Maybe this is just an accident and they're not even aware.

On the other hand: Is there a "heavy" syncing client which keeps a local copy already? Or do I have to write it myself? I think I kinda understood the protocol of the windows client.

[ddvk]

@flomlo With the scopes i meant in the authorization bearer jwt token. it a signed json, with a scopes property.

ps: to get the json, you can decode it on https://jwt.io

i'm sure there are offline parsers as well

[flomlo]

Hi,

I've got : for the rmapi client (working with the authdata stolen from the windows client) "scopes": "hwcmail:-1 mail:-1 sync:tortoise" and the one from the windows client is "scopes": "sync:tortoise hwcmail:-1 mail:-1", so kinda the same.

[flomlo]

I'm not too deep into the whole cloud-computing-thingy: Do you think reMarkable is using a standard google API which we can look up? Or do you think it is their own custom brew?

So would one need to reverse-engineer the whole API or can we just look it up?

Even with the new API I'm experiencing one could extract the information formerly received by calling the /docs-url. It will be just a bit a nasty parsing and way more network load.

[ddvk]

I'm not too deep into the whole cloud-computing-thingy: Do you think reMarkable is using a standard google API which we can look up? Or do you think it is their own custom brew?

So would one need to reverse-engineer the whole API or can we just look it up?

Even with the new API I'm experiencing one could extract the information formerly received by calling the /docs-url. It will be just a bit a nasty parsing and way more network load.

i think they are using the standard google blob storage apinow

[AaronDavidSchneider]

I have the same issue with rmapi and rmapy. Are there any news yet?

[ddvk]

I have the same issue with rmapi and rmapy. Are there any news yet?

someone has to reverse engineer the new api and implement it in rmapi. my account is not migrated yet

[flomlo]

Hi,

I'm a tad busy in the moment.
First thing I'ld implement again is the ability to push .pdf onto the device, as I'm missing that functionality a lot. I'm not too confident that I'll find time to implement anything else in the next 1-2 months.

I'll grant access to full uncensored logs (and if the trust is there, probably even more) to anyone interested (and capable) of reverse-engineering an updated version. I pledge to help as well as I can, just not confident that this will be enough.

[AaronDavidSchneider]

I started working on it for rmapy subutux/rmapy#26

[chlunde]

I also had issues with this, and my account was on sync protocol 1.5. I could not get my device to sync even after

• a factory reset
• trying to delete all files using rmapi
• trying to delete all files using the desktop application (under wine)

Finally I created a new account under a different email. This account is on protocol 1.0 and everything is working fine for now.

Possible causes:

• Fairly large ebooks
• Enrolled to beta SW
• rmapi user

I wonder if any of these triggered the issue...

[YigitDemirag]

I also had this issue. Nothing syncs between device-cloud, even after factory reset. rM2 IOS/OSX apps do not sync either. When I contacted with rM customer service they told me to replace the device but if I open a new account under different email, everything works perfectly. I told them to check sync API, still waiting the response..

[Uzay-G]

I also had this problem, and ended up just using wine to run the official wine desktop client on linux, pending a fix.

Steps:

1.Install wine
2. Download the windows desktop app exe from https://my.remarkable.com
3. Run wine <the file you downloaded>
4. open the ~/.wine/drive_c/ProgramFiles (x86)/reMarkable dir
5. Run wine reMarkable.exe

[aqwzs]

Ok, is there any update since?

[ddvk]

thanks to subutux/rmapy#25 I managed to add the new api to ddvk/rmfakecloud@433d2e0
and now I can start experimenting with adding it to rmapi, which will involve

• checking the sync type in the usertoken
• abstract new/old way of syncing and adapt the upload/download
• build and cache the server tree
• compare the existing tree with that of the server and update it

[ddvk]

i've made some progress, with the api, upload/delete seem to work, but it's still in an early fase and I don't have a migrate account to test (so far only against the simulator)

so if someone wants to play with it ( make backups first)

https://github.com/ddvk/rmapi/tree/sync15

[juruen]

A follow-up on this. We just merged @ddvk's awesome work to add experimental support for the new sync protocol. Please take a look at https://github.com/juruen/rmapi#warning-experimental-support-for-the-new-sync-protocol

[darthoctopus]

Can we reopen one of these issues? The new sync functionality doesn't appear to work for me as of 0.0.17 — files still appear on the rmapi tree but not on the device itself.

[kirelagin]

It would be helpful to have this issue pinned (this way it will be more noticeable than the warning in the readme, honestly).

[ddvk]

@darthoctopus what doesn't work? do you see a warning that rmapi is using sync1.5?

[darthoctopus]

I do see the warning, and I understand that the functionality isn't quite there yet. I mean only that perhaps one of the GitHub issues should be left open, to be closed only when the new sync is fully supported.

What happens for me is that despite the warning, the behaviour is otherwise the same as with the older version on the new sync API: rmapi push uploads PDF files, and they appear in the tree from rmapi ls, but don't ever get pushed to the device or the app. I don't get any error messages.

Thanks for all the work!

[ddvk]

@darthoctopus if you create a note on the tablet, do you see it later in rmapi?

[darthoctopus]

Yes, new notebooks do appear in rmapi after being created on the device