how to retrieve notes?

[alicethedreamer]

Question:
I took a lot of notes and highligted text on a epub file, which I understand are saved on .json file (stored under /.var/app/com.github.johnfactotum.Foliate/data/com.github.johnfactotum.Foliate). I recently opened the file of the same epub book from another backup directory, which didn't have any highlights, and now I cannot find the way to retrieve the previous notes and highlights.

1. Is there a way recognize which .json file is associated with the previous annotated epub file?
2. In short: How can I retrieve my notes?
3. This question is also related to the way to sync and backup those notes: What is the best way to back up foliate folders without losing notes? Is it possible make a backup and restore those .json files in a newly installed computer? (I have searched online and couldn't find anything specific other than:

Foliate stores your current position as well as the bookmarks and notes specifically for each book below ~/.local/share/ in the com.github.johnfactotum.Foliate/ directory. If you move this folder to a cloud store such as Nextcloud or Dropbox, the data can easily be synchronized across multiple computers. You can start reading on your desktop computer during the day and continue reading on your laptop later in the evening. here
I haven't read any tutorial how to do exactly that, it would be nice to have that on Wiki here.)

Version:

• Foliate version: Flatpak 2.6.3.
• OS/Distribution and version: [LinuxMint 20.3]
• Desktop environment: [Cinnamon]
• Installation method: [Flatpak]

[johnfactotum]

The filename of the .json file is the (URI-component encoded) unique identifier of the book. When you open a book, it will look for the .json file with the unique identifier defined in the book. So you don't have to do anything special when syncing or making backups. It should just work.

If it doesn't, there are a few possibilities:

• Make sure the .json files are located where they should be, in Foliate's data directory.
• Make sure that the book's unique identifier has not been modified.
• If you had upgraded from Foliate < 2.4.0: previously Foliate used the first identifier in the book, instead of the one designated as the unique identifier. This was changed in 2.4.0.

To debug and to manually work around the issue, I'd suggest:

1. Open the book with the --verbose argument. It should log a message saying writing to [...].json.
2. grep in the ~/.var/app/com.github.johnfactotum.Foliate/[...] directory to find the .json file containing your old notes of the book.
3. Examine the filenames and contents of these two files to see what might have caused the issue. If desired, override the former with the latter.

[alicethedreamer]

Thanks for the reply. I am not a developer or a coder, and I use the terminal only when really necessary. Can you add the full terminal code that are required to

Open the book with the --verbose argument. It should log a message saying writing to [...].json.

I have not been able to do that. I can change the folder & file name, of course.

[johnfactotum]

If using Flatpak: flatpak run com.github.johnfactotum.Foliate --verbose

[alicethedreamer]

I received "Can't open file" when I tried the --verbose argument using Flatpak. I removed the Flatpak version and installed the PPA for Ubuntu, copied the folder to the new location, can not use --verbose but the file is empty of any notes or annotation.
Going back to my original question: if there is one location where the .json files are kept on the disk, and foliate keeps only one file for each book, does that mean that when I opened the book from the backup book folder, it saved the newly opened book on the older one with the notes?
I have lost notes for two books now, which were both important. I tried to figure out a reliable way to store and backup .json files and I would appreciate if there is a documentation on how to do that.

[alexandrebarbaruiva]

@alicethedreamer try the following command:

nautilus ~/.var/app/com.github.johnfactotum.Foliate/data/com.github.johnfactotum.Foliate/ &

It'll open your files app (assuming you use nautilus, the default file manager for PopOS, Ubuntu and other distributions), and your notes will be inside one of those .json files right at the end. Usually the file starts with some info about the book and only then shows your notes and stuff

[johnfactotum]

To elaborate,

What happens when Foliate opens a book

1. It reads the unique identifier of the book. Then it encodes this identifier with encodeURIComponent().
2. It looks for a file named X.json, where X is the encoded identifier, in a directory called com.github.johnfactotum.Foliate in $XDG_DATA_HOME, which by default is ~/.local/share/ for non-sandboxed installations, and ~/.var/app/com.github.johnfactotum.Foliate/data/ for Flatpak.
3. If this .json file exists, it will load the last reading location as well as bookmarks and annotations stored in the file.
4. After the book is opened, it will save the current location as you read through the book. It will also save any bookmark or annotation that you make. This happens whether the .json file already exists or not. If it already exists, the existing .json file is modified. If it doesn't exist, then the .json file will be created.

Notice that the .json file is completely independent of the book. The only thing that links the .json file to a particular book is the fact that the name of the .json file is the unique identifier of the book. It doesn't care about where the book is stored. As long as you open a book of the same identifier, it will load the notes associated with that identifier.

How to backup or sync annotations

Given the above, to backup annotations, you simply have to backup the .json file. To restore your backup, you just put the .json file back to its original location.

For example, suppose you're reading a book with a unique identifier foobar. For brevity's sake I will use the non-Flatpak path here:

• To backup notes for this book, copy ~/.local/share/com.github.johnfactotum.Foliate/foobar.json to another location, say, ~/Backup/foobar.json.
• To restore the backup, copy ~/Backup/foobar.json back to ~/.local/share/com.github.johnfactotum.Foliate/foobar.json.
• To sync all annotations, configure the syncing solution of your choice to sync all .json files in ~/.local/share/com.github.johnfactotum.Foliate.

How does one get the unique identifier of the book

Well, you can open the About This Book dialog (Ctrl + I) to see all the identifiers of a book. Unfortunately, you can't tell which one is designated as unique from this (which is really an oversight that should be mended). Furthermore, it shows the parsed identifier, rather than the raw identifier used to name the .json file. One can trying viewing the book's source file to get the identifier, but that's a bit too much trouble.

So an easier and more reliable way is to run with the --verbose argument, in which case it will print something like

Gjs-Message: 01:23:45.678: JS LOG: writing to /home/user/.local/share/com.github.johnfactotum.Foliate/foobar.json

in which case one knows for sure, that this .json file is the one that Foliate is reading from and writing to.

Now, if you're trying to recover lost notes, this alone isn't helpful. This only tells you where Foliate expects the notes to be. However if you have the notes backed up somewhere, or the notes are really still there, just stored with a different filename, then simply copying or renaming the file will resolve the issue.

Finding orphaned data

As mentioned above, it's possible that the data is not lost, only that the data isn't associated with the correct unique identifier.

If that's the case, one can use something like grep to find the .json file associated with a particular book. For example, if the book is titled "Foobar", or if the notes you made contained "Foobar", running

grep -l "Foobar" ~/.local/share/com.github.johnfactotum.Foliate/*.json

or, for Flatpak,

grep -l "Foobar" ~/.var/app/com.github.johnfactotum.Foliate/data/com.github.johnfactotum.Foliate/*.json

will show a list of .json files containing the word "Foobar".

Alternatively, you can also search in Foliate's library view for orphaned data. While you probably won't be able to directly open the book, you can click on the three-dot menu and choose "Export Annotations…" to get a copy of the .json file.

[alicethedreamer]

I keep copies of the books folders as well as .json folders as backups. I was able to retrieve the annotations to one of the books, a recent one, by copying the .json file connected to the book from a backup folder to the .local/share/com.github.johnfactotum.Foliate/ folder.

3. If this .json file exists, it will load the last reading location as well as bookmarks and annotations stored in the file.

4. After the book is opened, it will save the current location as you read through the book. It will also save any bookmark or annotation that you make. This happens whether the .json file already exists or not. If it already exists, the existing .json file is modified. If it doesn't exist, then the .json file will be created.

I had lost the notes when I opened the book file from a backup folder. So I think what had happened is when I opened the book (not attached to the .json file) from the backup folder, it saved this new location and didn't load the annotations. I don't know how the .json file is not loaded properly but was saved with no notes in it, because I could find only one .json file that is connected with the book in these current and backup Foliate folders.

I couldn't retrieve the notes to another book which I had lost some time ago, I assume it was before Foliate 2.4.0.

From my experience, I conclude that opening the book file from another drive/folder location changes the .json file connected with book, and this remains an issue for me.

Thanks for the detailed answer.

[johnfactotum]

I don't know how the .json file is not loaded properly

Well, either it's not loaded, or the notes might have been lost while saving the file.

Currently it silently assumes that there's no data when it encounters any error while reading the file. Which means that if the .json file is corrupted in any way you would lose the whole file. Which is bad and probably should be changed.

On the other hand, it's not impossible that the notes might have been lost when saving the file. I know that GJS had changed how GObjects properties work in the past and it's possible that annotation objects were incorrectly serialized.

One might be able to test to see which of these is more likely. Inspect the .json file. If the JSON object doesn't even have an annotation property, then most likely there was an error when loading the file. On the other hand, if there is such a property, but its value is an empty array ([]), then most likely the annotations weren't saved correctly, or were accidentally deleted for some reason.

From my experience, I conclude that opening the book file from another drive/folder location changes the .json file connected with book, and this remains an issue for me.

Whatever reason that might have caused it, the location of the book file has no bearing on the issue. That much I am certain of. In any case this is the easiest possibility to test for: just open some books from different locations and see if you can reproduce the issue.