FR: Use JSONs for Custom Dictionaries

[chrisgrieser]

I have encountered a few bugs with various plugins in Obsidian, which, as I could determine, was caused by having a large custom dictionary file (~3800 lines, ~425kb) in my vault:

• Bug: Any occurrence of BC-tag-note breaks Breadcrumbs Indexing. SkepticMystic/breadcrumbs#361
• [BUG]: metadata.json cannot be written anymore kometenstaub/metadata-extractor#17 (this issue also includes more details)

it seems that Obsidian is simply having problems parsing large files, since it unsuccessfully searches for metadata in such a large file. the developer of the metadata extractor plugin suggested, that Various Complements should use JSON as a format for large custom dictionaries to avoid such problems, as well to increase performance Various complements as well as Obsidian in general.

[tadashi-aikawa]

Hi, @chrisgrieser.

I couldn't understand why using the JSON format could solve that problem. 🤔

the developer of the metadata extractor plugin suggested, that Various Complements should use JSON as a format for large custom dictionaries to avoid such problems

Could you tell me the link to the comment?

[kometenstaub]

Markdown files are scanned for metadata. It is not recommended to have content in markdown files that isn't markdown, because it trips up the parser and can make Obsidian slow. That has happened with XML files that had the md extension in the past, for example.
A JSON file (or any other non-markdown file) isn't taken into account by the metadata cache, so it can be in the vault without stalling Obsidian.
So it's not primarily about the JSON format, but rather the extension.

Additionally, using the JSON format should also make it easier to create data/have easier compatibility and can be very easily read and worked with in JavaScript.

[kometenstaub]

Plugins that use JSON are e.g. Obsidian Wordnet and Linked Data Vocabularies.

[tadashi-aikawa]

@kometenstaub
Thank you for your explanation!

A JSON file (or any other non-markdown file) isn't taken into account by the metadata cache, so it can be in the vault without stalling Obsidian.

Good. I have the same understanding as you :)

Additionally, using the JSON format should also make it easier to create data/have easier compatibility and can be very easily read and worked with in JavaScript.

I adopted the textbase format because I gave priority to human readability/writability, JSON is a little redundant for humans, and some IME dictionary format supports the same. (Of course, it is better to support both formats)

@chrisgrieser
So, can you resolve the problem by using other extensions instead of .md ? (ex: .csv, .txt, ...)

[kometenstaub]

Yes, other extensions should work fine. txt seems to be a good candidate if you want to keep your format and parsing.

[chrisgrieser]

yeah, txt would work!

I just tested adding a .txt as a custom dictionary, and it does seem to be recognized (number of items in the status bar increases), but the editor suggester does not appear for them.

[tadashi-aikawa]

It works.

[chrisgrieser]

ah, my bad, haven't properly refresh the custom dictionaries 🙈

works!

[tadashi-aikawa]

Related to #118

[tadashi-aikawa]

#118 (comment)
#118 (comment)

[tadashi-aikawa]

@chrisgrieser
I have released v6.1.0-beta4 🚀 If you share custom dictionaries with anyone, the new JSON format will be helpful :)

[chrisgrieser]

by now I do not have the need for json custom dictionaries, since your other suggested solutions works just fine for me, sorry! 😅

[tadashi-aikawa]

Sure! I know 👍 That is merely sharing information 😄

[tadashi-aikawa]

Released v6.1.0 🚀