Download/store only used icons

[LinguaOccultus]

Hello!

The reason is simple: if you download all the icon packs to be able to search them for icons you want to use, you have thousands of small files in the Obsidian folder. And, as commonly known, a large number of (small) files is VERY inefficiently handled by Windows operating systems (no, switching to Linux is no option! ;-) and makes backups VERY slow.

So, I would like to being able to search through the icons but only having those locally stored which I currently use. I think, this makes sense, but seems not to be possible currently. I don't know if this is (easily) being made possible, but I thought I start by asking here ...

Greetings!

[AlexanderMelde]

I support this request, having only those files that are really needed in the file system would also solve the syncing issues

[LinguaOccultus]

At least all those tiny icon files should be gathered in container files instead of laying loose in a directory.

However, I'm not sure how complicated it is to access the icons for use and how complicated the implementation is.
Maybe a tardeoff or first step could be, downloading everything as/into a container file and just unzip the used ones loose until a better/more complex implementation is found?

Whatever the solution will be, everything is better than hundreds of small separate files ...

[AlexanderMelde]

I have created the script shown in #157 for deleting any unused items from the file system. This might help already, even though other icons are of course no longer available once deleted (until you re-download them)

[FlorianWoelki]

Hey there, thanks for the suggestion, and thanks @AlexanderMelde, for the little script.

There might be several questions that come to my mind while thinking about a solution:

• What if you are using a lot of icons? The syncing issue can still persist
• Thinking about this kind of "offline" solution, it wouldn't be possible anymore to use icons when you are offline and have no access to the internet

Furthermore, you can also solve the syncing issue when you create your custom icon pack with all the icons you have inside. This way, only the icons are shown that you are using.

[AlexanderMelde]

What if you are using a lot of icons? The syncing issue can still persist

Given that the number of required icons will always be smaller or equal to the number of notes and folders a user has, this will especially help users with a small number of notes and icons.
Users with a lot of icons propably already have syncing issues due to the high number of files anyway, but the smaller ones can really benefit from it.

it wouldn't be possible anymore to use icons when you are offline and have no access to the internet

No new icons could be downloaded, but those already on the disk can still be re-used. You would just have a more limited selection of icons available. It would always be possible to provide a "keep everything offline" option for users that heavily depend on setting icons while being offline.

you can also solve the syncing issue when you create your custom icon pack with all the icons you have inside

True, i thought about that as well. Instead of deleting unused items with thescript, we could copy all those that should be kept to a new icon pack.

[LinguaOccultus]

I'm not sure what you guys are talking about regarding online/offline usage; I just use Obsidian local at the moment, install the extension and then download the icon packs using the extensions settings. Which results in folders witht hundreds of loose, small files which are generally a problem when it comes to performance and efficiency. Sure, a large Obsidian database would have/make the same issues as every other kind of similar situation.

Therefore, applications often use container/archive files instead. For example, when making backups on my NAS, I put such kind of data in some kind of containers like compressed archives or ISO files. This makes synchronizing my backups much more efficient and faster by far, even on mid to high end machines.

However, in this case, I cannot simply do so since the extension is looking for loose files in specific directories. Since I have to download a whole icon pack even when just using a single of them and I normally download all of them to be able to search them, the quickly results in a high amount of files.

So the question was if it would be possible to just download the used ones. Maybe the search can be made online instead as a treadeoff? Like Bootstrap and such handle this on their icon pages? And then download only the used ones. Or, the other qay round, at least put the packs into a single file or keep them in one (since they surely are hosted and downloaded as compressed archives or such) and access them inside. Or, just decompressing the used ones from them.

Everything would be better than the current situation ... but that's just my use case.

[AlexanderMelde]

I'm not sure what you guys are talking about regarding online/offline usage

• In order to download new icons, you would need to be online. I consider this an online usage.
• Offline usage can be seen as a "flight mode" with no internet connection. In this case you can only access previously downloaded and now locally stored icons.
• The synchronization i mentioned is happening due to the folder my files are stored in is synchronized via Onedrive. This however shouldn't be any concern to Obisidan, towards Obsidian it is just a regular C:/.../ file path on my local disk. The advantage of that sync is that i can use the same notes on different computers, without the official Sync subscription.

Which results in folders witht hundreds of loose, small files which are generally a problem when it comes to performance and efficiency

This is true, and with alle the many small files, Onedrive takes very long to synchronize as well.

Maybe the search can be made online instead as a treadeoff

I think this is the way to go, with an additional option to keep the current behavior of pre-downloading everything for users that want to search all items even when offline.

just decompressing the used ones from [compressed archives]

good idea, i wonder how much CPU load this would cause, but seeing that search only happens rarely (and its mostly just displaying used one), this seems like a great tradeoff!

[LinguaOccultus]

Ah! Sorry, I am German so I maybe misunderstood something here. I thought you were talking about the usage of Obsidian itself.

If possible (and allowed), maybe the icon packs could be boundled with the extension and installed with it. But I have/see no problem with that part, as you need to be online to install the extension itself, too. After that, everything is offline, anyway (until you update something or so).

Okay, running a sync also as mentioned and additionally the mentioned NAS backup in my local network. That sounded you are talking about the official (and paid) Obsidian Sync service at some point which I do not use by now.

The local backups are not perfoming much better; sure, local network is much faster and all, but Windows (for example) never handled many files efficiently and sometimes copies larger single files even faster. That's part of the problem, too.

So, you mean hosting and searching online and only downloading what is needed/used? Sounds good so far. I see where this is a problem without connection, but since this is the usual way today for most other situations ...

I do not think that it is a performance related question for todays systems extracting some files that small from a compressed archive (especially since games, for example, are using the same technique for textures or archive files while running). They do not even have to be compressed but the archive just used as a container, like a ISO file. On top of that, it could also use a local cache like installers do. I do like the »online variant« much more, but maybe both could be implemented, depending on the situation? However, depends on how much work that is and if it's worth it for »just« an extension.

However, this could be send to the Obsidian devs for implementation also since many other extensions could benefit from such a system; not to mention all of its users! I would aspect something like this by default — like the still missing central configuration (at least optional)!

[autonia]

Unfortunately, I've had to abandon using this plugin because it is essentially incompatible with the official Obsidian Synch. As others have stated, it clogs up the sync with all of the tiny icon files, and makes the timely update of actual markdown files too slow and cumbersome. Came here hoping to see that this was being addressed. Fingers crossed that is gets resolved eventually.

[AlexanderMelde]

@LinguaOccultus Hehe, also mindestens drei Deutschsprachige hier ;)

local network is much faster and all, but Windows (for example) never handled many files efficiently

exactly, having many small files seem to be bad no matter which system you use

I do not think that it is a performance related question for todays systems extracting some files that small from a compressed archive (especially since games, for example, are using the same technique for textures or archive files while running). They do not even have to be compressed but the archive just used as a container, like a ISO file. On top of that, it could also use a local cache like installers do.

Great idea, didn't think of the games but you are absolutely right. Especially when no compression is used.

I do like the »online variant« much more, but maybe both could be implemented, depending on the situation? However, depends on how much work that is

Online might be more difficult than the .zip Management as you propably would need to host a server that provides single .svg files. Storing and reading from .zip however seems very possible, there are several nodejs libraries to read .zip files. From those docs, i wrote the following sample code:

fs.readFile("font-awesome-solid.zip", function(err, data) {
    if (err) throw err;
    JSZip.loadAsync(data).then(function (zip) {
        zip.file("MagnifyingGlassDollar.svg").async("string").then(function (icondata) {
            console.log(icondata); // icondata is a string containing the svg
        });;
    });
});

this could be send to the Obsidian devs for implementation also since many other extensions could benefit from such a system

Not sure if they can provide anything useful here, do you think of a kind of file api that they provide to plugins which internally zips files?

like the still missing central configuration

What do you mean by this? I only know the options page with the several .json files in the .obsidian folder, they already exist.

[LinguaOccultus]

»Jawohl!« — just a joke to keep our english speaking fellows here entertained! ;-) I guess, we continue in English so everyone can follow ...

If a game, which normally uses much, MUCH more of the systems ressources most of the time, can do so even with compressed data, this should not be a problem for this case regarding the performance. And the point is to gather all those scattered files together and not reducing theit already tiny size. So, this should work fine, if used. And all compressed formats normally support an uncompressed archive option also; like 7-Zip's 7Z format, the ZIP format and what not. Since those already are the most used archive formats, compatibility also should be no issue, even on non-Windows systems.

About the online solution and since you are going in the CDN direction: yes, BUT! Many of those free icon collections are already available in this way, like the Bootstrap icons, for ecample. The question here is, is it allowed to use this and does the developer give away some control of it in this form? And can it be implemented with a foreign CDN at all? I guess so, if all are a »yes«, but that still does not necessarily mean, this is a good idea or the best way. But I still like the »online« variant best of all possible solutions we discussed so far ...

Regarding the implementation in Obsidian: Since this surely affects several extensions and — as we also agreed: Obsidian notes aswell at a given amout of them — it would make sense to implement a native function for it, right? I really do like the simple, easy and native style of Obsidian's MD infrastructure, but the same issue with the dozens of loose files arises here, too. So ... however, I guess this is MUCH more complicated since it would change the way the database works quite a lot! Therefore, maybe just a function for media file handling for extensions or an API would be at least woth a thought.

This is off-topic, but: the config is saved is save in the ».obsidian« folder you mentioned per database, yes. At least after you have two or more of them, this is a problem. For example, you need to update extensions in two separate locations. Or changed settings. Sure, you can set them individually, but I guess this could be solved much better than that by setting exclusions per database from a main configuration. Often discussed on their forums and I guess there will be changes since the application in total is very new still.

[FlorianWoelki]

I was pondering a little bit about this problem, and thanks to your suggestions, the following might be possible:

Currently, the process of selecting an icon from scratch is the following:

1. Browse Icon Packs and click on an icon pack that you want to use
2. The directory of the icon pack is being created in the .obsidian directory (by default)
3. Zip file is getting downloaded from the provided CDN's (or general servers) from the icon pack providers
4. Zip file is getting read -> all the individual files in this file are also getting read
5. A file gets created inside the .obsidian directory with its file content

A new proposal could look like this:

1. Browse Icon Packs and click on an icon pack that you want to use
2. The directory of the icon pack is being created in the .obsidian directory (by default)
   -> Also, possibly downloading the Zip file directly into the .obsidian directory.
   Open question: does obsidian sync handle big zip files with ease or do they have struggles there as well?

When the user tries to change an icon in any way (modal for changing the icon gets opened):

1. Already used icons are loaded first
2. Zip file is getting read with all the files and the contents and shows this inside the modal -> can take some time, might be a bad user experience
3. When the user selects a new icon, the file gets "copied" directly inside the .obsidian directory to the icon pack, so that it's getting easier the next time, when you want to choose that icon
4. Optional: when the user does not use that icon within n days, the file is getting automatically removed from the .obsidian directory.

[leonn3k]

From my own experience, Obsidian sync does not have any issues with syncing the size of files we'd be looking at. If a zip file is completely uncompressed, I expect it would be quite fast to open. As a test, I've re-compressed simple-icons as a zip file by using 7zip with the compression level at 0 (zero).

I'm currently at work on my lunchbreak so I can't do any benchmarking to see if it's worth the hassle speed-wise, but as part of the process of adding icons you could download the zip, read the files, and then add them to a new uncompressed zip (or some other filetype) to make them faster in the future. Additonally, as others have mentioned, loading the zip or the files into memory would make it a lot faster to access, assuming the Obsidian plugin system allows for that.

[fr-gg]

Is there any ETA on when this will be fixed?

[FlorianWoelki]

Is there any ETA on when this will be fixed?

No. I am currently testing the functionality and fixing bugs.
Obviously, because this is a free time project, I can dedicate time to it during my free time.

[FlorianWoelki]

This is now implemented in version 2.2.0