Inefficient

[jasmeetchadha]

It uses a million tokens everytime it connects to OpenAI.... can you do something about this?

Either reduce the token request payload for updates? or give us a control for setting it to update once a month or so... ?

Thanks

[brianpetro]

The plugin is designed to be efficient. While there are plenty of ways it can be improved further, you must be encountering a bug somewhere if you are using a million tokens each startup.

Are you using Linux? It sounds like your Embeddings may not be being saved to the disk. And there have been reports of this happening on Linux systems.

If not, check out this comment for more information #32 (comment)

[jasmeetchadha]

Thank you for the response.

First, I have to apologise and thank you for the plug-in. It’s thanks to you that I have this function and perhaps will have something else.

Second, no I don’t use Linux and have maybe 1000 notes. Not too long. When I check the embeddings.Json in the folder it doesn’t have anything in it. Not sure if that’s the issue but equally not sure why the embedding file is not getting saved.

I host on Dropbox and have all files available offline too.

[brianpetro]

No apologies are necessary! But thank you anyway. I know it's frustrating when software doesn't work, and I appreciate you taking the time to help me solve this issue. It helps both me and others who aren't willing to report the issues they encounter like this.

I might enable another storage method, IndexedDB, and make it a toggleable option in the settings. This might help with persisting on one device, but I'm not sure about the syncing between devices. Not sure about the ETA for this, though.

In the meantime, if you see any errors on the console log, please take a screenshot of the error so I can look into it. This may or may not be occurring. It depends on where the error is regarding clearing the file contents. For example, dropbox may be "updating" the file with the blank version for some reason, which wouldn't throw an error.

I'll be thinking about this. Thanks for your help in figuring this out!

[Mearman]

I have 2000 notes, I think I may be having a similar issue as I managed to burn through my $20 API budget before I switch to paths only

[brianpetro]

@Mearman That amount seems too high unless you average >15 pages per note. So you must be encountering a bug somewhere.

The log_render_files setting will display which notes get embedded in the console. Please be on the lookout for notes that constantly appear on this list whenever you open a new note. This is evidence that the note isn't being embedded correctly or is failing to persist.

Also, check that your .smart-connections/embeddings-2.json file contains a good amount of data. Mine is ~150MB for ~1,500 notes.

PS- thanks for #39

[jasmeetchadha]

Unsure why but my .smart connections/embeddings.json is 1kb and empty inside.

[brianpetro]

@jasmeetchadha so that's the bug.

If the Embeddings aren't saving then they re-embed every time you open Obsidian.

Which operating system are you using?

[jasmeetchadha]

Windows 11

[brianpetro]

@jasmeetchadha are you using any syncing between devices?

Thanks for helping me figure this out.

[jasmeetchadha]

It’s hosted on Dropbox but I mostly use it from the same device

[brianpetro]

Since you mostly use it from the same device, can you try to exclude the .smart-connections/ folder from the syncing?

This way we can see if that's what's preventing the file from saving correctly.

[jasmeetchadha]

Just did that and running smart connections again. Let’s see how this goes

[jasmeetchadha]

The openai requests were sent, I can see some connections being made, but the embeddings.json file remains 1kb and empty.

Also I notice... nothing got written at embeddings.json (since 1407hrs) even though I ran the sync at 1425. Somehow the plugin isnt writing on that file

[brianpetro]

@jasmeetchadha thank you for the detailed report. I'm going to have to think about this.

There are some other storage options, to start, IndexedDB, that may be accessible, but I haven't tested any of them on the scale of an entire vault. But I've been thinking about adding this alternative storage as a setting for people in your situation where the file isn't saving.

Thanks again for your help in solving this issue and hopefully we will have everything up and running for you soon!

[brianpetro]

I'm closing this since it's a duplicate of #45

Since version 1.2.1, there is now a button in the settings to test if writing files is working. It should return an error that we can use to debug this further if it fails. Please follow up using the newer issue above with any errors. Thanks for your help in solving this issue!

[jasmeetchadha]

Thank you. I will test again and open a new issue if there is any error

appreciate your help