Word exclusions

[tophee]

My default note template includes some data fields (with a double colon at the end), like this:

categories:: 
similar:: 
different:: 
useful for:: 
key points:: 

In short notes, these seem to weigh in heavily in the note's vector (or is it just block similarity) so that the proposed smart connection are not at all smart. They are more or less random notes that also are short and include these fields.

A quick solution for this would be an option to exclude words that end with a double colon.

But the solution could be optimised a little bit because those fields are meaningless only if they are empty, i.e. if there is nothing after the double colon. Once there is something there, it becomes meaningful to distinguish between, say, similar:: cat and different:: cat. If we simply exclude the words with the double colon, the two lines would look identical to the plugin (whether this actually matters is another question, I can see that this may not matter at all, but I don't know).

So a more sophisticated and also more flexible solution would be to allow regex-based exclusions. I totally see how that could generate a lot of support questions, so I would be happy with the simple version.

Oh, and just in case you are wondering why I'm not putting those fields into the YAML frontmatter: it's because I want to be able to use links.

Edit: I just realized that not even the yaml frontmatter is excluded by default (no idea why I thought it was). Being able to exclude that would be great. Or is it already possible via header exclusions? Will try.

[RealRavens]

A general word or phrase exclusion could be good. Same thing happens when you have a functional tag like #MOC for example that may bring up notes with related text and themes which is correct, but I may not want "map of content" type notes, but the notes themselves. So having the option to ignore certain things like this would be useful.

[kuku652]

Same here
YAML and dataview mess up my Smart Connection

[tophee]

I also noticed that not even yaml frontmatter is excluded. It looks like we need block exclusion, both yaml blocks and code-blocks...

As for the dataview fields I mentioned in the OP, I am currently trying to work around the problem by butting these fields under a heading of their own and excluding thst heading in the smart connections settings.

I think the best solution would be regex-based exclsuions. @brianpetro , any chance you would allow users to define regular expressions to define what the plugin should ignore?

[brianpetro]

As for the dataview fields I mentioned in the OP, I am currently trying to work around the problem by butting these fields under a heading of their own and excluding thst heading in the smart connections settings.

Good thinking!

Re: regex: I haven't though about this one much. Might be too complicated for a general audience, but maybe I'll slip it in to a future update.

Regarding Dataview variables and frontmatter variables:

I think it would make sense to blanket-exclude them if they are empty. In my experience, it's been the empty variables that cause the most issues.

Also, I think that specifying which of those variables should be treated with priority or outright ignored would be a big improvement.

Thanks for sharing that workaround!
Brian 🌴

[tophee]

Glad you agree to exclude variables when they're empty.

Another thought: I'm not sure if this makes a big difference, but the nature of those variable is that they can stand in no particular order (and may even be completely unrelated to each other, meaningwise). Yet, as I understand how embeddings are usually generated, variables that stand next to each other will end up in the same chunk that gets vectorized, perhaps together with som bits of text content from the note. As a result, the meaning of both the variables and the vector might be a bad representation of both the variables and the content. My hunch is that it might be better to vectorize each (non-empty) variable separately. But maybe I'm overthinking this. Or maybe I am misunderstanding how smart connections works. What do you think?

The idea of weighing variables had not occurred to me and I'm failing to imagine a use case for such weights (except for 0 vs 1, i.e. excluding or including then). My first reaction would be: if you start attaching weights to stuff, why only to variables? You might as well say that the headings "summary" and "abstract" should have additional weight, but I know too little to be able to judge whether this would even be useful. My gutt feeling is that such weights introduce a risk that the connections the plugin shows become less smart or that serendipity is reduced, because it would seem inevitable that weights will reinforce what we already know, believe or assume...

But apart from this scepticism against more nuanced/elaborate weighing, I think an option to exclude variables (even when they are not empty) makes complete sense, given that a common use case for these variables is to store technical information in them, i.e. information that has nothing to do with the meaning of the note content. You could even exclude some commonly used fields (such as "date created" or "cssclass") by default.

As for regexes: I completely see how regex can be confusing/frustrating for users. How about you use regex behind the scenes when you implement the exclusion of empty fields and if you put that regex into the data.json file in such a way that users can add additional regexes? That way the feature remains invisible to most users but those who want to use it will be able to do so.

[brianpetro]

You're on the right track here.

Most of that is in line with what my plans are, but with a few tweaks.

The data.json file for a potential regex input is also a good idea of the regex is used in a way that makes it compatible.

Thanks for your thoughts on this
Brian 🌴