Attention Sinks

[lukasfolle]

Not 100% sure if this is still SOTA but do you plan to bring the idea of attention sinks into Tabby?
https://github.com/tomaarsen/attention_sinks?tab=readme-ov-file

I think the idea is really awesome and could help dealing with especially large repos - even combined with treesitter.

[wsxiaoys]

It will somehow help—at least by allowing an infinite output length for the chat model, which could potentially assist with issue #1349.