llama: significance of truncating input to ctxLen - 4

[iboB]

Why truncate to ctxLen - 4? Why is that 4 significant.

This is kept for now as per llama.cpp demos, but we should investigate.

[pminev]

I was looking at the commits and PRs:

• Longer and infinite output ggerganov/llama.cpp#71
• ggerganov/llama.cpp@e2d490d
• this was a linked problem: https://github.com/ggerganov/llama.cpp/pull/1789/files,

Finally I talked with G.Gerganov and it was added to secure space (at least 4) in KV cache for the new generated tokens. I didn't ask him further why it's explicitly 4, but it seems like when new tokens are generated the input will be truncated, in order to have enough space again.

[iboB]

this can be closed now. There is a link to this issue in the code for reference