Never exit the main loop in interactive mode.

[tjohnman]

If the end of stream token mark is found, when in interactive mode, ask for user input instead of exiting the main loop.

In case of running out of token budget, reset it and ask for user input.

With these changes, embd can end up empty and cause a crash in the next iteration of the loop, so we check for its size as well.

[rabidcopy]

So if I'm understanding this right, this PR handles both the end of text token AND the exit that occurs when you run out of tokens/context? Where exactly is the "memory" left after that? Does it reset and behave like you just started your initial prompt again with no user input/response history or? Sorry if this is obvious. Edit: Very useful nonetheless. Seems to work. Beats having to reload the model and have the initial prompt parsed again.

[tjohnman]

So if I'm understanding this right, this PR handles both the end of text token AND the exit that occurs when you run out of tokens/context? Where exactly is the "memory" left after that? Does it reset and behave like you just started your initial prompt again with no user input/response history or? Sorry if this is obvious. Edit: Very useful nonetheless. Seems to work. Beats having to reload the model and have the initial prompt parsed again.

It shouldn’t affect the memory. It just resets the counter that holds how many tokens it can generate before reaching the maximum specified in the parameters.

However, I did a mess with the commits. Perhaps this pull request should be rejected and done properly again. I don’t know if the mess can be fixed manually now.

I have little experience with pull requests, sorry.

[tjohnman]

Yep. I messed up and included changes from other stuff I was working on.

[rabidcopy]

This is a bit sloppy and hacked together and the embd.back() = 13; last_n_tokens.back() = 13; part may be unnecessary or even harmful at best. Mostly just a personal tweak that to ignore end of text tokens and running out of tokens while continuing generation. Posting for myself in the future.

        if (params.interactive) {
            if (embd.size() && embd.back() == 2) {
                fprintf(stderr, " [end of text]\n");
//                is_interacting = true;
//                embd.back() = 13;
//                last_n_tokens.back() = 13;
            }
            if (remaining_tokens == 0) {
                fprintf(stderr, " [0 tokens remaining]\n");
                remaining_tokens = params.n_predict;
//                is_interacting = true;
//                embd.back() = 13;
//                last_n_tokens.back() = 13;
            }
        } else {
            // end of text token
            if (embd.size() && embd.back() == 2) {
                fprintf(stderr, " [end of text]\n");
                break;
            }
        }

[tjohnman]

Made a proper pull request with just the necessary changes.
#298