Investigate Tab completion reliability

[fbricon]

I found that tabCompletion was not reliable with Granite models, eg. I could see some Completion: result being returned from the Continue LLM Prompt/Completion output view, but not proposed as tab completion.

We may need to fine tune tabAutocompleteOptions, see https://docs.continue.dev/customize/deep-dives/autocomplete#configuration-options and/or there might be some integration issue with Granite, in upstream Continue code, that'd need some love.

[adietish]

I can confirm this is several ways:

• I dont get (inline) suggestions nor autocompletions at all.
• At times, there are no (model-)queries in the log.

When model queries happen there's no suggestion in the editor

There are instructions for the case where you dont have these: https://docs.continue.dev/customize/deep-dives/autocomplete#im-not-seeing-any-completions
It didnt help for me.

[adietish]

I now have code suggestions coming up when using granite-code:3b. Turns out that it strongly depends on where I am at in the editor and what I do:
they come up if I start a new line (hit Return) at a certain position in my code, they then dont come up in the next line.

[jamescho72]

Please try again with
granite-dense:2b
or
granite-moe:3b

The tags fp16 is also faster response and may help.

[adietish]

Hi @jamescho72
I tried it with granite-dense:2b. Autocompletes are still as unreliable. The model is much faster though.

[adietish]

Afaics the majority of cases where no suggestions are shown are caused by suggestion filtering in continue. I stumbled upon other cases, too but couldn't replicate them though. More digging is needed.

A suggestion returned by the model is ignored if it is repeating previous or following lines:

• https://github.com/continuedev/continue/blob/8d968e324c5ad97255697097ef2cc8f5426f7186/core/autocomplete/filtering/streamTransforms/lineStream.ts#L244-L247:

    if (lineIsRepeated(nextLine, trimmedLine)) {
      fullStop();
      break;
    }

• https://github.com/continuedev/continue/blob/8d968e324c5ad97255697097ef2cc8f5426f7186/core/autocomplete/filtering/streamTransforms/lineStream.ts#L505-L510

    if (line === previousLine) {
      repeatCount++;
      if (repeatCount === MAX_REPEATS) {
        fullStop();
        return;
      }

Screen.Recording.2024-11-25.at.15.58.14.mov

[deboer-tim]

Fred and I aren't able to get consistent results out of 2b either, I would use granite3-dense:8b for tab completion.

[adietish]

Fred and I aren't able to get consistent results out of 2b either, I would use granite3-dense:8b for tab completion.

According to my findings in code, the quality of the model (the suggestion provided by the model to be more precise) has an effect. I'll switch to 8b then.

[adietish]

I'm currently trying to figure why a suggestion provided by granite isn't displayed:

Steps:

1. ASSERT: edit the following javascript in an editor:

class Calculator {
  constructor() {
    this.result = 0;
  }

  add(number) {
    this.result += number;
    return this;
  }

  subtract(number) {
    this.result -= number;
    return this;
  }

  multiply(number) {
    this.result *= number;
    return this;
  }

  divide(number) {
    if (number === 0) {
      throw new Error("Cannot divide by zero");
    }
    this.result /= number;
    return this;
  }

  getResult() {
    return this.result;
  }

  reset() {
    this.result = 0;
    return this;
  }
}

1. EXEC: insert a new line after divide(number) {
2. ASSERT: the following is suggested in the new line: this.result /= number
3. EXEC: start typing thr

Result:

Screen.Recording.2024-11-28.at.15.43.09.mov

The suggestion disappears as soon as you hit r. If you have a breakpoint at the end of the methodBaseLLM#streamComplete you'll see the model suggesting:

throw new Error("Cannot divide by zero");

But this suggestion isn't displayed. It's cached though and if you get back to the prior line the suggestion is displayed there.

[adietish]

PR continuedev/continue#3128 fixed the problem described at #110 (comment)

[adietish]

To me, latest main branch behaves quite consistently. I didn't spot major flaws lately:

Screen.Recording.2024-12-03.at.15.22.17.mov