on_query_completions fails to return custom completions when some characters are used

[joshgoebel]

Mac build 3083 here.

First lets make a simple completion:

{
    "scope": "source.coffee",
    "completions":[
        { "trigger": "constructor:\twow",
        "contents": "constructor: (${1:@instance}) ->\n\t$0"}
    ]
}

So when I type "c" I see "constructor:" in the completion list. Now lets create a plugin:

class MagicCompletions(sublime_plugin.EventListener):
    def on_query_completions(self, view, prefix, locations):
        return [("colon","i'm another completion")]

No issues... now autocomplete includes both "colon" and "constructor:"... but lets add a few other characters to the trigger:

class MagicCompletions(sublime_plugin.EventListener):
    def on_query_completions(self, view, prefix, locations):
        return [("colon(a, b)","i'm another completion")]

Now auto-complete ONLY shows "colon(a, b)" [which seems to work just fine] but "constructor:" has completely disappeared...

Problem characters seem to be parens, commas, spaces, but it may actually be any non [a-z]... Not sure since I don't have the actual Sublime source to figure out why the heck this might be behaving this way.

[joshgoebel]

It also seems to break completing words from the current file although I swore that was working earlier. The example above is 100% consistently broken for me though.

[FichteFoll]

Can confirm in source.python scope (where is also shows the "class" snippet).

What's also odd is that the API-injected completion is not cleared when I comment the code out and save → reload the file. I have to restart ST to unload it properly.

Word completion is broken for me too while the event listener is active. While usually the word completions do not show up from just typing c they do once I write com. Either way, this does not happen when the listener is loaded.

Everything I know about on_query_completions is available here.

[joshgoebel]

While usually the word completions do not show up from just typing c they do once I write com.

I think that's just an autocomplete preference somewhere.

What's also odd is that the API-injected completion is not cleared when I comment the code out and save → reload the file.

Is that the expected behavior (from ST's perspective)?

I would assume there is just a Python interpreter running in another thread... change the class works because the file is reloaded (redefining the class) but just removing the class - the file is reloaded, but there is no "magic" to tell Python it should undefined the class that was previously defined by the file.

Is ST intended to handle that case?

[FichteFoll]

Yes. When a module is loaded sublime_plugin.py instantiates all EventListeners it finds (subclasses of the class) and assigns it to a new module global named plugins. They are then added to a mapping of all callbacks if they define such a method (e.g. on_query_completions).

When the plugin is then unloaded (which happens before it is reloaded), it accesses the plugins attribute it assigned earler and removes all instantiated (or non-instantiated for commands) classes from the mappings.

[iamfredng]

I have a same issue. When I try to use "sublime.INHIBIT_WORD_COMPLETIONS". the result is different.

When I use "_" to replace "( ) . ,". It works.

[evandrocoan]

@iamfredng, I used a Unicode Character similar to replace them like:

import sublime
import sublime_plugin

class FileNameComplete(sublime_plugin.EventListener):

    def on_query_completions(self, view, prefix, locations):
        completion_string = "auto.complete"

        # replaces dot `.` by a `ꓸ` (Lisu Letter Tone Mya Ti)
        return [[completion_string.replace(".", "ꓸ"), completion_string]]

It works. I tested a little, the bad symbols are:

1. (
2. !
3. ,
4. .
5. |
6. <
7. ;
8. :
9. &
10. *
11. $
12. ...
13. Man, it is everything. I will stop trying to.

Seems better just use ''.join(e for e in completion_string if e.isalnum()):

import sublime
import sublime_plugin

class FileNameComplete(sublime_plugin.EventListener):

    def on_query_completions(self, view, prefix, locations):
        completion_string = "auto.complete"

        # removes all special characters
        return [[''.join(e for e in completion_string if e.isalnum()), completion_string]]

1. https://stackoverflow.com/questions/5843518/ Remove all special characters, punctuation and spaces from string

[randy3k]

this bug is annoying when working with https://github.com/tomv564/LSP and language servers return completions with non-alphanum chars.

[randy3k]

I believe I manage to fix it by temporarily masking word_separators

import sublime
import sublime_plugin

class TestComplete(sublime_plugin.EventListener):

    def on_query_completions(self, view, prefix, locations):
        completion_string = "auto.complete"

        # temporarily masking word_separators
        word_separators = view.settings().get("word_separators")
        view.settings().set("word_separators", "")
        sublime.set_timeout(lambda: view.settings().set("word_separators", word_separators), 0)
        return [(completion_string, completion_string)]

PS: this trick doesn't work with completions which contain spaces.
PS2: it seems that it would break some WORD_COMPLETIONS detection.

[rchl]

Just (re)discovered this when investigating problem with AngularJS package inhibiting completions (angular-ui/AngularJS-sublime-package#88).

Before finding this issue, I've created minimal testcase that returns [('$bbbbb', 'bbbbb completion')] completions from on_query_completions. As already clear, it's the dollar sign that causes inhibit-like-behavior in this case.

Very annoying bug.

[dpronin]

Hi,
I'm experiencing something resembling what you're talking about
Could you please take a look at the issue?
Thank you

[wbond]

#48 and #1061 are both the same issue as this.

However, technically this isn't a bug, but rather the (current) intended behavior. I working on digging in to see what the history of this behavior is, and if we can reasonably change it.

To summarize, completions are first gathered from snippets, then the API, then .sublime-completions files and finally the buffer. If API or .sublime-completions triggers contain non-word characters, it stops looking at subsequent sources for completions. Technically this happen with snippets too, but the way that the snippets are queried and matched, it doesn't end up being a problem in practice.

[gerardroche]

Keep in mind that, to add completion descriptions you have to use the \t (tab character), which is a non-word character. So when a user has any completions with descriptions installed, it causes ST to stop looking at subsequent sources.

Many users will have completions with descriptions and non-word characters installed, so the effectiveness of their completions is not as good as it probably could be.

[wbond]

@gerardroche Actually, we split on \t to get the "real" trigger before any of the other work happens.

[wbond]

We discussed this internally, and came to the conclusion that it won't be viable to change the behavior in the 3.2 dev cycle. From experience, the auto-complete system is rather complex with lots of edge cases, and there are many years of plugins that have worked with the current implementation. Making changes is pretty much guaranteed to break things in subtle ways that may not be detected until a stable release.

My hope is to write up some official docs on the completions and how they interact with snippets and so forth, so that package developers can avoid some of the sharper edges.

[evandrocoan]

However, technically this isn't a bug, but rather the (current) intended behavior.

I would call this a very, very, very, ..., very, annoying intended behavior. Please, change it. If you think you cannot reasonably changed, add a setting to switch between the old behavior to the new behavior.

---

This is one of the things in Sublime Text which really gives me inspiration to abandon Sublime Text and use Visual Studio Code:

1. Crashes like Crashed after black box showed up #2519 which is recently happening on development builds
2. Annoying bugs like this

I did not moved yet because Visual Studio Code has bugs I consider a bigger problems than the Sublime Text ones: microsoft/vscode-textmate#52 TextMate scope selectors: scope exclusion is not implemented

And other minor annoyingly issues:

1. Show tooltip bellow the mouse cursor, instead of above it microsoft/vscode#30797 Show tooltip bellow the mouse cursor, instead of above it
2. Add border options to selectionBackground microsoft/vscode#30793 Add border options to selectionBackground
3. Consolas font rendered narrower than Sublime Text one microsoft/vscode#30794 Consolas font rendered narrower than Sublime Text one
4. And probably much more others I did not find yet.

I just used Visual Studio Code for about a week and tons of issues/things to fix showed up. Then, moving from Sublime Text to Visual Studio Code would be a very big/hard task, because there will be a lot of things to fix. But as Visual Studio Code is open source, someday I may actually get the time to fix them. I am just not sure how may years would take from me, in order to do it.

If the resolution for this issue is won't fix, or Sublime Text keeps crashing, I will definitely have to start moving out from Sublime Text. I will not completely stop using Sublime Text because there are some things I believe Visual Studio Code will never be able to do like:

1. Open a file with 5GB of size. However, it is not every day I do that, and it would be fine just use Sublime Text for that.
2. How much RAM it will use? May be Visual Studio Code use so much RAM for a few views/windows opened, that I cannot use it, i.e., makes it unusable.

Moving or not away from Sublime Text, is a question I recently started asking more often. And it usually stops at "-There is probably more stuff to fix on Visual Studio Code than Sublime Text". But currently my balance whether I should or not go for Visual Studio Code could be more shaped like this:

These bigger black rocks would be Visual Studio Code bugs, and these white rocks would be Sublime Text bugs. Right now, I cannot move from Sublime Text because I definitely do not have the time to deal with these big problems from Visual Studio Code (unless some new version Sublime Text starts heavily crashing, or the current version keeps occasionally crashing and no fixes ever come up). May be six months or a year from now, I will have the time to deal with Visual Studio Code problems, and I will be able to move away from Sublime Text unfixable bugs (on its core features like this issue and crashes/segmentation faults).

I say these things like fixing these bugs on Visual Studio Code would be the happiest thing in the world, but if you pay attention to the picture just above, you will notice several big black rocks (Visual Studio Code rocks) just hidden under the surface. These would be the bugs from Visual Studio Code I do not know yet because I only used it for about 1 week long. All white rocks (Sublime Text rocks) are over the surface because I know all of them, as I have been heavily and daily using Sublime Text for years.

May be after a year using Visual Studio Code, it shows up new big bugs as these ones in Sublime Text, i.e., crashes/segmentation faults or extremely time consuming bug fixes, which if are not fixed, makes Visual Studio Code unusable. Also, I cannot forget about the greatest packages ever created for Sublime Text. Right now I got 230 third-part packages installed on my Sublime Text. Probably there will not be an equivalent version available for all of them on Visual Studio Code, which would probably be an enormously time-consuming task to reimplement all of them over there.

I just hope Sublime Text fix this bug, whether on the version 3.2 or 3.3, but also stop occasionally crashing, because it is also very annoying using an editor you cannot trust to save your stuff because it can crash at any moment and make you lose your work.

[FichteFoll]

@evandrocoan, please try to keep the discussion on-topic. I took the liberty to separate your VSC comparison that is entirely irrelevant to this issue from the relevant complaint.

I'm sure there are reasons for your discomfort and I'm not trying to invalidate them, but an issue about how something behaves in the completions system is definitely not the place to discuss them.

[Resike]

@wbond The problem is that even after the \t split it's not just the trigger but the description also gets put into the completion buffer, and due the fuzzy search this will throw a lot of unwanted hits.
Any chance to only fix the \t to at least use the description part properly? #2046

[gerardroche]

@wbond I remember testing if descriptions were causing the same issue and verified that they were. Unless something has changed. Maybe it was the description content causing the issue, see @Resike's comment e.g. if it had non-word characters like spaces in it.

[deathaxe]

At least plugins can return completions flagged with INHIBIT_WORD_COMPLETIONS or INHIBIT_EXPLICIT_COMPLETIONS to prevent subsequent completions. This is what some of them actually do.

With that said, I don't see a qualified reason why none-word characters should cause ST to stop looking for subsequent completions - at least when returned by a plugin.

As explicit completions within a .sublime-completions file can't define a INHIBIT_WORD_COMPLETIONS flag, this heuristic may help to hide possibly unwanted words captured from the buffer in situations a completion seems to be capable to provide more sophisticated results.

But even then the results from the buffer which match the beginning of explicit completion triggers could be filtered out and the remaining ones could be added to the bottom of the completions list.

Example:

If a sublime-completions file contains {"trigger": "myfunc(a, b)\tfunction", ... } any captured word myfunc from buffer could be removed from the completions results. If the buffer contains myfunc2 which is not contained in a sublime-completions file, it could be added at the end.

[Resike]

@deathaxe The issue here is if you start typing function (aka the description) it will trigger the myfunc trigger from the completion. Maybe it would be good to have a better separator (or a better way to handle) between the trigger and it's description in the sublime-completions files that is not escaped.

[deathaxe]

@Resike You actually describe another issue which is not related with the first post as \t and following descriptions do not stop subsequent completion lookups.

Actually I'd hesitate to exclude descriptions from fussy search. Even though I also find it results in inaccurate results sometimes, the description may contain keywords which are easier to remember then the function name itself. To be able to lookup a function by description is therefore a useful feature sometimes.

[Resike]

@deathaxe It might be usefull on very few completion triggers, but i know that from experience that it makes it competely unusable in some cases. And the snippet files does not include the description in the fuzzy search, so i don't know why the completion files should.
So either way both completion and snippet files should work exactly as the same, or add a Sublime setting to include/exclude the fuzzy search in the description.

[dpronin]

I totally support the former colleague because both 'sublime-completions' and 'snippets' do the same thing just by different ways. So, my expectation is for them to work the same way, or, indeed, provide an option to configure them to work differently or leave them to work as they have so far

[gerardroche]

Keep in mind that, to add completion descriptions you have to use the \t (tab character), which is a non-word character. So when a user has any completions with descriptions installed, it causes ST to stop looking at subsequent sources. -- @gerardroche

@gerardroche Actually, we split on \t to get the "real" trigger before any of the other work happens. -- @wbond

@wbond I've retested this and can reproduce the issue of \t characters blocking completions. It may be split "to get the real trigger", but there must be bug or logic error. I'll show steps to reproduce it:

1. Create a PHP file e.g. test.php

2. fill with this content:

   <?php
   
   buffer_b
   buffer_a
   b_buffer
   a_buffer
   dup_buffer_a
   dup_buffer_b
   dup_buffer_a
   php
   
   

3. create a completions file foo.sublime-completions in your user directory.

   {
       "scope": "source.php",
       "completions": [
           "foo"
       ]
   }
   

4. put your cursor on a line after "php" line 12 and trigger completions Alt+/ on linux. You will get all the buffer completions and the file completion (which is put first in the completion list):

5. Now update the file completion with a description (containing a \t character) and try trigger the completions again, it will auto complete "foo" because all the buffer completions have been blocked:

   {
       "scope": "source.php",
       "completions": [
           {"trigger": "foo\tdescription", "contents": "foo"}
       ]
   }
   

6. Update the file completions with multiple completions so that you see more clearly that the buffer completions are blocked:

   {
       "scope": "source.php",
       "completions": [
           {"trigger": "foo\tdescription", "contents": "foo"},
           {"trigger": "foobar\tdescription", "contents": "foobar"},
       ]
   }
   

Maybe others can confirm this too?

[dpronin]

Guys, are there any updates about this issue?

Thank you

[deathaxe]

Caused by #4901