MEGA-ISSUE: Syntax highlighting in language-python

[savetheclocktower]

IMPORTANT: Some issues have already been fixed!

If you’re still on the regular 1.113 release, you might be suffering from a problem that has already been fixed. Many fixes landed on master in #859. You are encouraged to download the latest rolling release — both (a) to see whether what you’re reporting has been fixed, and (b) so that you can enjoy the fixes that have been reported since the 1.113 release.

---

This will serve as a catch-all issue for any problems you may encounter with syntax highlighting in REPLACE files (language-python). If you have problems with the new syntax highlighting (or folds, or indents) that are specific to .py files, keep reading.

Something isn't highlighting correctly!

If you’ve found any highlighting regressions since 1.113:

First, please scroll down and see if someone else has reported the issue. If so, then you need only sit back and wait for a fix — most issues will be fixed in version 1.114!

If not, please comment with

• A small amount of sample code to reproduce the issue
• Screenshots (before and after would be ideal, but just the “after” is OK for obvious things

I want to go back to the old highlighting!

You can easily opt out of the new Tree-sitter highlighting for any language, but first please:

• subscribe to this issue so you'll know when the problem is fixed
• remove the config change when the next release comes out so that you can enjoy the new grammar once again

To revert to the old Tree-sitter grammar for this language only, add the following to your config.cson:

".python.source":
  core:
    useLegacyTreeSitter: true

To revert to the TextMate-style grammar for this language only, add the following to your config.cson:

".python.source":
  core:
    useTreeSitterParsers: false

[matbme]

Python parser also seems to be out-of-date. The .wasm in language-python package is from 10 months ago, and the tree-sitter-python repo has seen a couple of updates since then.

The not in inside an if statement also doesn't get properly highlighted.
Example:

if tag not in self.all_steps:
    self.all_steps[tag] = 0

Inspector:

if_statement
  condition: comparison_operator
    identifier
    attribute
      object: identifier
      attribute: identifier

[savetheclocktower]

Awesome, thanks. I'll update to the latest tree-sitter-python today and get these fixes in.

[matbme]

break statement also isn't highlighted. Probably missing definition in highlighs.scm, should be alongside continue, which is marked as @keyword.control.repeat._TYPE_.python

[savetheclocktower]

OK, updated to latest tree-sitter-python and applied those fixes. Here's the commit. Thanks! Let me know if you find anything else.

[asiloisad]

The modern tree-sitter auto-indent is broken if multiple cursors are used. Tested in safe mode.

1. a state before
   
2. press Enter
3. state after
   a) legacy tree-sitter
   
   b) modern tree-sitter
   

[Askaholic]

Hello! I'm a long time user of Atom and have been following the pulsar blog for a while. Love to see the amount of work that folks are still putting into this editor! With the release of the modern treesitter implementation I decided to see if it was finally time to start migrating over from my Atom install.

There are a few things that I noticed right off the bat which I'm not necessarily sure are bugs but that nevertheless stood out to me.

1. Variables in assignment statements are colored red. This is confusing to me as I'm used to all occurrences of a variable having the same color. Maybe it's just a matter of getting used to, but it also seems odd to me that special python variables like __file__ and __name__ also get this same color.

2. Code folds on brackets don't show the closing bracket. In atom if you fold on a dictionary or list as shown below, it will render as {...} or [...], however it seems with modern treesitter the closing bracket is dropped {... or [....

Anyway, thanks for all the hard work keeping this editor going!

$ pulsar --version
Pulsar  : 1.114.0
Electron: 12.2.3
Chrome  : 89.0.4389.128
Node    : 14.16.0

Modern Treesitter

Folds

Legacy Treesitter

Folds

[savetheclocktower]

@Askaholic thanks so much for the detailed feedback! Item 1 is an intentional design decision (which I can elaborate on when I get a chance), and item 2 is unintentional (and therefore a bug).

I'd encourage you to give it a week or so to see if you get used to the variable highlighting differences… but if you still hate them then, we can tell you how to use your user stylesheet to get the old highlighting behavior back. Meanwhile, I'll investigate the issue with folds.

Thanks for the report!

[Askaholic]

Here's another issue I noticed with the indentation hinting. It doesn't seem to get class attributes with annotations correct:

Before:

After:

I would expect the cursor to be one level of indentation back from where it is.

Here are other examples that I've tried which have the same result:

class Foo:
    field1: int = 10

class Foo:
    field1: int = 10,

class Foo:
    field1: list[int]

However, these work correctly:

class Foo:
    field1

class Foo:
    field1,

class Foo:
    field1 = 10

[savetheclocktower]

That was dumb of me for assuming that every occurrence of a colon was a hint that the next line should be indented — which I think would be true if not for type annotations. I can update it to enforce that the colon has to occur at the end of the line.

[savetheclocktower]

That was dumb of me for assuming that every occurrence of a colon was a hint that the next line should be indented — which I think would be true if not for type annotations. I can update it to enforce that the colon has to occur at the end of the line.

Almost forgot about this one, but got it into #968. Thanks for the report, @Askaholic!

[asiloisad]

If first two characters are upper cased, then element has support.other.function.python class instead of support.type.constructor.python.

[savetheclocktower]

If first two characters are upper cased, then element has support.other.function.python class instead of support.type.constructor.python.

Easy fix; added to #968. Thanks!

[Askaholic]

Hey, I've been using pulsar for the last few months and its been working great for me! However, I noticed that the visual issue with folds that I described here is still happening on pulsar 1.118.

Thanks for continuing to work on this project!

[savetheclocktower]

Hey, I've been using pulsar for the last few months and its been working great for me! However, I noticed that the visual issue with folds that I described here is still happening on pulsar 1.118.

Thanks for continuing to work on this project!

I should've at least given you the workaround for your first issue (variable assignments not sharing a color with variable usage). You can add this to your user stylesheet:

@import "syntax-variables"; // (this line may already be present)

.syntax--source.syntax--python .syntax--variable {
  color: @syntax-text-color;
}

As for the other issue: it slipped through the cracks, I'm afraid. I'll take a look today and make sure that a fix gets made for 1.119. Thanks for the reminder!

[savetheclocktower]

2. Code folds on brackets don't show the closing bracket. In atom if you fold on a dictionary or list as shown below, it will render as {...} or [...], however it seems with modern treesitter the closing bracket is dropped {... or [....

This is finally fixed in this commit. #1028 should land just before the 1.119 release in mid-July. Thanks for your patience!

[asiloisad]

There is a problem with links if string is more than link. Pulsar v1.122.0

[savetheclocktower]

There is a problem with links if string is more than link. Pulsar v1.122.0

Seems like a problem with my tree-sitter-hyperlink parser. I'll take a look. Thanks!

[savetheclocktower]

@asiloisad, this is fixed in #1118 and will go out next month. Thanks!

[mzivic7]

That was dumb of me for assuming that every occurrence of a colon was a hint that the next line should be indented — which I think would be true if not for type annotations. I can update it to enforce that the colon has to occur at the end of the line.

There are more cases like this:

# wrongly indenting next line:
my_list[3:6]   # list slicing
my_list[::-1]   # list slicing (syntax: string[start:end:step])
if True: print("ok")   # oneliners
for x in my_list: print(x)   # also oneliner, same with while

# these are NOT wrongly indenting:
x = lambda a : print(a)   # lambda functions
my_dict = {"a": 1, "b": 2}   # dicts

So only when colon is at the end of the string, excluding comments, next line should be indented.

[savetheclocktower]

There are more cases like this:

#1172 should fix these. Thanks for the report!