Unexpected Mode Result

[bdke]

Info

• Version: 3.5.0-alpha6
• OS: Windows 10

Lexer Enum

[Sugar("\"")]
[Push("defaultString")]
StartQuote,
[Sugar("\"")]
[Mode("defaultString")]
[Pop]
EndQuote,
[UpTo("\"")]
[Mode("defaultString")]
StringValue,
[Sugar(";")]
End,
[AlphaNumDashId]
Identifier,
[Sugar("=")]
Assign

Parser Rule

String Rule: STRING: StartQuote StringValue* EndQuote
Statement: statement: Identifier Assign STRING End
Program: program: statement*

Content

test = "3 3";

Output Result

Parser Error: unexpected EOF ('3 3 (line 0, column 8)'). Expecting StringValue, EndQuote, .
Tokens:

Identifier [test] @line 0, column 0 on channel 0
Assign [=] @line 0, column 5 on channel 0
StartQuote ["] @line 0, column 7 on channel 0
StringValue [3 3] @line 0, column 8 on channel 0
EndQuote ["] @line 0, column 11 on channel 0
End [;] @line 0, column 12 on channel 0
<<EOS>>

[b3b00]

Hello @bdke, first question is :have you tried a stable release like 3.4.0 ? An alpha release is... well an alpha release, not fully ok by nature 😉

[b3b00]

I will look at it nevertheless but please report if still KO on 3.4.0

[bdke]

Same result
this time with NullReferenceException when accessing ParseError.ErrorMessage

[bdke]

complete repo is here in case of me misunderstood the WIKI

[b3b00]

Hello, if I understand well your parser you want to capture all that is between quotes as a string. So what don't you simply use the String generic token ?

Right I now there is an issue with 3.4.0. 3.4.1-alpha1 is an alpha but quite stable, 3.4.1 stable release will be published soon. On branch dev (ie 3.4.1-alpha1) parse is ok. I am publishing 3.4.1 so you can test.

See branch https://github.com/b3b00/csly/tree/bugix/%23495 for my unit test.

[b3b00]

version 3.4.1 is now available. Please test with it and tell if OK.

[bdke]

i want Python's f string so i approached the ModeAttribute
Thanks for the help

[bdke]

seems it is still having the same result

[b3b00]

@bdke could you elaborate on "having the same result" ? the same result as with 3.5.0-akpha6 or as with 3.4.0 ?
And if same as 3.5 alpha6 could you tell what you expect ? based on the sample you give on this is issue , not your complete repo please.

[bdke]

my repo has the same result with 3.5.0-alpha6 with 3.4.1 installed
the unit test passed
but not working on my repo
should be my problem now

[b3b00]

Ok, do so now is weekend for me and I will have ver little time to look at it before monday. But I will definitly look at it as soon as I can

[bdke]

thx a lots

[b3b00]

If you have time to make a pull request to csly containing a reproducing test case it would greatly help. You could add it beside my unit test standing your PR brunch FROM the bugfix/495 branch.

[bdke]

Created pull request for reproducible case #496

 [Production("statement : Identifier Assign STRING End")]
 [Production("statement : '$'[d] Identifier Assign STRING End")]
 public string Statement(Token<Issue495Token> id, Token<Issue495Token> assign, string value,Token<Issue495Token> end)

this will create same error as my repo does

[b3b00]

@bdke , i've found a potential explanation. Usings modes, upto tokens and explicit tokens like $ in your rule

    [Production("statement : '$'[d] Identifier Assign STRING End")]

This causes modes to mess and lexer is not being able to scan correctly what's bryween quotes in your lexer. I still have to find a way to fix it. In the mean Time you should be able to circumvent it using a tokens defined in the lexer enum as a sugar token. In facture it seems already defined as DollarSign.

[b3b00]

@bdke , issue fixed 😄 .
it was indeed a compatibilty issue between explicit tokens and generic lexer modes. I have to fix the wiki to mention some limitations : basically when using explicit tokens these are implicitly only on mode default.

version 3.4.2 is available to test.

Thanks for reporting. I will look at modes more accuratly to maybe find other hidden issues.

Close the issue if you're happy with the new version.

[bdke]

it is working now
thanks a lots

[b3b00]

@bdke , I've tried to implements ftrings in IndentedWhile parser. here is what it looks like :

Lexer : (

csly/src/samples/IndentedWhile/parser/IndentedWhileTokenGeneric.cs

Line 99 in c9d0f5c

#region fstring 100 ->

)

Parser :

csly/src/samples/IndentedWhile/parser/IndentedWhileParserGeneric.cs

Line 148 in c9d0f5c

public WhileAST fstring(List<WhileAST> elements)

Doiing so I've found an issue with modes and indents. So i've published v 3.4.3