Support for escape sequences in String

[Remo]

I've been looking around to find a way to support escape sequences inside my string.

I've got [String] STRING : "\"" "\\"; in my grammar file. It works very well for single and multi line strings.

Sometimes, when I'm only asking for a specific character like a line break, I don't want to have an actual line break in my code, especially because different editors will have different line breaks.

What I wanted in my input is something like \n.
In the long run, I would like to support all of these:
https://learn.microsoft.com/en-us/cpp/c-language/escape-sequences?view=msvc-170

Any pointer where I have to go to find a solution for this?

[b3b00]

@Remo, I don't well understand your issue. What i have guessed is that you want CSLY to automatically replace escaped sequence. So for example replace \n by a (char)10 character, or (char)09 by a tabulation. Is this what want ?
CSLY can not know what is the escaped sequence meaning, it could depend on the defined language. So it's up to the user to replace those sequence with the expected value.

[b3b00]

@Remo, you could use lexer callbacks to replace to replace all escaped sequences before the parse phase. this could be something like (warning not tested)

using sly.lexer;


namespace escape
{
    [CallBacks(typeof(TestCallbacks))]
    public enum CallbackTokens
    {
        [String] STRING
    }


    public class TestCallbacks
    {

        [TokenCallback((int)CallbackTokens.STRING)]
        public static Token<CallbackTokens> EscapeEscapedSequences(Token<CallbackTokens> token)
        {
            string value = token.Value;
            value = value.Replace("\\", "\\\\");
            value = value.Replace("\n", "\\n");
            value = value.Replace("\r", "\\r");
            value = value.Replace("\t", "\\t");
            // ... address all escaped sequences 
            token.SpanValue = new ReadOnlyMemory<char>(value.ToCharArray());
            return token;
        }
    }
}

[Remo]

Thanks for your reply!
Let's say I've got the following input:

Property: "Hello \"There\"\nSecond line"

My lexer:

[Sugar] COLON : ":";
[KeyWord] PROPERTY: "Property";
[String] STRING : "\"" "\\";

And then a simple rule like this;

PROPERTY COLON STRING

What I would like to see in C# is a string like this:

Hello "There"
Second line

What I currently get is: Hello "There"nSecond line

I'm not expecting csly to do the magic for me, but the \n is gone. So basically I would like csly to replace \" with ", but leave all other backslashes in place.

[b3b00]

oh ok I see. when lexing string escape char (here \) is purely removed !
This works well with the string delimiter " but remove usefull information for other escaped sequences like \n.
I will fix this right now and push a new nuget version. stay tuned

[b3b00]

@Remo i've publish CSLY nuget 3.3.0, you can test it. In fact i 've found some other issues fixing yours, so great thank to report.

I've done a simple test with cslyviz - escape So you can have a look. As graphviz automatically substitute \n with a line feed it produces a new line (see below).

if you look at the JSON (on the source tab) you will see the \n

[b3b00]

let me know if it works as expected.

[b3b00]

You can also have a look at my commit fe72161, it embeds a test case that more or less matches what you're trying to do using a lexerCallback

[Remo]

Wonderful, everything is working great! I don't think I would have been able to fix this myself, at least not yet. I still don't understand the code of csly fully, but I'm a happy user for sure.