Leex line wrap issue

[archimed-shaman]

Looks like there is a bug with line counting in parsetools. To reproduce it, create test.xrl:

Definitions.
Rules.
[a]+[\n]*= : {token, {first, TokenLine}}.
[a]+ : {token, {second, TokenLine}}.
[\s\r\n\t]+ : skip_token.
Erlang code.

and run:

[archimed@pc leex_test]$ erl
Erlang/OTP 17 [erts-6.1] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V6.1  (abort with ^G)
1> leex:file(test).
{ok,"./test.erl"}
2> c(test).
{ok,test}
3> test:string("aaaa\naa").
{ok,[{second,1},{second,3}],3}

Parser detects three lines instead of two in the input

aaaa
aa

The only cause, I found here, is pushing back an newline:
Note that pushing back a newline will mean the line numbering will no longer be correct.
However, there is no any character back-off.
In my opinion, this issue appears, because the parsing FSM does not rollback the line counter after attempting the first rule "[a]+[\n]*=".
I'm not sure, the patch will not break something else :) But the existing unit tests are passed, so I hope, it would be useful.

Here is the dbg trace. You can notice the moment, when the line counter becomes invalid.

4> dbg:tracer().
{ok,<0.46.0>}
5> dbg:p(all, c).
{ok,[{matched,nonode@nohost,26}]}
6> dbg:tpl(test, dbg:fun2ms(fun(_) -> return_trace() end)).
{ok,[{matched,nonode@nohost,25},{saved,1}]}
7> test:string("aaaa\naa").
(<0.33.0>) call test:string("aaaa\naa")
(<0.33.0>) call test:string("aaaa\naa",1)
(<0.33.0>) call test:string("aaaa\naa",1,"aaaa\naa",[])
(<0.33.0>) call test:yystate()
(<0.33.0>) returned from test:yystate/0 -> 3
(<0.33.0>) call test:yystate(3,"aaaa\naa",1,0,reject,0)
(<0.33.0>) call test:yystate(1,"aaa\naa",1,1,reject,0)
(<0.33.0>) call test:yystate(1,"aa\naa",1,2,1,1)
(<0.33.0>) call test:yystate(1,"a\naa",1,3,1,2)
(<0.33.0>) call test:yystate(1,"\naa",1,4,1,3)
(<0.33.0>) call test:yystate(2,"aa",2,5,1,4)
(<0.33.0>) returned from test:yystate/6 -> {1,4,5,"aa",2,2}
(<0.33.0>) returned from test:yystate/6 -> {1,4,5,"aa",2,2}
(<0.33.0>) returned from test:yystate/6 -> {1,4,5,"aa",2,2}
(<0.33.0>) returned from test:yystate/6 -> {1,4,5,"aa",2,2}
(<0.33.0>) returned from test:yystate/6 -> {1,4,5,"aa",2,2}
(<0.33.0>) returned from test:yystate/6 -> {1,4,5,"aa",2,2}
(<0.33.0>) call test:yysuf("aaaa\naa",4)
(<0.33.0>) returned from test:yysuf/2 -> "\naa"
(<0.33.0>) call test:yyaction(1,4,"aaaa\naa",1)
(<0.33.0>) returned from test:yyaction/4 -> {token,{second,1}}
(<0.33.0>) call test:string_cont("\naa",2,{token,{second,1}},[])
(<0.33.0>) call test:string("\naa",2,"\naa",[{second,1}])
(<0.33.0>) call test:yystate()
(<0.33.0>) returned from test:yystate/0 -> 3
(<0.33.0>) call test:yystate(3,"\naa",2,0,reject,0)
(<0.33.0>) call test:yystate(4,"aa",3,1,reject,0)
(<0.33.0>) returned from test:yystate/6 -> {2,1,"aa",3,4}
(<0.33.0>) returned from test:yystate/6 -> {2,1,"aa",3,4}
(<0.33.0>) call test:yyaction(2,1,"\naa",2)
(<0.33.0>) returned from test:yyaction/4 -> skip_token
(<0.33.0>) call test:string_cont("aa",3,skip_token,[{second,1}])
(<0.33.0>) call test:string("aa",3,"aa",[{second,1}])
(<0.33.0>) call test:yystate()
(<0.33.0>) returned from test:yystate/0 -> 3
(<0.33.0>) call test:yystate(3,"aa",3,0,reject,0)
(<0.33.0>) call test:yystate(1,"a",3,1,reject,0)
(<0.33.0>) call test:yystate(1,[],3,2,1,1)
(<0.33.0>) returned from test:yystate/6 -> {1,2,[],3,1}
(<0.33.0>) returned from test:yystate/6 -> {1,2,[],3,1}
(<0.33.0>) returned from test:yystate/6 -> {1,2,[],3,1}
(<0.33.0>) call test:yyaction(1,2,"aa",3)
(<0.33.0>) returned from test:yyaction/4 -> {token,{second,3}}
(<0.33.0>) call test:string_cont([],3,{token,{second,3}},[{second,1}])
(<0.33.0>) call test:string([],3,[],[{second,3},{second,1}])
(<0.33.0>) call test:yyrev([{second,3},{second,1}])
(<0.33.0>) returned from test:yyrev/1 -> [{second,1},{second,3}]
(<0.33.0>) returned from test:string/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string_cont/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string_cont/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string_cont/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string/4 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string/2 -> {ok,[{second,1},{second,3}],3}
(<0.33.0>) returned from test:string/1 -> {ok,[{second,1},{second,3}],3}
{ok,[{second,1},{second,3}],3}

[OTP-Maintainer]

Patch has passed first testings and has been assigned to be reviewed

---

I am a script, I am not human

---

[archimed-shaman]

I've just noticed, may be the same fix is necessary for token, tokens, and skip_token functions

[nox]

?line is deprecated.

[nox]

Also, please read https://github.com/erlang/otp/wiki/Writing-good-commit-messages.

[archimed-shaman]

The fix for 'token' and 'tokens' functions is ready.
Also, I've updated the previous commit messages.

[OTP-Maintainer]

The summary line of the commit message is too long and/or ends with a "."
Make sure the whole message follows the guidelines here: https://github.com/erlang/otp/wiki/Writing-good-commit-messages.

---

I am a script, I am not human

---

[OTP-Maintainer]

Patch has passed first testings and has been assigned to be reviewed

---

I am a script, I am not human

---