Repeated Group Fails

[abhillman]

The following fails:

use logos_derive::Logos;

#[derive(Logos, Debug, PartialEq)]
pub enum Token {
    #[regex(r"([a-b]+\.)+[a-b]")]
    NestedIdentifier,
}

#[cfg(test)]
mod test {
    use crate::Token;
    use logos::Logos;

    #[test]
    fn t0() {
        let mut lex = Token::lexer("a.b");
        assert_eq!(lex.next(), Some(Ok(Token::NestedIdentifier)));
    }
}

Graph:

{
    1: ::NestedIdentifier,
    2: [a-b] ⇒ 1,
    3: [
        [a-b] ⇒ 5,
        _ ⇒ 2,
    ],
    4: . ⇒ 3,
    5: {
        [a-b] ⇒ 5,
        _ ⇒ 4,
    },
    6: {
        [a-b] ⇒ 6,
        _ ⇒ 4,
    },
    8: {
        [a-b] ⇒ 6,
    },
}

Output:

assertion `left == right` failed
  left: Some(Err(()))
 right: Some(Ok(NestedIdentifier))

Left:  Some(Err(()))
Right: Some(Ok(NestedIdentifier))
<Click to see difference>

thread 'test::t0' panicked at tests/tests/capture_repeat.rs:17:9:
assertion `left == right` failed
  left: Some(Err(()))
 right: Some(Ok(NestedIdentifier))
stack backtrace:
   0: rust_begin_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:298:5
   4: capture_repeat::test::t0
             at ./tests/capture_repeat.rs:17:9
   5: capture_repeat::test::t0::{{closure}}
             at ./tests/capture_repeat.rs:15:12
   6: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Workaround: #[regex(r"[a-b]+(\.[a-b])+")]
Versions: 0.14.0 and 0bcfb6a

[abhillman]

Another similar failure:

use logos_derive::Logos;

#[derive(Logos, Debug, PartialEq)]
pub enum Token {
    #[regex(r"([a-b])+b")]
    ABPlusB,
}

#[cfg(test)]
mod test {
    use crate::Token;
    use logos::Logos;

    #[test]
    fn t0() {
        let mut lex = Token::lexer("ab");
        assert_ne!(lex.next(), Some(Err(())));
    }
}

Graph:

{
    1: ::ABPlusB,
    2: b ⇒ 1,
    3: {
        [a-b] ⇒ 3,
        _ ⇒ 2,
    },
    5: {
        [a-b] ⇒ 3,
    },
}

Output:

assertion `left != right` failed
  left: Some(Err(()))
 right: Some(Err(()))

thread 'test::t0' panicked at tests/tests/capture_repeat.rs:17:9:
assertion `left != right` failed
  left: Some(Err(()))
 right: Some(Err(()))
stack backtrace:
   0: rust_begin_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:298:5
   4: capture_repeat::test::t0
             at ./tests/capture_repeat.rs:17:9
   5: capture_repeat::test::t0::{{closure}}
             at ./tests/capture_repeat.rs:15:12
   6: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[jeertmans]

Hello, thanks for reporting this issue! Could you please give a bit more details: Logos' version, the terminal output, and also the debug graph (see #379 for how to get that graph, this is a work-in-progress PR for the book)?

[abhillman]

Hello, thanks for reporting this issue! Could you please give a bit more details: Logos' version, the terminal output, and also the debug graph (see #379 for how to get that graph, this is a work-in-progress PR for the book)?

Done! Thanks for the tip about graphs. Looking at the second case's graph, it's clear to me why this fails.

Without studying this more, I'm not sure if the design intentions of logos would suggest "the graph is wrong" or "the graph is right [but with a potential warning or error message to the user indicating issues related to ambiguity or priority]." As noted, there is a workaround which can be accomplished by rewriting the regex.

Thoughts, @jeertmans and/or other maintainers?

[jeertmans]

Hello, thanks for reporting this issue! Could you please give a bit more details: Logos' version, the terminal output, and also the debug graph (see #379 for how to get that graph, this is a work-in-progress PR for the book)?

Done! Thanks for the tip about graphs. Looking at the second case's graph, it's clear to me why this fails.

Without studying this more, I'm not sure if the design intentions of logos would suggest "the graph is wrong" or "the graph is right [but with a potential warning or error message to the user indicating issues related to ambiguity or priority]." As noted, there is a workaround which can be accomplished by rewriting the regex.

Thoughts, @jeertmans and/or other maintainers?

Can you please include the graph's output here?

If the graph does not produce the expected behavior, and that it is not documented as a limitation of Logos, then this is definitely a bug :-)