Fix determination of f-string expression spans

[hauntsaninja]

The original goal here was to get rid of the regex dependency, but it turns out the regex failed many of the test cases added in this PR.

[hauntsaninja]

Here's a basic property test (note that the hypothesis.assume means in practice we're unlikely to get valid strings that are particularly long, unless you start playing around with hypothesis.target):

import ast
import hypothesis
from hypothesis import strategies as st


def check_program(program):
    tree = ast.parse(program)

    print(program)

    str_bits = [c.value for c in tree.body[0].value.values if isinstance(c, ast.Constant)]
    expr_spans = list(iter_fexpr_spans(program))
    starts = [a for a, _ in expr_spans]
    ends = [b for _, b in expr_spans]
    expected_bits = [
        x for i, j in zip([2] + ends, starts + [-1])
        if (x := program[i:j].replace("{{", "{").replace("}}", "}"))
    ]
    assert str_bits == expected_bits, (str_bits, expected_bits)


@hypothesis.given(st.text(alphabet="{}' "))
@hypothesis.settings(max_examples=1000, suppress_health_check=[hypothesis.HealthCheck.filter_too_much])
def test_parse_fany(text):
    try:
        check_program(f'f"{text}"')
    except SyntaxError:
        hypothesis.assume(False)


@hypothesis.given(st.text(alphabet="{}' "))
@hypothesis.settings(max_examples=1000, suppress_health_check=[hypothesis.HealthCheck.filter_too_much])
def test_parse_fexpr(text):
    try:
        check_program(f'f"{{{text}}}"')
    except SyntaxError:
        hypothesis.assume(False)

[JelleZijlstra]

Thanks, this is great!

For the tests could you add test_trans.py with tests like the ones you list in the PR description.

[JelleZijlstra]

Thank you!