oss-fuzz 69058: TokenError

[nedbat]

This link seems to be private, so copying details here... https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=69058

Project: coveragepy
Fuzzing Engine: libFuzzer
Fuzz Target: fuzz_parse
Job Type: libfuzzer_asan_coveragepy
Platform Id: linux

Crash Type: Uncaught exception
Crash Address: 
Crash State:
  _removeHandlerRef
  _tokenize
  generate_tokens

This is the claimed stack trace:

 	 === Uncaught Python exception: ===
	TokenError: ('EOF in multi-line string', (2, 0))
	Traceback (most recent call last):
	  File "fuzz_parse.py", line 33, in TestOneInput
	  File "coverage/parser.py", line 265, in parse_source
	  File "coverage/parser.py", line 143, in _raw_parse
	  File "coverage/phystokens.py", line 179, in generate_tokens
	  File "tokenize.py", line 461, in _tokenize
	TokenError: ('EOF in multi-line string', (2, 0))

The provided test case is an 8-byte file:

% hexdump -C /dwn/clusterfuzz-testcase-minimized-fuzz_parse-5820066691088384
00000000  ff 8d a7 dc 0a 27 27 a7                           |.....''.|
00000008

I've tried to reproduce this problem, and cannot:

from coverage.parser import PythonParser
parser = PythonParser(text="\xFF\x8D\xA7\xDC\n''\xA7")
parser.parse_source()

produces:

Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/coverage/parser.py", line 265, in parse_source
    self._ast_root = ast_parse(self.text)
                     ^^^^^^^^^^^^^^^^^^^^
  File "/Users/ned/coverage/trunk/coverage/misc.py", line 381, in ast_parse
    return ast.parse(text)
           ^^^^^^^^^^^^^^^
  File "/usr/local/pyenv/pyenv/versions/3.11.9/lib/python3.11/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<unknown>", line 1
    ÿ�§Ü
     ^
SyntaxError: invalid non-printable character U+008D

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/ned/coverage/trunk/fuzz.py", line 3, in <module>
    parser.parse_source()
  File "/Users/ned/coverage/trunk/coverage/parser.py", line 268, in parse_source
    raise NotPython(
coverage.exceptions.NotPython: Couldn't parse '<code>' as Python source: 'invalid non-printable character U+008D' at line 1

Somehow they have a TokenError, but coverage.py does not. I don't understand how they are getting their error.

[devdanzin]

It's possible to raise an EOF TokenError by decoding the bytes to cp273, but it is caught and results in a NotPython exception. Also, it's about a multi-line statement, while the original error is about a multi-line string. It only works with Python 3.11 or below, 3.12 won't raise a TokenError. Seems to be a coincidence.

from coverage.parser import PythonParser
text = b"\xFF\x8D\xA7\xDC\n''\xA7".decode("cp273")
PythonParser(text).parse_source()

  File "/mnt/c/Users/ddini/PycharmProjects/coveragepy/coverage/parser.py", line 271, in parse_source
    self._raw_parse()
  File "/mnt/c/Users/ddini/PycharmProjects/coveragepy/coverage/parser.py", line 154, in _raw_parse
    tokgen = generate_tokens(self.text)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/c/Users/ddini/PycharmProjects/coveragepy/coverage/phystokens.py", line 179, in generate_tokens
    return list(tokenize.generate_tokens(readline))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/tokenize.py", line 525, in _tokenize
    raise TokenError("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (2, 0))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/c/Users/ddini/PycharmProjects/coveragepy/tok.py", line 22, in <module>
    PythonParser(text).parse_source()
  File "/mnt/c/Users/ddini/PycharmProjects/coveragepy/coverage/parser.py", line 277, in parse_source
    raise NotPython(
coverage.exceptions.NotPython: Couldn't parse '<code>' as Python source: 'EOF in multi-line statement' at line 2

Here's the code I used to try to find matching errors:

from encodings.aliases import aliases
from coverage.parser import PythonParser, NotPython

btext = b"\xFF\x8D\xA7\xDC\n''\xA7"

encs = sorted(set(aliases.values()))
for enc in encs:
    try:
        text = btext.decode(enc)
    except:
        continue
    parser = PythonParser(text)
    try:
        parser.parse_source()
    except NotPython as n:
        print(enc, n)

[nedbat]

@devdanzin said on Mastodon:

Maybe the minimized test case is a seed for the fuzzer, instead of direct input?

"A minimized testcase, which is a fuzzer input that can be used to reproduce the bug."

pigweed.dev/pw_fuzzer/guides/reproducing_oss_fuzz_bugs.html

[nedbat]

This is fixed in commit 364282e

[nedbat]

This is now released as part of coverage 7.5.2.