Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix critical oversight in lexer buffer refilling
Since the lexer buffer wraps, the refilling gets handled in two steps: First, iff the buffer would wrap, the buffer is refilled until its end. Then, if more characters are requested, that amount is refilled too. An important detail is that `read()` may not return as many characters as requested; for this reason, the first step checks if its `read()` was "full", and skips the second step otherwise. This is also where a bug lied. After a *lot* of trying, I eventually managed to reproduce the bug on an OpenBSD VM, and after adding a couple of `assert`s in `peekInternal`, this is what happened, starting at line 724: 0. `lexerState->nbChars` is 0, `lexerState->index` is 19; 1. We end up with `target` = 42, and `writeIndex` = 19; 2. 42 + 19 is greater than `LEXER_BUF_SIZE` (= 42), so the `if` is entered; 3. Within the first `readChars`, **`read` only returns 16 bytes**, advancing `writeIndex` to 35 and `target` to 26; 4. Within the second `readChars`, a `read(26)` is issued, overflowing the buffer. The bug should be clear now: **the check at line 750 failed to work!** Why? Because `readChars` modifies `writeIndex`. The fix is simply to cache the number of characters expected, and use that.
- Loading branch information