Separate semantic analysis and parsing

[jyn514]

This is basically a rewrite of the entire parser. It is very much a work in progress and all the tests are failing. Additionally, I think I've lost some of the unit tests during the rewrite, I need to find them in the git history. That said, most of the work for the rewrite is done at this point and everything left is just cleanup.

Changes

Most of these could be broken up into separate PRs, except for the main 'separate parsing and analysis' change.

• Instead of having ExprType::{Add(left, right), Sub(left, right), ...}, have ExprType::Binary(BinaryOp, left, right), which makes life much easier for the constant folder and backend. This also makes the code much easier to understand since BinaryOp is now a struct instead of a function.
• Clean up declaration_specifiers a lot. This is now sane and won't randomly miss specifiers if they occur in the wrong order. Cannot be in a separate PR.
• Add benchmark for nested parentheses. rcc now reliably handles 3000+ parentheses in a row when before it had trouble even with 300.
• Implement limited span merging. This merges spans for expressions, statements, and declaration but is not very good at retrieving the original subspans. For example, this will show the whole function declarator as an error:

int f(int i, int j, void);
<stdin>:1:6 error: invalid program: void must be the first and only parameter if specified
int f(int i, int j, void);
     ^^^^^^^^^^^^^^^^^^^^

• Correctly parse qualifiers for pointers and variables (Parse qualifiers correctly #347). Cannot be in a separate PR.
• Remove the unused Type::Bitfield
• Rename the AssignmentToken variants to not look dumb (AddAssign instead of PlusAssign)
• Move most of the scaffolding for typedefs earlier in the parser to avoid having enormous match statements everywhere. This also makes is_decl_specifier much more reliable. The cost of course is that's it's super hacky, but it works very reliably.
• The backend is no longer responsible for desugaring complex assignment (good riddance!). Cannot be in a separate PR.
• Variables are now given a Metadata when they are declared which is reused across scopes. This means the backend no longer has to have any idea of what a scope is, making it easier to do codegen. In particular, there are no more bugs where the frontend's scope is different from the backend's scope.

Action Items

These should be reverted or fixed before merging.

• The lexer now turns identifiers into keywords again. This broke the preprocessor and needs to be reverted, it was mostly for testing.
• I tried making the Location a trait instead of a type. This failed miserably.
• Added derive_more for displays. I only used it in one trivial place, it should either take the place of impl Display for Expr or be removed altogether before this is merged.
• codespan is used only for storing the Files table. This is kind of a waste and I should either switch to codespan-reporting once and for all or write my own Files real quick to remove the unnecessary dependency. I'll make a follow-up PR for this, this one is big enough.
• There is still a lot of commented-out code that needs to be removed.
• The Lexer needs a design decision. Right now it implements Iterator<Item = Result<Token, LexError>> which I like because it reflects its semantic purpose. However, the parser only accepts Iterator<Item = Result<Token, CompileError>>. Either the lexer needs to yield CompileError or there needs to be a wrapper that turns all the LexErrors into the more generic CompileError. I went with a third option: the parser accepts any iterator over Result<Token, E: Into<CompileError>

Issues

Closes

• Separate semantic analysis from parsing #151
• Allow adding const to function parameter in definition #346
• [ICE] fuzzer reached unreachable code #282
• [ICE] infinite recursion on union containing itself #261
• [ICE] crash on &= for struct  #244
• [ICE] panic on %= #214
• Remove constexpr field from Expr #143
• int() should parse as function #141
• Cannot perform pointer arithmetic in complex assignment #121,
• Binary promotion for complex assignment #25
• Add restrict keyword #2

#139 is fixed in the parser but now crashes because cranelift hasn't implemented boolean ops (bytecodealliance/wasmtime#1133)

Addresses #59, but only as a misfeature - it ignores all of the keywords listed there.

Makes a great deal of progress towards #266.

cc @pythondude325

[jyn514]

Not quite sure what I was thinking with the comment here. The error is for void i; and things like that.

[pythongirl325]

Some comments

[pythongirl325]

Should this (and the following) parsing function even be in the analysis module?

[jyn514]

Yes, why wouldn't they be? These are helper functions for other functions in analyze, but I don't see why they would go somewhere else.

[pythongirl325]

I think this loop can be rewritten to be a little clearer, maybe even with iterators.

[pythongirl325]

Can you give this comment some context

[jyn514]

i; (in a scope where i has not previously been declared) declares a new variable called i with type int. This 'feature' was removed in C99 but is still common in real-world code.

http://port70.net/~nsz/c/c99/n1256.html#Forewordp5

[jyn514]

The bit that allows this in C89 is 3.5.2:

• int , signed , signed int , or no type specifiers

See https://stackoverflow.com/questions/26488502/which-section-in-c89-standard-allows-the-implicit-int-rule