-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
208 changed files
with
6,931 additions
and
20,018 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,179 @@ | ||
| # Changes to the Go Runtime over time | ||
|
|
||
| ## v4.12.0 to v4.12.1 | ||
|
|
||
| Strictly speaking, if ANTLR was a go only project following [SemVer](https://semver.org/) release v4.12.1 would be | ||
| at least a minor version change and arguably a bump to v5. However, we must follow the ANTLR conventions here or the | ||
| release numbers would quickly become confusing. I apologize for being unable to follow the Go release rules absolutely | ||
| to the letter. | ||
|
|
||
| There are a lot of changes and improvements in this release, but only the change of repo holding the runtime code, | ||
| and possibly the removal of interfaces will cause any code changes. There are no breaking changes to the runtime | ||
| interfaces. | ||
|
|
||
| ANTLR Go Maintainer: [Jim Idle](https://github.com/jimidle) - Email: [[email protected]](mailto:[email protected]) | ||
|
|
||
| ### Code Relocation | ||
|
|
||
| For complicated reasons, including not breaking the builds of some users who use a monorepo and eschew modules, as well | ||
| as not making substantial changes to the internal test suite, the Go runtime code will continue to be maintained in | ||
| the main ANTLR4 repo `antlr/antlr4`. If you wish to contribute changes to the Go runtime code, please continue to submit | ||
| PRs to this main repo, against the `dev` branch. | ||
|
|
||
| The code located in the main repo at about the depth of the Mariana Trench, means that the go tools cannot reconcile | ||
| the module correctly. After some debate, it was decided that we would create a dedicated release repo for the Go runtime | ||
| so that it will behave exactly as the Go tooling expects. This repo is auto-maintained and keeps both the dev and master | ||
| branches up to date. | ||
|
|
||
| Henceforth, all future projects using the ANTLR Go runtime, should import as follows: | ||
|
|
||
| ```go | ||
| import ( | ||
| "github.com/antlr4-go/antlr/v4" | ||
| ) | ||
| ``` | ||
|
|
||
| And use the command: | ||
|
|
||
| ```shell | ||
| go get github.com/antlr4-go/antlr | ||
| ``` | ||
|
|
||
| To get the module - `go mod tidy` is probably the best way once imports have been changed. | ||
|
|
||
| Please note that there is no longer any source code kept in the ANTLR repo under `github.com/antlr/antlr4/runtime/Go/antlr`. | ||
| If you are using the code without modules, then sync the code from the new release repo. | ||
|
|
||
| ### Documentation | ||
|
|
||
| Prior to this release, the godocs were essentially unusable as the go doc code was essentially copied without | ||
| change, from teh Java runtime. The godocs are now properly formatted for Go and pkg.dev. | ||
|
|
||
| Please feel free to raise an issue if you find any remaining mistakes. Or submit a PR (remember - not to the new repo). | ||
| It is expected that it might take a few iterations to get the docs 100% squeaky clean. | ||
|
|
||
| ### Removal of Unnecessary Interfaces | ||
|
|
||
| The Go runtime was originally produced as almost a copy of the Java runtime but with go syntax. This meant that everything | ||
| had an interface. There is no need to use interfaces in Go if there is only ever going to be one implementation of | ||
| some struct and its methods. Interfaces cause an extra deference at runtime and are detrimental to performance if you | ||
| are trying to squeeze out every last nanosecond, which some users will be trying to do. | ||
|
|
||
| This is 99% an internal refactoring of the runtime with no outside effects to the user. | ||
|
|
||
| ### Generated Recognizers Return *struct and not Interfaces | ||
|
|
||
| The generated recognizer code generated an interface for the parsers and lexers. As they can only be implemented by the | ||
| generated code, the interfaces were removed. This is possibly the only place you may need to make a code change to | ||
| your driver code. | ||
|
|
||
| If your code looked like this: | ||
|
|
||
| ```go | ||
| var lexer = parser.NewMySqlLexer(nil) | ||
| var p = parser.NewMySqlParser(nil) | ||
| ``` | ||
|
|
||
| Or this: | ||
|
|
||
| ```go | ||
| lexer := parser.NewMySqlLexer(nil) | ||
| p := parser.NewMySqlParser(nil) | ||
| ``` | ||
|
|
||
| Then no changes need to be made. However, fi you predeclared the parser and lexer variables with there type, such as like | ||
| this: | ||
|
|
||
| ```go | ||
| var lexer parser.MySqlLexer | ||
| var p parser.MySqlParser | ||
| // ... | ||
| lexer = parser.NewMySqlLexer(nil) | ||
| p = parser.NewMySqlParser(nil) | ||
| ``` | ||
|
|
||
| You will need to change your variable declarations to pointers (note the introduction of the `*` below. | ||
|
|
||
| ```go | ||
| var lexer *parser.MySqlLexer | ||
| var p *parser.MySqlParser | ||
| // ... | ||
| lexer = parser.NewMySqlLexer(nil) | ||
| p = parser.NewMySqlParser(nil) | ||
| ``` | ||
|
|
||
| This is the only user facing change that I can see. This change though has a very beneficial side effect in that you | ||
| no longer need to cast the interface into a struct so that you can access methods and data within it. Any code you | ||
| had that needed to do that, will be cleaner and faster. | ||
|
|
||
| The performance improvement is worth the change and there was no tidy way for me to avoid it. | ||
|
|
||
| ### Parser Error Recovery Does Not Use Panic | ||
|
|
||
| THe generated parser code was again essentially trying to be Java code in disguise. This meant that every parser rule | ||
| executed a `defer {}` and a `recover()`, even if there wer no outstanding parser errors. Parser errors were issued by | ||
| issuing a `panic()`! | ||
|
|
||
| While some major work has been performed in the go compiler and runtime to make `defer {}` as fast as possible, | ||
| `recover()` is (relatively) slow as it is not meant to be used as a general error mechanism, but to recover from say | ||
| an internal library problem if that problem can be recovered to a known state. | ||
|
|
||
| The generated code now stores a recognition error and a flag in the main parser struct and use `goto` to exit the | ||
| rule instead of a `panic()`. As might be imagined, this is significantly faster through the happy path. It is also | ||
| faster at generating errors. | ||
|
|
||
| The ANTLR runtime tests do check error raising and recovery, but if you find any differences in the error handling | ||
| behavior of your parsers, please raise an issue. | ||
|
|
||
| ### Reduction in use of Pointers | ||
|
|
||
| Certain internal structs, such as interval sets are small and immutable, but were being passed around as pointers | ||
| anyway. These have been change to use copies, and resulted in significant performance increases in some cases. | ||
| There is more work to come in this regard. | ||
|
|
||
| ### ATN Deserialization | ||
|
|
||
| When the ATN and associated structures are deserialized for the first time, there was a bug that caused a needed | ||
| optimization to fail to be executed. This could have a significant performance effect on recognizers that were written | ||
| in a suboptimal way (as in poorly formed grammars). This is now fixed. | ||
|
|
||
| ### Prediction Context Caching was not Working | ||
|
|
||
| This has a massive effect when reusing a parser for a second and subsequent run. The PredictionContextCache merely | ||
| used memory but did not speed up subsequent executions. This is now fixed, and you should see a big difference in | ||
| performance when reusing a parser. This single paragraph does not do this fix justice ;) | ||
|
|
||
| ### Cumulative Performance Improvements | ||
|
|
||
| Though too numerous to mention, there are a lot of small performance improvements, that add up in accumulation. Everything | ||
| from improvements in collection performance to slightly better algorithms or specific non-generic algorithms. | ||
|
|
||
| ### Cumulative Memory Improvements | ||
|
|
||
| The real improvements in memory usage, allocation and garbage collection are saved for the next major release. However, | ||
| if your grammar is well-formed and does not require almost infinite passes using ALL(*), then both memory and performance | ||
| will be improved with this release. | ||
|
|
||
| ### Bug Fixes | ||
|
|
||
| Other small bug fixes have been addressed, such as potential panics in funcs that did not check input parameters. There | ||
| are a lot of bug fixes in this release that most people were probably not aware of. All known bugs are fixed at the | ||
| time of release preparation. | ||
|
|
||
| ### A Note on Poorly Constructed Grammars | ||
|
|
||
| Though I have made some significant strides on improving the performance of poorly formed grammars, those that are | ||
| particularly bad will see much less of an incremental improvement compared to those that are fairly well-formed. | ||
|
|
||
| This is deliberately so in this release as I felt that those people who have put in effort to optimize the form of their | ||
| grammar are looking for performance, where those that have grammars that parser in seconds, tens of seconds or even | ||
| minutes, are presumed to not care about performance. | ||
|
|
||
| A particularly good (or bad) example is the MySQL grammar in the ANTLR grammar repository (apologies to the Author | ||
| if you read this note - this isn't an attack). Although I have improved its runtime performance | ||
| drastically in the Go runtime, it still takes about a minute to parse complex select statements. As it is constructed, | ||
| there are no magic answers. I will look in more detail at improvements for such parsers, such as not freeing any | ||
| memory until the parse is finished (improved 100x in experiments). | ||
|
|
||
| The best advice I can give is to put some effort in to the actual grammar itself. well-formed grammars will potentially | ||
| see some huge improvements with this release. Badly formed grammars, not so much. |
Oops, something went wrong.