feature(spoon-kotlin): use the Spoon metamodel to analyze and transform Kotlin code

[monperrus]

Hi all,

We would like to propose a new item in the Spoon roadmap: use the Spoon metamodel to analyze and transform Kotlin code.

This would be useful because there is no major Kotlin analysis and transformation tool, with an API as powerful as Spoon's one (@martinezmatias would need this for his research). The Spoon community would reach a new target, in a super active and growing community.

This would be doable, because the concepts of Kotlin are heavily based on Java, and there is likely little information to add in the metamodel to support Kotlin. We would need to:

• Write KotlinTreeBuilder, based on a Kotlin parser / compiler. We could likely reuse the official one from @JetBrains.
• Write KotlinPrettyPrinter.

And last but not least, this would be a lot of fun!

What do you think?

--Martin (@monperrus) and Nicolas (@petitpre)

[martinezmatias]

Hi @monperrus @petitpre
I will be amazing.
Right now, we are parsing Kotlin ASTs using Detekt which if I am not wrong, also uses the official parser from @JetBrains. We have some results of parsing Kotlin code. As far I know -not expert on Detekt-, the API of Detekt to retrieve AST is not "clear", i.e., one has to write a rule, which then is matched on the AST from the app under analysis.

[monperrus]

Kotlin is the fastest growing language on Github: https://octoverse.github.com/projects#repositories

[monperrus]

found by @martinezmatias

Simple Kotlin Source AST and Syntax Parsing, Editing, and Writing
https://github.com/cretz/kastree

[jlundhol]

In addition to detekt and kastree mentioned in earlier comments, here are two more potential options. In contrast to detekt and kastree, these are not using the PSI module of the official compiler for parsing

kotlinx.ast: Generic AST parsing library for kotlin multiplatform
https://github.com/kotlinx/ast

Kotlin grammar tools: Tokenization and parsing Kotlin code using the ANTLR Kotlin grammar
https://github.com/Kotlin/grammar-tools

[martinezmatias]

Hi @jlundhol
Another tool:

Better-parse: A nice parser combinator library for Kotlin: https://github.com/h0tk3y/better-parse

[monperrus]

FYI:
Jesper Lundholm's thesis.

KtSpoon : Modelling Kotlin by extending Spoon’s Java Metamodel (2021): http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-304429

[breandan]

@jlundhol Can you comment on your parser's support for error recovery? Does it need to perform symbol resolution or it possible to define a source code transformation on an incomplete code snippet? Thank you!

[jlundhol]

Hi @breandan ,
it depends heavily on the Kotlin IR module which will generate IR error nodes for errors such as unresolved references. I didn't implement any handling of error nodes, though. Hence, it won't build a model of code containing errors.

[breandan]

Thank you! From what I have seen, Spoon is able to handle whole-classes, but methods must be enclosed. While some parsers have better support for error recovery, it is unclear what should occur when transforming a partial code snippet, so it makes sense to only consider fully-formed classes. Otherwise I quite like your fluent API design and look forward to using it. Out of the ones I have tried, it seems to be the most reasonable.

[monperrus]

The code now lives on https://github.com/SpoonLabs/KtSpoon/.

Thanks @jlundhol!

[algomaster99]

@monperrus Is that a private repository or is the link incorrect? I am unable to access it.

[monperrus]

made public, thanks @algomaster99

[monperrus]

FTR: Reflekt: a Library for Compile-Time Reflection in Kotlin.

[monperrus]

FTR: https://github.com/fbsamples/kotlin_ast_tools

[rex-structorum]

FYI: kotlin-spec contains an official ANTLR grammar.