AST traceability?

[breandan]

Possibly unrelated, but I am looking for a way to recover the relevant source code corresponding to nodes in a path. Are you aware whether any of the ASTMiner-supported parsers provide an unparser that preserves the original source code formatting or otherwise can be traced back to token offsets in the original file? Thanks.

[SpirinEgor]

Hi!
I didn't get it fully. Do you want to restore source code by a set of paths?

What do you mean under unparser? Restoring source code from the AST? If yes, then better to use ANTLR. It keeps syntax tokens, e.g. parenthesis, so restoring code should be done by traversing the tree in preorder. But this will not restore the original formatting.

If you are interested in token offsets, we are currently working on adding this to all parsers. You can watch progress in the #194 pull request draft. First, it will be supported with JSON AST storage. But we planned to add new path-based storage that will help with storing such information.

[breandan]

Thanks, #194 looks like what we need in the short term. I think what you really want is full support for parsing-unparsing, which allows you to do bidirectional transformations, i.e., structurally modify the AST and preserve the original formatting, e.g., for self-supervised learning and metamorphic testing. There are some languages which support these kinds of source-to-source transformations, but they are not very usable from a practical standpoint. I was looking into OpenRewrite at some point, but it had trouble parsing partial code snippets with error recovery. It would be nice to have a DSL for this at some point.

[breandan]

Although perhaps it would be better to leave source code transformation to a dedicated library. It appears that a Kotlin parser similar to what I was describing was recently implemented by @jlundhol in INRIA/spoon#2554. PSIMiner also appears to implement some kind of transformations, perhaps @SpirinEgor could comment on whether they plan to add support for more general tree transformations?

[SpirinEgor]

We thought about general code transformations. It is a very language-specific operation, so I don't think that in a near future there will be a tool for that with multiple language support.

About PSIMiner -- its transformations may be useful and there are interfaces to support multiple languages. But adding new languages, especially not JVM seems to be hard work.

I also may suggest these tools:

• Gorshochek -- A tool for C++ code modification to augment data for clone detection tools
• AST transformations -- About
  An IntelliJ-based IDE plugin for Python AST transformations