VRL AST for program reflection and visualization

[binarylogic]

This is something I wanted to put on the map so that we can factor it in as we continue to mature VRL. Especially with #6139 on the horizon.

It is very likely that we'll need a way to visualize a VRL program and build it with a GUI while still letting power users write VRL programs directly (think editor settings where you can switch between a GUI and the "source" JSON). In order to achieve this, I assume we'll need the ability to build an AST from a VRL script.

[binarylogic]

@JeanMertz should this have been closed?

[JeanMertz]

@binarylogic that depends on how far you want to go with this issue.

If the request is "the VRL parser should construct an AST", then yes, we have this now. If you want this AST to be ready to be used in a UI application, then I'm sure there's more work to do, but I don't know yet what that work involves.

[JeanMertz]

To give a bit more context related to the issue description:

The VRL parser (and lexer) construct an AST, which it then feeds to the VRL compiler. This then turns the AST into a runable program.

In the case of the UI, the UI backend itself would generate the AST from direct GraphQL (or similar) queries, and let the compiler generate a valid program from that AST, so the UI would skip the lexing/parsing step, but use the AST types to construct a syntactically valid AST, which the compiler the semantically validates and uses to generate a runnable program.

All of this is there, but I'm sure once we have a second system that wants to construct a valid AST, we'll find some edge-cases that we need to tackle, as usually happens when you go from 1 to n users of a system.

[JeanMertz]

I'm adding this to the backlog. @leebenson has been using the VRL AST a fair bit in his recent work, and while things work, they aren't that ergonomic.

I'll do a pass, see if there are any quick wins, and file issues for anything more involved.

[jszwedko]

I'm going to close this since I don't think we have a concrete use-case at the moment. We can reopen if we find one.