Champion "Replace/original and code generation extensions"

[gafter]

• Proposal added
• Discussed in LDM
• Decision in LDM
• Finalized (done, rejected, inactive)
• Spec'ed

See also

• [Proposal] add supersede modifier to enable more tool generated code scenarios. roslyn#5292 [Proposal] enable code generating extensions to the compiler roslyn#5561
• An implementation branch is at https://github.com/dotnet/roslyn/tree/features/source-generators
• Use-cases by @alrz: Discussion: Code Generator Catalog
• Spec: https://github.com/dotnet/roslyn/blob/master/docs/features/generators.md

[MgSam]

Spec: https://github.com/dotnet/roslyn/blob/master/docs/features/generators.md

[MgSam]

@mattwar Are you guys still actively working on this feature or has it been tabled for now due to the implementation complexity?

[asdfgasdfsafgsdfa]

No news on this?
Would be really helpful for a ton of things...

[MgSam]

I just read @MadsTorgersen 's design notes that said this feature is blocked from somewhere else not on the language end.

Could you guys please elaborate? Is the VS tooling the roadblock here now? In my view, this feature is far and away the most useful one out of the entire backlog for C#.

I know the brass likes to dedicate resources to issues that community puts a lot of votes on, so I've made a UserVoice issue as well. Please vote for this feature here and on UserVoice!

[CyrusNajmabadi]

Yes. VS tooling is the roadblock. This work will require substantial resources to get through the tooling side in any sort of sensible and efficient manner. It requires coordination between many different teams as well. As such, while still super great and valuable, it's unclear what the right timeline/delivery avenue would be for it.

[asdfgasdfsafgsdfa]

Sorry if this is a dumb question but what exactly does tooling mean here?
The "debugger UI" in Visual Studio?
What parts would need to get changed?
Isn't this just a case of simply generating different sequence points in the pdb or am I completely off the track here?

Sounds like there's nothing the community can do, or is there?

[CyrusNajmabadi]

but what exactly does tooling mean here?

Everything about the experience in a product like VS. That includes, but is not limited to:

1. IntelliSense. Clearly people will want to generate items that then show up in IntelliSense. How does that happen efficiently?
2. Navigation. People will want to navigate to generated code. How do we do that, and what's the right experience overall?
3. Debugging. How do you debug through the code that is generated.
4. Project-System. How do you present generated code in the project system?
5. Refactoring. How do refactorings work properly in a context where some of the code they're analyzing was generated by previous code, but the changes they make may cause more transformations to happen?

etc. etc.

[iam3yal]

@CyrusNajmabadi Just for clarification it won't be strictly specific to Visual Studio, right?

[CyrusNajmabadi]

I'm not sure i understand your question @eyalsk . Can you clarify?

[iam3yal]

@CyrusNajmabadi I'm asking whether these features you mentioned like IntelliSense, Navigation, Debugging are going to be available in say Visual Studio Code or any other editor? as opposed to being available in Visual Studio only.

[CyrusNajmabadi]

The owners of those respective products would have to make a decision on that. :)

[iam3yal]

@CyrusNajmabadi Got'cha, thanks. 😉

[MgSam]

Why couldn't you guys just add the feature to the compiler and the IDE support will come whenever it comes? This is certainly the strategy TypeScript takes, where language/compiler features sometimes don't get VS support for months/years after they're introduced.

[CyrusNajmabadi]

Why couldn't you guys just add the feature to the compiler and the IDE support will come whenever it comes?

Because the IDE experience won't meet the user experience bar we've set for language changes.

Plus, it will just be a terrible experience. Think about it this way, if there's no IDE support, and you have a code generator that generates symbols that you try to reference, then you'll get tons of error squiggles that never go away as you're trying to use the IDE.

[MgSam]

What doesn't make sense is the arbitrarily different bar for different MS products:

• T4 has shipped with VS forever and IDE support for it is awful. Yet it is widely used both in and outside of Microsoft.
• As I said before, TypeScript regularly ships features with zero VS support until an indeterminate time later. VS isn't even the lead platform.

Basically, my point is C# language evolution shouldn't be constantly hamstrung by Microsoft's internal politics. The language team should be able to add features and let IDE support catch up (as it should be). That's how nearly every other open source language works.

Put the feature behind an opt-in flag until you're comfortable the VS support is up to snuff. In the meantime, JetBrains will probably have R# and Rider fully supporting within 3 months. If the feature is forced to wait for the stars to align inside Microsoft, it'll likely be a minimum 5-10 years before it actually ships.

[AArnott]

I'm stoked about the C# 9 built-in feature. I'd love to try a prototype.

[HamedFathi]

About the source generator proposal, Can it brings Rust`s macro and procedural macro as a new game-changer feature into C#?

https://doc.rust-lang.org/rust-by-example/macros.html
https://medium.com/@phoomparin/a-beginners-guide-to-rust-macros-5c75594498f1

Example: Implementing Ruby’s HashMap syntax in Rust

(Procedural) Macros in Rust is so powerful, We can generate source codes with custom DSLs, syntax and much more.
Any idea?

[CyrusNajmabadi]

Custom DSLs are very intentionally not in scope for this.

[Pzixel]

@HamedFathi rust macros are known for poor IDE integration, especially procedural ones. I don't think it worth taking them as they are even if LDM was interested in it, which they aren't.

[bugproof]

I really like how macros work in Rust. So things like Serde (mapping to struct directly ) are possible without using reflection.

This would change completely the way you can create your serializer and it would improve performance a lot. You could generate your mapping/serialization methods.

Now the workarounds are - reflection caching, expression trees(but that requires compiling during runtime) or using some external tool for generation. Or using some kind of AOP like Fody , PostSharp... but it adds a dependency.

@Pzixel there is always room for improvement and plugins are constantly evolving.

[Pzixel]

@bugproof rust macros are bad for two reasons

1. they don't allow you to intoinspect types, they work on text level. So you can't write something like #[derive_via(MyTrait, Field(my_delegating_field)] that work for any trait, because you can't introinspect MyTrait's members
2. they don't integrate well with IDE and they force recompile everything, because they are native. If they could be compiled into some IR then it could be much better. I discussed this matter with intellij-rust author so I think he's quite aware about macros drawbacks and ways of improvement.

I'd like to have macro system in C#, but LDM don't think so. This is why I probably will switch to scala when next major version comes. Partial evaluation, HKT, functional paradigm - all the nice stuff.

MS has different priorities for C#, sadly.

[CyrusNajmabadi]

@Pzixel @bugproof please move this discussion somewhere else. It is offtopic for this issue.

[charlesroddie]

https://github.com/dotnet/roslyn/blob/master/docs/features/source-generators.md
Generators produce one or more strings that represent C# source code to be added to the compilation.

It's great that this is being worked on, but I wonder what is C#-specific about this? Isn't there a way of making this language-independent, in the sense that:

1. Source generators can be written in any .Net language if the appropriate interfaces to Roslyn are made. Generative F# type providers generate .Net classes from literals or external sources, and a number of them could be converted into source generators for wider user in .Net. (cc @cartermp )
2. Source generators can be used by any .Net lanuage which has made the necessary connections. If they generated IL it would be a more level playing field, and if they have to generate C# then as long as the C# is not somehow inside the referencing project then it might still work.

[CyrusNajmabadi]

It's greate that this is being worked on, but I wonder what is C#-specific about this? Isn't there a way of making this language-independent, in the sense that:

Source generators can be written in any .Net language if the appropriate interfaces to Roslyn are made. Generative F# type providers generate .Net classes from literals or external sources, and a number of them could be converted into source generators for wider user in .Net. (cc @cartermp )

Isn't this just a description of IL... ?

If you want a non-C#-specific, language-independent, way to pass information into a compilation... isn't that just a dll?

Source generators can be written in any .Net language if the appropriate interfaces to Roslyn are made

Isn't that IL?

Basically, why would we need a special feature for this since it's already the fully supported scenario?

[333fred]

but I wonder what is C#-specific about this?

@charlesroddie to be clear, this proposal is not being worked on. This proposal is replace/original, which adds new keywords to the language to support metaprogramming and AOP. The proposal being worked on is purely additive, and as you said, is not C# specific. It's so not C# specific, in fact, that it is not tied to C# 9, and we are considering it a compiler feature, not a language feature.

[Jose10go]

I think #45648 is a good middle-way proposal between this proposal (original/replace) and the Source Generators proposal being implemented for C# 9.(https://github.com/dotnet/roslyn/issues/45648)

[sighoya]

I like source generators, but I'm not used to insert code over strings, that's ugly.

Maybe some limited form of dlang templates + String to macro expansion:

static foreach(String type in ["class","struct"])
    static foreach(String fieldName in ["a","b"])
         static foreach(String fieldType in ["Int,","Float"])
             $type {
              public $FieldType $fieldName;
             }

Generating 2^3 types. In contrast to Source generators, we only put those code fragments in strings which are changing and expand them by $ to valid C# code.
Or more safely, we could introduce new types for block kws, general identifiers and type names which can be checked against the provided strings by the user, s.t. only the "unknown" string will be red marked by the compiler/IDE.

static foreach(BlockKwString type in ["class","struct","unknown"])
    static foreach(IdentifierString fieldName in ["a","b"])
         static foreach(TypeString fieldType in ["Int,","Float"])
             $type {
              public $FieldType $fieldName;
             }

Also, mixins for code insertion would also be beneficial.

[gfraiteur]

We are maintaining a private fork of Roslyn that allow for code rewriting. It is used in our product Metalama. We are considering making this fork open source, but given the additional work it requires, we would like to probe the level of interest first.

Please vote and contribute to this GitHub discussion and detail the use cases you have in mind.

The principal interface is the ISourceTransformer interface and it allows for a complete replacement of the Compilation object.