Consider moving to Roslyn infrastructure

[JoshVarty]

Is there any interest in moving PTVS to Roslyn infrastucture (Workspace, Project, Document etc.)?

The same has been done for F# and TypeScript and it seems to have made it easier to support a wider range of features many users expect in Visual Studio.

Are there any issues that might make this prohibitively difficult or impossible?

[crwilcox]

Can you provide more details on what you are looking to change? Do you have particular documentation for this and an idea of the scale of this work? From the original prompt I am not sure of what this is all involving.

[JoshVarty]

My understanding is that we can take advantage of common infrastructure provided by Roslyn that will make it easier to support a broader range of language service features. I've been using PTVS daily for a few months now and there are some features from other languages that I miss. eg. (More complete QuickInfo, HighlightTag, etc.)

When F# moved to a Roslyn-based project infrastructure, non-Microsoft contributors found it easier to implement language service features.

The scale of the work is large. I would be willing to begin working on this in my spare time in a feature branch.

The initial commit that kicked off this work for F# can be found here.

If we move to Roslyn infrastructure I think we'll have a much easier time reaching and maintaining feature parity with other language services. The APIs Roslyn provides are typically much easier to work with than the Visual Studio APIs themselves.

[JoshVarty]

@OmarTawfik @KevinRansom Maybe you guys can comment on why F# moved to Roslyn infrastructure and any benefits you've seen by doing so.

[int19h]

Can you clarify what you mean by "Roslyn infrastructure"? Is this about the project system, basically?

If yes, does it support filesystem-based projects?

[OmarTawfik]

@JoshVarty Basically, this is a rough road map of the major things needed to be done:

1. Roslyn needs to add PTVS to its IVT list (so that you can reference Roslyn APIs).
2. You start creating a Roslyn workspace and attaching your files/documents to it at solution/project load time.
3. Different services that you implement are MEF exported so that Roslyn picks them up and displays them in IDE.

Integrating with Roslyn was quite fast/straight forward. Most of my time was spent dealing with MEF and trying to debug why something isn't loaded correctly in VS.

See the discussion on dotnet/fsharp#913 for more info

Note that this has nothing to do with the project system. F# project system is still VS code. This has some drawbacks like not being able to use language services (like find all references) cross-projects in the same solution. Work spaces are not tied to the project system. BUT it has been critical in getting more contributes to work on the language services, and we've seen amazing progress there.

/cc @Pilchie @CyrusNajmabadi @jasonmalinowski from Roslyn IDE team can provide info as well
/cc @KevinRansom @brettfo folks working currently on F#

[CyrusNajmabadi]

This is a fairly large undertaking, but the general value here is in being able to share tons of code, and to deliver features more quickly and with more consistency across the VS languages.

For example, if you were a roslyn language, all you would need to implement outlining/structure-guides would be this API:

https://github.com/dotnet/roslyn/blob/master/src/Features/Core/Portable/Structure/BlockStructureService.cs

This is a very simple query + data API. We do the work of managing threading and backing off. And we call into you (in cancellable manner). All you need to do is provide BlockSpans (i.e. data) and we take care of the rest. i.e. we'll diff those results with the previous spans, and let the editor know about only what has changed. We also take care of the UI, making it so that collapsed outlining spans are properly classified:

The idea here is that Roslyn handles the orchestration, and the individual language only needs to work on supplying the data.

Roslyn provides the orchestration for several dozen features, vastly reducing the burden on every language to do that itself.

[CyrusNajmabadi]

A similar feature that you would get very cheaply would be the new FindReferences window. Roslyn has implemented support for this window with lots of cool facilities (including streaming results, highlighted text, filtering, grouping and previews):

To get support for that new window (with all of that difficult stuff taken care of for you), you'd just need to implement:

https://github.com/dotnet/roslyn/blob/master/src/EditorFeatures/Core/FindUsages/IFindUsagesService.cs

This API gives you a IFindUsagesContext which allows you to push results into the window as your language service finds them:

https://github.com/dotnet/roslyn/blob/master/src/EditorFeatures/Core/FindUsages/IFindUsagesContext.cs

If you used roslyn, you'd just have to find the data and push it into the context object. Everything else would be taken care of for you, by us.

--

Another feature that you would essentially get very cheaply would be things like the light-bulb experience. i.e.:

While your language is still responsible for the analysis and determining what changes would be made, all you need to do is take in one of our snapshots and make the changes you want. Our lightbulb system then diffs the snapshots and determines what to show in the preview window.

In order for a language to try to be like C#/VB/TS/JS it must actually reimplement literally tens to hundreds of thousands of lines of code that we've been working on for a long time. It's practically impossible to actually do this, and it's far more effective to just leverage this work across all the langauges and leave the individual languages responsible for only having to understand their own language and then answer data-oriented questions about it.

[CyrusNajmabadi]

Now, moving onto Roslyn is not an easy undertaking. Our infrastructure makes certain assumptions about how a Language Service is implemented. For example, that it avoids the UI thread like the plague. That it can generally operate in a very fine-grained, cancellable manner. That certain types of operations are 'cheap'.

Impedance mismatches here caused much pain for F# when they tried to move over.

--

Also, while the Roslyn user-feature-set APIs are relatively simple and reasonably mature, the APIs around the project system are incredibly baroque and harder to deal with. That's because they're written to interface with all the old IVSHierarchy views of the world, and the ancient csproj/vbproj project systems.

TS had a reasonable time integrating in here. but that's mostly because it was very young and didn't need much work to change things. F# had a harder time being much more established. And the same will likely be true for PTVS.

--

If PTVS wants to do this, it should be because they see enough value in the wide feature-set that Roslyn provides and simplifies. And it should be done with deep discussions between the teams about so both sides understand how the other works and good planning can be made around what work would be necessary.

Moving to Roslyn is not simple or cheap. But it can be very valuable once the initial investments are made.

[int19h]

There's also https://github.com/Microsoft/RTVS, where I assume the same costs and benefits would apply. @jflam, @MikhailArkhipov, take a look.

[zooba]

Thanks for the info on this. We'll consider it, of course, but since what we currently have works well there isn't a huge motivation to migrate. We've also been burned many times in the past by taking dependencies on Roslyn (we're currently shipping a fork of the interactive window because of this, rather than depending on the "built-in" one), so I'd like to see the API proven stable before we commit to it.

So please do keep us in the loop as development continues, and we're happy to look at API proposals to see how well they suit Python (and R), but I don't think we'll be prioritizing any wholesale migration.

[jflam]

One thing that I would really like to see is more code sharing across RTVS and PTVS. Right now development is too siloed. So some questions to consider:

1. How expensive is this change? A swag of dev-months would be helpful.
2. Is there a doc that outlines the benefits of the Roslyn infrastructure that we can read ahead of discussions between the teams?
3. Will it result in more sharing of code between RTVS and PTVS? I consider it a useful side effect if we can also share more code across F# and Roslyn.
4. Does the Roslyn infrastructure help drive more code sharing between VSCode and VS? Perhaps @DustinCampbell could chime in on this.

Thanks!

[DustinCampbell]

Plus @Pilchie and @CyrusNajmabadi as well.

[jflam]

@DustinCampbell, @CyrusNajmabadi wrote a book already in these comments :)

[DustinCampbell]

yes, but all of that is just a technical discussion and you're looking for something more.

We've not documented this because, well, it really isn't recommended (regardless of @JoshVarty's suggestion). There are a lot of caveats with doing this, such as needing IVT to Roslyn because the APIs TypeScript and F# are using are not public. It's true that they get a ton of mileage out of it, but it's not without problems.

The biggest issue we've had is implicitly breaking other teams when new Roslyn bits are inserted into VS due to the fact that Roslyn's internal APIs do churn. The number of times TypeScript has been broken in VS due to a Roslyn insertion is not small. We are very nervous about providing IVT to other languages, and we only did it for F# because, well, they're on our team. 😄

Caveats aside, TypeScript and F# have gotten a lot of mileage in being built using Roslyn as a basis for their VS language service. However, the reality is that the language neutral facilities that these languages are using from Roslyn should be pushed down into the VS platform so that everyone can use them.

Anyway, if you want to talk seriously about doing this beyond just the engineering water cooler, it's a larger conversation with our management. However, note that there is definitely some aversion to doing this because of past (and current) problems.

[jflam]

Thanks for the caveats, particularly the ones around breaking changes around APIs. One of the things that would be very helpful is to adopt a discipline around breaking changes more akin to how Windows deals with them i.e., silently break them at your own peril as you will be yelled at by dev directors across the org. The breaking change notification alias is scrutinized heavily by the entire division because failure to read them means that your being broken is now your fault :)

That said, my suggestion fell mostly upon deaf ears once upon a time.

I do have a chat scheduled with @Pilchie next week. I can bring this up among other things with him then.

Any additional comments about VSCode + VS code sharing? Thanks!

[DustinCampbell]

Any additional comments about VSCode + VS code sharing? Thanks!

Building on Roslyn infrastructure would get you benefits in VS, but it's nothing special for VS Code. The Roslyn API layers that we're talking about here aren't portable and take dependencies on the VS editor.

[jflam]

Got it. Thanks!

[DustinCampbell]

And if you want to turn that chat with @Pilchie into breakfast over in cafe 16, let me know. 😄

[JoshVarty]

I think I was a little too over-eager here.

I hadn't fully considered the implications of taking a dependency on Roslyn including the fact that we'd be another customer taking a dependency on their internals. (Something that understandably makes their development harder)

It also sounds like there isn't a huge motivation from PTVS to move either (everything works well now) so it probably won't be worth the work right now.

There might be a point in the future where we wouldn't need to take an IVT dependency on Roslyn (or the infrastructure was migrated to the VS Platform itself) and it might be worth reconsidering then.

[zooba]

Maybe not over-eager - I'm glad the connection has been made - but we have very high priority items right now that are not helped by this change. We will consider this for our next major feature release, but with 100% of our resources committed to very time-sensitive work it's just not possible right away.