How to take forward the prototype of a Visual F# editor based on Roslyn

[KevinRansom]

---

Please Note: in order to satisfy community curiosity about this exciting set of new features, we are providing this snapshot of the current state of our planning. There are more details for us to figure out yet. You should expect to see us working on this in a few weeks.

Srivatsn worked over Christmas and did this:

This is F# colorized and with a lightbulb fixer implemented at the Roslyn workspace layer in Visual Studio. It uses Roslyn to handle the editing and the FSharp Language Service to parse the code. As we all know this model works and is well understood, TypeScript uses it for their editor experience. Currently not all of the APIs necessary are public, some are a bit ugly and need redesign, for now we should get an internals visible added to roslyn and allow the roslyn team to evolve a decent public Api for this based on our usage scenarios.

He maintains a branch here: https://github.com/srivatsn/visualfsharp/tree/roslynls By now it is based on quite an old branch fsharp compiler branch 303 commits behind (October 30th 2015)

I believe we all agree that scrapping our existing IDE experience and building on top of this mechanism is the best chance for Visual F# to try to keep up with the huge pace of IDE enhancements that Visual C# and VB are currently achieving. Also the IDE and fixer model will allow us to vastly improve the Visual F# error experience, because often the error isn’t at the line of code the compiler says, so we can create fixers that offer 4 or 5 solutions with preview to solve an error. The only real thing for discussion is how and when to take this forward. Building the editor gives us a solid foundation for creating an IDE experience that will work cross platform

We propose to:

• We add Sri’s prototype into the current master branch which targets DEV15
• We add a switch to the Visual F# IDE settings that enables this language service and disables the existing Visual F# language service, that will allow Visual F# early adopters and OSS devotees to try out the new IDE experience without losing all of their existing functionality
• We build the tooling on the newly integrated F# CompilerService APIs rather than the old FSharp.Compiler.LanguageService APIs.
• Priority is get to parity with existing IDE / language service
• To get to editor parity is probably not a lot of effort, the existing Visual F# IDE relies a lot on default editor behavior

Parity consists of:

• Colourization
• Intellisense
• Build integration
• Debugger integration
• Error reporting to error window
• Execute in FSI
• Brace matching
• Smart indent is currently wrong for the Roslyn prototype
• Encourage OSS contributions

Powertools

The existing Visual F# Power Tools may not work correctly in this new language service. We will encourage the powertools contributors to migrate their implementation to this new editor. Ideally they could contribute the implementation of many of the higher order features

Roslyn Features

Identify easily separable IDE features that already exist in Roslyn, and open up for grabs issues on fsharp repo to implement them, with pointers to existing Roslyn implementations. The rework will often be required because our AST is completely different from the Roslyn one.

• A ton of new tests will need to be created to ensure that experience is correct
• _Consider …. Implementing Editor TestApp support for Visual F# _
• This needs to be sharable at least in binary form with OSS community
• Once we reach parity, remove the switch and the Legacy Visual F# editor

Note:

1. The initial work is to make language service switchable, this will allow Dev 15 previewers to continue to use current IDE code including the Visual F# Powertools, while also giving the ability to use the new one.
2. The Visual F# PowerTools features will likely require porting to the new Roslynized language service (it is not yet known if they can coexist). Ideally they would be submitted as feature PR's but doing them as in add-in is a possibility too and so successfully working with the community to figure out the best way to package and work on these features is key to this being successful.
3. The F# compiler service APIs have been integrated it into the Visual F# build which gives us a solid API to get information about the code.
4. This approach may lead us towards a vscode/omnisharp integration strategy too, though projects such as FsAutoComplete also provide multi-editor integration
5. Initially this will require us using Roslyn Internal APIs. The Roslyn team will be using our experience implementing this to figure out what needs to be public, and what the Public API shapes should look like.

I hope this gives you a taste of what we are thinking, it is not firmly committed yet, although we are all very excited about it and are very hopeful this will be the approach we take. Please don't read too much into any particular phrasing I have chosen. This is certainly a malleable project, and will evolve through our joint work.

Thanks
Kevin

[cloudRoutine]

@KevinRansom did you catch this discussion a couple of weeks ago?
At first glance this new direction seems at odds with what we're moving towards with VFPT, but maybe you see a way for them to work synergistically?
@vasily-kirichenko @dungpa what do you guys think?

[forki]

Since you were talking about VSCode integration. Please also talk to @Krzysztof-Cieslak who already brought excellent vscode support (from day 0!) and also atom support.

[dsyme]

@cloudRoutine It would make sense for VFPT.Core to be brought in as a component here (and potentially be used to run compiler logic out of process).

Either way I'd expect that nearly all of the VFPT implementation could play a role in this toolchain: VFPT has seen a huge amount of use and the implementation does hundreds of important extra little logical things in interpreting the information coming from FCS. It's inevitable that it needs to be brought in as part of the story here.

@forki I think VSCode is a far off thing for this - a long term stretch goal - I'm not sure Roslyn Workspaces being used to host any editor inside VSCode as yet

[cloudRoutine]

@dsyme I was hoping that'd be a possibility over

The F# Powertools features will require porting to the new Roslynized language service. Ideally they would be submitted as feature PR's

as it'd be unfortunate to lock the other editor and IDE options out further developments in F# tooling support. Otherwise it becomes a lot of redundant reimplementation of what are fundamentally the same tasks and the editors that don't have as dedicated maintainers fall by the wayside. Reducing the workload for maintainers of individual plugins, in my mind at least, is the most beneficial for the F# ecosystem. Then if someone wants to add F# to say Brackets or Kate there's a low threshold to gain a lot of functionality.

[Krzysztof-Cieslak]

1. This approach is intended to lead us towards a vscode/omnisharp integration strategy too.

This point is bit worrying, but let's just assume it was some kind of bad phrasing.

I would just like to know what will be implications for FSharpCompilerServices project with this change:

• will FCS include new Roslyn API?
• will "old" FCS APIs be changed because of that?

I would also be interested what will be place of VFPT.Core (BTW, we should kick V out of this name) in all this - in issue linked by @cloudRoutine we've discussed possibility of using this core set of functionalities built on top of FCS across all editors (or rather F# extensions systems - so VFPT, Xamarin, FSAC). If I could suggest anything, building Roslyn API for VS on top of VFPT.Core could be really interesting, and potentially most benefit for whole community. So we would have following set up:

FCS  ==> (Visual) F# Power Tools Core
                  |__ VS Rosyln API
                  |__ VF# Power Tools
                  |__ Xamarin
                  |__ Ionide
                  |__ etc...

Credit to @cloudRoutine for fantastic graph ;)

[dsyme]

@cloudRoutine All F# editor tooling will continue to see major improvements driven by the various contributors: F# is a multi-editor language and nothing changes in that. FCS, the Visual F# Power Tools "Core" components and FsAutoComplete are critical components in enabling good multi-editor support. They are totally part of the future. To varying degrees they will play a role here too.

Equally, the VIsual F# Tools will greatly benefit from a visual revamp based on Roslyn. It will be amazing to get that done - I can totally see the logic in this and am excited by it.

You are totally right to point out that the Visual F# Power Tools "Core" components are already there and ready to use to help enable much of this (however this applies less the "parity" features, which aren't covered by VFPT). I'm sure @otawfik-ms and @KevinRansom will be taking a close look at the VFPT Core libraries in order to get maximal engineering efficiency here - no one is interested in duplicating work.

This is about putting the core Visual F# Tools IDE work on a modernizing trajectory. It is especially modernizing from the perspective of the Visual F# Tools Team, who sit next alongside and in partnership with the overall Roslyn team, and who have seen the TypeScript team successfully adopt the Roslyn IDE APIs to good effect.

[dsyme]

will FCS include new Roslyn API?

No. FCS as a component continues to be published as a nuget package from fsharp/FCS. @KevinRansom is referring to the enabling subset of the FCS APIs brought in by PR #853 - see that for details. That subset is needed no matter what work gets done in the Visual F# IDE Tools.

will "old" FCS APIs be changed because of that?

No, though it may drive additional improvements to FCS, much as VFPT and other clients has.

Also, note that the Roslyn APIs in question here are about implementing the "workspaces" or "IDE" APIs. As @KevinRansom mentions, TypeScript's VS tooling uses these. This hasn't made TypeScript as a language dependent on Roslyn (which is I think what you're concerned about, reading between the lines of your questions) except for the TypeScript VS tooling - which was easier to implement via this route.

[dsyme]

@KevinRansom One question - "build integration" and "debugger integration" are partly or mostly handled by the F# Project System, which as I understand it is not being replaced by this work?

AFAIK the interactions of these features with the language service is as follows:

• "debugger integration" includes breakpoint validation and name reporting based on parsing, both implemented by the Language Service
• "build integration" includes unifying the errors that come from the build with the errors reported by the language service, this "unification" is I assume done by the code in FSharp.Editor.dll. IIRC it was tricky to get right so the implementation there should prove useful.

[dsyme]

@KevinRansom One "parity" feature not mentioned is the navigation bar (off by default but turned on by the Visual F# Power Tools)

[KevinRansom]

Everyone ...

We intend to use this approach to be able to leverage the work of the Roslyn IDE team as well as modernizing our IDE experience. We absolutely love the work of the powertools team and have a high degree of respect for the way they have driven forward the user-experience for F# developers. We are glad for their efforts and hope to be able to help them continue on the new IDE surface area. It is a certainty that the powertools could benefit from leveraging the Roslyn APIs a whole bunch of Visual Studio specific interface goo will evaporate from their codebase. However, the Roslyn APIs are not yet fully public in all of the useful areas ... they are open sourced, but they do not all have public visibility yet, we expect this effort to shine a light on a lot of APIs that will need to be made public. And I expect the Roslyn team to respond quickly to ensure that they are.

It has long been the plan of record that Roslyn cross platform tooling for VSCode will be accomplished by integrating the Roslyn Workspaces layer using OmniSharp; for us the nice thing about this approach is that we will automatically benefit from that work by interfacing this way. I am very excited to know that we may not have to even design some new editor features, knowing that a similar design already exists in Roslyn, we as a community may be in a position to adapt what they have and specialize it for F#.

The ability to have Roslyn style code fixers and Analyzers for F# code is an outstanding functional gain for our F# tooling. The amazing work by Krzysztof and also the Xamarin guys to build on the existing powertools Apis still exist and are still relevant and are still amazing. We desperately hope that the powertools will continue to be enhanced and will provide advanced F# Editor functionality appropriate to both our tools and also these other editors.

The way I think about these features is an attempt by us to make our tooling modern by capturing some of the Roslyn IDE excitement

@dsyme we can certainly consider adding the navigation bar to the @ parity list. Or it can be a community feature, either works.

I hope you find that this approach to our editors is exciting and something you will feel happy helping us to realize ... it is a certainty that we, the Managed Languages team, are very excited on about the opportunity to get started on this.

Kevin

[isaacabraham]

+0.99 - certainly having seen the way that Typescript is supported in VS now with basically a superior OOTB experience to F# in such a short space of time sounds to me that the community can do great things with this approach. I've only done a tiny bit of work on VFPT so I can't claim to be an authority but as long as we keep the process lightweight and try to get reuse where possible across the existing VFPT toolchain etc. etc. then I'm all for this.

[cloudRoutine]

@rneatherway read this thread #913 Basically, MS team is going to rewrite VFT using Roslyn Workspaces, which means 1. IDE support will be on pair with C#/VB (and will always be up to date) 2. VFPT will stop working. So I think the best thing we can do is to wait until MS team rolled out a Roslyn based VFT and start to contribute to it.
^ fsprojects-archive/zzarchive-VisualFSharpPowerTools#1317 (comment)

@KevinRansom is there a rough idea when this VFT revamp is going to start?

[dsyme]

One thing not mentioned above (understandably) is about moving the existing VisualFSharp.Unittests IDE unit tests forward as part of this work.

On the whole these unittests are written as a fairly high level in terms of user actions like "open a file", "move cursor to position" etc. The content should basically be reusable and if it isn't we should work out why.

The "Salsa" layer implements this "DSL" and simulates the state that VS holds and unit tests FShap.LanguageService.dll with no actual Visual Studio DLLs loaded - mocks to Visual Studio interfaces are used). The Salsa layer would presumably need to be re-implemented with essentially the same API.

Regardless of how we do it, the tests themselves are a hugely important asset for the Visual F# Tools. There are literally thousands of corner cases in colorization, quick info, intellisense, error lists, type providers and so on captured by those tests.

I'd go as far as to say that passing a revamped version of those unit tests will be a key litmus test for Visual-F#-Tools-on-Roslyn. Also those test cases should make the whole thing much cheaper, since less investment is needed in retesting the "parity" features.

I'd be glad to help/advise in how to take this test content forward - my recent PRs simplified Salsa to the point where it's considerably easier to see what's going on, and means that I'm now more familiar with the code.

[OmarTawfik]

@cloudRoutine I also prioritize not breaking VFPT until the new Roslyn-based VFT reaches parity/maturity, then we would see how to do the switch. I'm looking now at the codebase to see how can this be done with maximum efficiency possible.
@dsyme noted. I'll go through the existing Salsa setup to make sure we achieve test-parity as well.

[enricosada]

this prototype and roslyn workspaces mean also a new project system? ref #496

[NumberByColors]

@enricosada No, this would just be the editor, not the project system.

[eriawan]

@NumberByColors

Any plans to support Visual Studio CPS? I believe it's somehow linked with how F# overall experience from creating new project, coding and interacting with the IDE (including using Code Lens), to producing the final build (including file organizations, build order) and interacting with MSBuild.

Also I believe from this Andrew Arnott's blog the migrations are clear: moving forward to CPS.

PS: If you want, I'll open new issue instead for this Visual Studio CPS topic.

[enricosada]

@eriawan see #496

I think CPS can be used with xproj and project.json (see #806 ). You can already do xproj but it's bugged because vs support for xproj is in flux because the migration from dnx to dotnet/cli. visualfsharp side there are some todo, like we need to read references list and pass that to language service ( from cps references or dotnet/cli project model, dont know ). But i opened an f# xproj and lot of stuff worked right.

[NumberByColors]

@eriawan @enricosada At some point it would be great to revamp the project system, but as far as I know that's orthogonal to the editor revamp.

[ghost]

@dsyme why would that be closed? I didn't find a more recent issue that better tracks the work there.

[dsyme]

@oamsath OK, reopened :)

[eriawan]

@NumberByColors @enricosada
Now that the decision about the demise of JSON project model I believe it's done (the VS and .NET Core team) that the model is going back to MSBuild project model, including the planned support for CPS.

Is there possibility that maybe xproj is being used for MSBuild Xplat? Do you guys know what will be the future roadmap of F# project model for Visual Studio tooling outside VS Code?

[cartermp]

@eriawan Eventually the F# project model for Visual Studio will sit atop CPS. No timeline at the moment, though.

[enricosada]

@eriawan just to distinguish the file project (fsproj/project.json) from project system (cps/current vf# code).

the xproj already works today ( no intellisense, only build/publish). But it's the vs project of project.json so it's going to die too (it's a msbuild target)

Xplat, I think, as project file is the fsproj run by netcore msbuild. Maybe I a bit modified to use ( the imported fsharp compiler target, not your fsproj ) a netcore task

The project system (vs ide) I hope the cps, but maybe the current one can run msbuild as it currently does

[eriawan]

@cartermp

thanks for the info. Could we initiate a task or issue to track F# tooling and project model support for CPS? I'd love to track and contribute to it 😄

@enricosada
Yes, the current one .NET Core build can run MSBuild.
But I'm not sure for CPS on xproj-MSBuild, I haven't been able to build it successfully using the latest MSBuild from MSBuild latest build. I haven't visited to test it again, though.

Afaik, last time I checked the MSBuild repo at MSBuild github a month ago, there are no definite plan for CPS on X-plat. CMIIW...

[cloudRoutine]

@eriawan it really depends on the underlying dependencies of CPS Core. It seems like they intend for it to be usable outside of VS (on windows at the least)

A well-designed project system can be used inside and outside the Visual Studio IDE to enable increased productivity and flexibility. For example, a developer may want to test their project system outside of the IDE or even host the project system in an entirely different environment.

If the dependencies are all within the scope of the .net standard library it should be feasible to implement an xplat project engine for F# using CPS.Core.

[dsyme]

Closing this discussion, we are well down this path