Add cargo integration

[emberian]

• Adding dependencies
• Updating the lockfile
• Build the project
• Run tests
• Run benchmarks

[emberian]

All of this should happen in the GUI.

[vosen]

I've been working on a project system lately and the issue of cargo integration really had become a sticking point. I'm unsure how to proceed so please comment and leave suggestions.

Introduction

Probably not everyone reading this is 100% familiar with VS projects so quick explanation is in order.

Project system is a part of every language implementation in Visual Studio. It's the code that handles loading project file to a nice hierarchy of nodes as seen in Solution Explorer and interactions with VS (building, adding files, adding references, etc).

Project file is a file that stores all information about the project. For example .csproj files are project files for C# projects. Project files can have an arbitrary format. Currently, most VS languages (C#, C++, VB.NET, F#) use MSBuild build files as a project backend, that has a nice property of projects being compilable without the presence of VS and eases linking to projects in different languages.

Problems

There are few goals and use cases that we have to cover with project system that problematic due to cargo being a rather inflexible build system.

1. Post-build events

   This covers executing arbitrary shell commands after the build. For example adding a manifest to the compiled executable. This, along with pre-build events are two most commonly used extension points in VS projects. Currently cargo can't do this.

2. Passing --cfg flags

   Currently cargo can't do this.

3. Passing --emit flags

   At least having an option for "--emit obj" would be useful. Currently cargo can't do this.

4. Passing --target flags

   Want to cross-compile for android or iOS? Can't do this. Though it probably doesn't work on windows anyway. Turns out, cargo can do this.

5. Passing -C flags

   Required for building windows desktop apps, to pass -mwindows flag to the linker. Currently cargo can't do this.

6. Settings lints as warnings/allowed/denied/forbidden

   Currently cargo can't do this.

7. Integration with cargo projects

   Project systems should be able to consume cargo dependencies. Project system should result in a Cargo.toml file that is consumable for other projects that don't use VisualRust.

8. Integration with VS projects

   It should be easy to reference VisualRust from other VS projects (at least from ones that are backed by MSBuild).

9. Arbitrary files as part of the project

   Not really something that we want from cargo, but VS projects should be capable of containing arbitrary files. Things like application manifest, icons files or C headers.

10. Arbitrary project configurations

    MSBuild-backed projects (C#, VB.NET, C++, F#) allow for multiple arbitrary Configurations that can control many of the project settings. Cargo sort of has this with five built-in profiles. Not hugely important, but nice to have.

11. Arbitrary project layout

    It's not super important but it'd be weird to tell users "Sorry you can't have your main.rs in this location because cargo says so" and "Sorry your entry point file can't be named root.rs because cargo says so".

Proposed solutions

1. Use cargo for projects, building and dependencies

   Cargo.toml becomes a project file. In UI we only expose stuff that is supported by cargo. All problems with the exception of 7 are left unsolved. Problem 9 is especially painful and something really below the bar set by project systems of other languages.

2. use MSBuild for projects, cargo for building and dependencies

   Pretty much the same as Solution 1, but solving problems 8 and 9. We'd use MSBuild file just to keep a list of files.

3. Use MSBuild for projects, rustc for building, cargo for dependencies

   Most of the problems are solved, except 7, because we can't generate Cargo.toml that builds the file. Also this is currently impossible, cargo doesn't do package management at the moment. Only building.

4. Use MSBuild for projects, rustc for building, cargo for dependencies, optionally generate Cargo.toml

   The same as 3 with the marked difference that we allow to generate Cargo.toml if the project doesn't do any of the problematic things (Post-build events, settings lints, arbitrary entry point files, etc).

I'm currently leaning towards solution 3, but that would mean postponing cargo support until after rust-lang/cargo#358 is done.

[Boddlnagg]

Solution 3 sounds reasonable. As an additional note, I think it would be useful if the MSBuild project file can be generated from a cargo manifest, maybe even directly using cargo (does cargo support plugins?). This would allow to build all cargo-based projects with Visual Studio, as long as all cargo features are somehow supported in Visual Rust projects.

I've seen this in IntelliJ for Scala SBT support: SBT is a build system that also deals with dependencies (like cargo) and there is (1) an SBT plugin that generates IntelliJ project files (also watching for changes in the SBT manifest with the possibility to automatically regenerate and reload the project files) and (2) an IntelliJ plugin that uses SBT for building. The combination of those two works well and something similar might be possible here. Visual Studio is not IntelliJ, though, of course.

[vosen]

Thanks. I've checked IntelliJ and they do it differently, they separate project file and build system. So it's closer closer to our solution 2. When it comes to Cargo integration, Cargo --> MSBuild can be done by exposing project template (an entry in "New project..." window) for "Create VisualRust project from existing cargo project". I can't think of any cargo feature that can't be replicated with MSBuild (at least when it comes to building).

[ghost]

Here is my vision.

We should think not only about how nice it will be in VS-only usage scenarios but also about interoperation with non-VS world. Inability to generate cargo.toml file is huge drawback, it means that any Rust project created with VS will be useless to those who don't use VS and will require them to write toml file manualy. Using other file format for project files means that it would be necessary to convert it to/from cargo.toml. I think that supporting cargo.toml as a project file is a must. Just imagine - you can download any (well, probably not) Rust project and it can be just opened and compiled in Visual Studio.

Since using cargo.toml for project files has some issues as you described there should be possibility to use more VS friendly project file.

So there will be two options:

1. If you want to develop open source Rust project, probably cross-platfrom, upload it to crates.io, etc, then you use cargo.toml as a project file. In this case developers are forced to use only things supported by cargo.
2. If you want to stick to Visual Studio as the only development enviroinment for your project, interoperate with other VS languages, etc, then you use VS-specific project file. In this case it is as much as possible compatible with VS and uses concepts familiar to VS developers.

[xilec]

All known to me rust project is on github. All they use cargo for building, that's why cargo support is very important feature for VisualRust and that's why, to my mind, it must be one of next features implemented in Visual Rust.

I agree with Ghost that there are 2 working use case for rust developing in visual studio:

1. Cross-platform developing (open source project)
2. Multi-project visual studio specific solution

But I suppouse that make additional build system over cargo.toml in Visual Studio is too much overhead for this task. To my mind, generation rsproj from cargo.toml and inculding cargo.toml into this project is quite suitable approach as from user side so from developing side.

IMHO, in ideal way (if it is possible), we assisiate cargo.toml with visual studio. When it file is being opening first time, we are creating rsproj-file and opening it, otherwise updating rsproj and opening.
Probably, we should add setting build mode (using cargo or rustc) in project setting.
Thus, to my mind, solutions #2 and #3 from Vosen must be working and quite convenient for end user. In this case, we get flexibility-msbuild(interoperabilaty with other VS languages, using configurations of VS solution) in VS with almost the same rsproj-file for project (probably in cargo building mode some parameters will be given from cargo.toml file and they will be readonly in rust-project properties).

[vosen]

Thanks for your thoughts everyone, the plan is to go with solution nr 2. Here are my rationales and plans.

How it will work in practice is that we will rely on .rsproj project file + Cargo.toml.
Rationale for that is twofold: relying solely on Cargo.toml would disallow us from including non-.rs files in the project. We can and do automatically detect module files that are included by the crate (so if you have source mod foo; mod bar; fn main() {} we automatically detect and include files foo.rs and bar.rs in the Solution Explorer window). The problem is that it's not 100% bullet-proof (mainly because of macros), which means that sometimes you required to manually add files to the project. And losing ability to add arbitrary files (eg. bitmaps, icons) to the project would be painful.

Integrating with cargo will be the main focus of 0.2 release. That's because I want to get some kind of usable 0.1 version out. Cargo integration touches every facet of Visual Rust:

• Dependencies

  Currently, we don't handle dependencies at all. We will support dependencies by having nice "Package management" window a la "Manage Nuget Packages" (we might even be able to copy it wholesale as nuget vs extension is apl2-licensed). Also added dependencies will show up in "References" node with some quick info.

• Building

  Currently, we call rustc using MSBuild. We wil be calling cargo through MSBuild for the sake of compatiblity, but it should be easy and painless.

• Project system (nodes shown in "Solution explorer")

  Currently, we assume that library projects start in src/lib.rs and application projects start in src/main.rs. When moving to cargo those assumptions will be lifted. There's one unanswered question though: should we allow for Cargo.toml in a different location than .rsproj? I'm leaning towards disallowing it.

• Project configuration (configuration shown when you choose project Properties)

  This will be tedious as we will have to expose every little cargo setting and make sure it works. Shouldn't be too difficult, but as I said, tedious.

This of course relies on existence of a Cargo.toml parser that would allow us for painless loading/editing/saving. I'm not sure if one exists. Worst case scenario we will have to roll our own on top of toml package.

[Boddlnagg]

This sounds good to me! Especially the part about managing Cargo dependencies just like Nuget does (i.e. in the way VS users know it) ... this will be great, when it works :-)

One concern is the output parsing: Currently we run rustc through MSBuild (as you said in "Building"), and parsing its output is already quite difficult. If we switch to run cargo instead (which is the only reasonable thing to do), it will be even more difficult, since cargo forwards the output of rustc, but also for errors in dependencies, etc. This could be further complicated by a plan to animate the output of rustc/cargo: rust-lang/rust#24335. Maybe an option can be added to rustc/cargo so the build output is printed in a more machine-readable format (e.g. JSON)? I think this would also be beneficial for other similar projects (editor plugins, etc), but I don't know how they currently handle this.

Also, this cargo PR looks relevant for us.

[vosen]

IIRC cargo sends rustc errors to stderr and everything else to stdout.

[lostmsu]

I started Cargo.toml wrapper to support this task: https://github.com/lostmsu/Rust.Cargo

[vosen]

@lostmsu I can't compile your project (even after converting from portable project and adding nuget reference to hypertoml). But the thing about parsing Cargo.toml is slightly more subtle than just serializing/deserializing. We should be able to round-trip Cargo.toml. What I mean by that is that when we receive file like

# THIS IS A VERY IMPORTANT COMMENT
[package]
name = "hello_world"
version = "0.1.0"
authors = ["Your Name <[email protected]>"]

and the user changes package name from hello_world to goodbye_world we can't lose the comment at the top during deserialization and serialization. And AFAIK HyperTomlProcessor doesn't support such mode.
That's why I've been working on support for round-tripping mode in toml-rs. It's going slowly but I think it's very much doable. In the meantime I recommend working on some other part of cargo support: moving build system system to cargo or porting nuget packaging UI.

[lostmsu]

@vosen how are you planning to use Rust code in your VS extension?
I actually started modifying HyperToml to make it work with round-trips. Added a build instruction to Rust.Cargo.

[emberian]

@vosen So, what's the current status/design of this?

[verysimplenick]

@vosen cargo needed 👍

[vosen]

@lostmsu I couldn't build your project, so I went with toml-rs
@cmr I've redone toml-rs internals to support our user case for Cargo.toml, progress here. I plan to finish up generic toml parsing, build higher level cargo configuration library in Rust (with C API) and then wrap it from C#. Other than that, no progress.

[Boddlnagg]

@vosen Any progress on this recently?

[retep998]

This one thing is basically the only blocker for a bunch of people I know that want to use this, including myself.

[vosen]

Not much progress from me. I know it's high priority, but I've got sucked into fixing #133. I've prioritized #133 because I think that having buggy implementation of syntax highlighting is more embarrassing than no support for cargo.

[LaylBongers]

I would personally disagree with that, atom's rust plugins have pretty bad syntax highlighting but they integrate really well with cargo for linting. Because of this, I can actually use atom even if the highlighting is off now and then, while VisualRust is entirely unusable for me because of the lack of cargo integration.

[briansmith]

Not much progress from me. I know it's high priority, but I've got sucked into fixing #133. I've prioritized #133 because I think that having buggy implementation of syntax highlighting is more embarrassing than no support for cargo.

Even though I am very eager to see the Cargo integration, I agree with this. I often use a different editor because the syntax highlighting in it works better. (In particular, that other editor does brace matching, which is very important for Rust.)

[xilec]

@vosen, I mostly agree with retep998 and LaylConway. We can compile with cargo from console. It's not so big deal. But, we can't use Visual Studio debugger without building with cargo inside Visual Studio. IMHO, it's much more valueable feature than syntax highlighting. At least, we can use some another text editor for syntax highlighting, but debugger is still not allowed at all, because almost all programs slightly more complex than "Hello world" use cargo.

[chris-hawley]

Cargo integration still appears to be missing....

[matklad]

FYI: cargo metadata sub command may be useful to learn about dependencies and project structure.

[huettenhain]

Cargo support would be great, but this thread has been silent for some months, so I am not sure if this is happening. On the other hand, it seems impossible to write much beyond "Hello World" without using external crates.

Is there any workaround to compile & run a project with dependencies, in VisualRust?

[Boddlnagg]

@huettenhain Project and build support for Cargo has been added in #234, but it has not been released yet. You can try it with a build from AppVeyor.

[huettenhain]

@Boddlnagg Thanks for the suggestion, the Cargo support looks good. Unfortunately, the plugin is not working - the linked build of VisualRust fails to build the default template, let alone anything more complex. I can post the error message if requested, but I do not think this is the right thread for that.