Bevy Tool

[JonahPlusPlus]

Summary

The Bevy tool (from here on referred to as "tool") is a CLI/GUI utility for managing projects and editors and their dependencies. This isn't anything official; it's just an idea.

This discussion is meant to explore the pros and cons, possible features and use cases of the tool, and ultimately help decide whether or not to create it. Depending on the feedback, this discussion may be transformed into an RFC. This document is also to help convince the community that a beginner-friendly experience can be created without a traditional editor.

Table of Contents

• Summary
• Table of Contents
• Motivation
  • Library Solution
  • Binary Solution
  • User Experiences
• User-facing explanation
• Implementation strategy
  • Installers
  • Tool Design
  • Code Generation Metabuild
• Drawbacks
• Rationale and alternatives
• Prior Art
• Unresolved questions

Motivation

There has been much discussion of an editor and how it would be created (#5043). There are two general visions: an editor library and an editor binary. They have the following general characteristics:

Library Solution

• Summary
  • A DIY framework for creating an editor as a subproject
  • Adds extensions at compile time
• Pros
  • Decentralized (can get the crate from a vendor like Crates.io or build from source code)
  • Familiar to Rust developers (understandable to modify)
• Cons
  • Requires rebuilding
  • Unfamiliar to traditional game devs

Binary Solution

• Summary
  • A traditional executable to be downloaded and installed
  • Adds extensions at run time
• Pros
  • Familiar to traditional game devs (install just like Unity, Godot, or Unreal)
  • Easy to modify (just click a button to install extensions)
• Cons
  • Centralized (servers are needed to host builds and extensions, typically)
  • Harder to develop (extensions have to be DLLs (unstable Rust ABI) or IPC clients (requires developing a protocol))

Personally, I believe that a library solution is the better of the two (after being convinced after having it explained to me by @bjorn3), as it offers a "Rusty" experience, where extensions are just crates, and allows for dogfooding (the editor can just be another Bevy app, extensions are just plugins, etc.). Compare that to a traditional experience, where there is a distinction between extensions and plugins, since extensions have to be loaded in at runtime, and so there will be more work required to set the foundation for such an editor.

But it can't be ignored that a binary solution opens the door to beginners and non-programmers, who could use extensions to work with node editors. By strictly following a library solution, it requires more of users than a binary solution.

Instead of just creating a library, I propose also creating a tool to simplify the creation of editors and projects. It could also be used for quickly adding plugins by generating code from a plugin-provided installer program to be used with the library.

In order to get a better idea of the use-cases for such a tool, how about we explore it's usage by some hypothetical users:

User Experiences

John is an absolute beginner to game development. He has written little to no code and barely touched the command line, and needs something to ease him into it. Unity and Unreal offer a solution for him: GUI installers and node programming editors allow him to start creating a game without knowing how to write code.

What Bevy can do is provide scripts (bat/ps1/sh) or simple binaries to install Rust and install the tool and open it up in its GUI mode. From there, the tool can be used to create desktop shortcuts, create an editor, install a few extensions (including a node editor), and create a new project with some basic plugins. John was able to do this, without writing a single line of code, or typing any commands.

Sally has experience with game development, but not with Rust. She can install Rust, and even write a bit in it, but struggles with the usual stuff, like the borrow checker and lifetimes. She has ideas that she really wants to explore in Bevy, but is held back by her lack of experience in Rust.

However, the Bevy community has graciously provided plugins and extensions that she can install via the tool. Among these is a plethora of scripting plugins for all sorts of languages. Sally didn't really need the tool to do it for her (it's already easy to add plugins in Bevy, so she could have figured it out), but she appreciated the convenience of being able to quickly switch out plugins until she found the ones she likes.

Grant has experience with Rust and Bevy. He may be on a small team of developers making small games. He can do everything the tool does himself, but he still uses the tool for efficiency and organization.

Because the tool organizes his editors, he can start work on new projects without creating a new editor or changing his path to point to a specific editor. The tool also allows him to add plugins really quickly. What this means is that Grant can prototype new ideas in Bevy without writing the boilerplate himself.

May is part of a large team that is in the middle of developing a large game. Her team already has a system created for using Bevy and the editor, which is a workspace member of the project (so as to be included in the project's VCS) and is called via a cargo alias. Her team has no need for the tool, since they are managing their own editor and have all the plugins they need.

From these experiences, we can conclude that the tool makes Bevy more accessible and also makes Bevy users more efficient. It is also optional; the users weren't forced to use it but chose freely. In the case they don't need it or it gets in their way, they can still use Bevy as usual, just like Unity and Unreal. The tool also loses usability as development progresses.

User-facing explanation

The tool is installed via cargo install bevy. It can also be installed via a script/binary that can be downloaded from Bevy's website, which also installs Rust if necessary and also opens the tool afterward.

The tool's GUI can be opened by calling bevy without any arguments. On the first run, the tool will set up the necessary file structure, and if it is the first run in the GUI, it will display options for changing settings and creating desktop and/or start menu shortcuts.

The tool also has commands that can be called via CLI (each can be shortened to their first letter):

• bevy [n]ew <name> (--editor/-e): Creates a new project (or editor)
• bevy [e]ditor (<name>): Opens the current editor (or a named editor)
• bevy [a]dd <plugin>(@<version>) (--editor/-e <name>): Adds a plugin (at latest or given version) to a project (or editor)
• bevy [r]emove <plugin> (--editor/-e <name>): Remove a plugin from a project (or editor)

When you create a new project, it creates the following structure in the current directory:

.
├── .git
├── Bevy.toml
├── Cargo.toml
├── .gitignore
└── src
    └── main.rs

It's like cargo new --bin but adds Bevy.toml and modifies Cargo.toml and src/main.rs. The Bevy.toml file is where project settings (like plugins) for the tool can be modified. It has the following format:

[plugins]
bevy_atmosphere = "0.5"
bevy_something = {
    version = "0.2",
    config =  { # used for configuring the code generation
        key = "value"
    }
}

Code generation works by plugins providing installer scripts (See Implementation strategy). These scripts are used to write to a generated file. src/main.rs now looks like:

use bevy::prelude::*;

fn main() {
  App::new()
    .add_plugins(DefaultPlugins)
    .include_plugins()
    .run();
}

include_plugins calls a macro that includes and parses the generated file. In the case of a new editor, DefaultPlugins is replaced with EditorPlugins.

In Cargo.toml, Bevy is a dependency by default. The version of Bevy used is the one associated with the version of the tool, unless changed in the tools config files. metabuild is also set to Bevy's metabuild, so the codegen runs when cargo run or cargo build is called. include is also set to include Bevy.toml, so published crates don't forget it.

The config files are .bevy/config.toml files. By default, this file is created at ~/.bevy/config.toml, but can be used elsewhere to configure the tool's behavior. The default structure of ~/.bevy looks like:

.bevy
├── config.toml
└── editors
    └── default
        ├── Bevy.toml
        ├── Cargo.toml
        └── src
            └── main.rs

As you can see, the tool automatically creates an editor named "default". By default, .bevy/config.toml looks like:

[editor]
profile = "default"

By creating your own .bevy/config.toml files (or editing this one), you can change the current editor for your path.

The need for separate .bevy/config.toml and Bevy.toml files arises from how .bevy/config.toml settings propagate top-down, while only the lowest Bevy.toml are effective.

Implementation strategy

This tool can be created later, after an editor library has been implemented. However, it is important that this is an official part of Bevy, rather than as a separate project, since it will work better if plugin creators feel an obligation to provide installer scripts out of a sense of correctness. It would also help to discourage the creation of other codegen tools and their standards if there is something official, which will prevent a possible fracturing of the community (very low risk and not an issue now, but if someone makes such a tool, then it might encourage others).

Installers

An installer script is just a binary target that is part of a crate. It is specified in Bevy.toml like so:

[installer]
target = "install"

When the installer runs, it communicates back to the tool via stdin/stdout/stderr IPC. An installer program looks like:

use bevy_editor::install::*; // imports macros

fn main() {
  input! {
    key: String,
    code: Option<u32>,
  } // reads stdin; can convert values that impl FromStr and supports optional parameters
    // (missing required parameters panics and writes to stderr)
  match code {
    Some(c) => if c > 5 {
      error! { message: format!("Failed: {c} was greater than 5"), custom_payload: #c }
      // allows for passing a message and other information to be logged (JSON?) via stderr
    },
    None => {}
  }
  output! {
    @insert_plugin(bevy_something::SomethingPlugin(#key));
  } // writes output to stdout (@ is converted to the `App` identifier, ex: "@"=>"app.")
}

These macros allow plugin creators to define powerful configuration options for users. This installer would then output the following to the generated file:

{ // anonymous scope to keep it more hygienic (still possible to misuse `App` identifier)
    @insert_plugin(bevy_something::SomethingPlugin("key"));
}

Then, from the user's program, App::include_plugins would call the include_plugins! macro (users could also use the macro directly). So, App::include_plugins looks like:

fn include_plugins(&mut self) -> &mut Self {
    include_plugins!(self); // @ => self.
    self
}

and evaluates to:

fn include_plugins(&mut self) -> &mut Self {
    {
        self.insert_plugin(bevy_something::SomethingPlugin("key"));
    }
    self
}

Tool Design

The tool would be another Bevy app, since it can dogfood the UI for the GUI mode. As for the CLI, I believe a system should be upstreamed to Bevy, as the editor could also support a CLI, as well as many Bevy apps and games. I have some ideas for an Action system, which could expose commands to CLIs and GUIs. It would also support a history of actions, allowing for undoing/redoing actions, making commands like bevy [u]ndo possible for quick fixes (as well as bevy e undo and bevy e redo for the editor).

Code Generation Metabuild

When cargo run is called, a metabuild script executes that runs the code generation. The metabuild doesn't actually generate code itself, instead, it calls a command (maybe bevy build?) that can get settings from .bevy/config.toml and Bevy.toml files in the path. (This is a pretty simple metabuild, so maybe bevy new can just generate a simple build script?)

Plugins are downloaded from Crates.io (or some other specified source, like a git repo or a path). Each one is verified to have a correct Bevy.toml file. They are then put with the other dependencies (the feasibility of this would need investigating, and maybe the source code can be cached in .bevy/plugins?). Their installers then run, and the output is written to some file to be referenced by the include_plugins! macro. Any failures cause the entire build to fail.

Drawbacks

Any changes require rebuilding the project or editor.

It puts a responsibility on plugin creators to include an installer and document it. However, if an installer doesn't exist for a given plugin, the user can still use it normally:

use bevy::prelude::*;

fn main() {
  App::new()
    .add_plugins(DefaultPlugins)
    .include_plugins()
    .add_plugin(bevy_thing::ThingPlugin)
    .run();
}

Rationale and alternatives

The other option that provides a smooth experience for beginners is the traditional editor design with IPC extensions. However, I don't like the idea of extensions being different from plugins; I would rather keep things homogeneous. I don't mind DLLs for that reason (you would just load extensions at startup and add them to the app) but the Rust ABI is unstable, so it isn't feasible. That all being said, IPC extensions can be hot reloaded, so there are benefits, but is that really necessary? It seems that other engines support IPC or DLL extensions so they can run an add-on store (and hide their source code).

Prior Art

Some people might find the idea of code generation foreign and perhaps dangerous. I don't think there is an issue: macros are a huge part of Rust and build systems are common (though some are pretty bad, so it isn't surprising that some might have developed an aversion to all).

Unresolved questions

I still think the code generation could be more hygienic. Maybe the macro could check if the App identifier conflicts with any identifiers in the installer and alias it if so? Like:

{
    let app_1323532 = app;
    // Generated Code
    let app = Applet(0); // cause of the conflict
    app_1323532.insert_resource(app);
    app_1323532.add_plugin(AppletPlugin);
}

[alice-i-cecile]

For cross-linking: previously discussed in #436.

[JonahPlusPlus]

Thanks, I wasn't aware of that discussion. Some of those ideas could be incorporated into this (bevy doctor and bevy prepare really caught my eye).

[JonahPlusPlus]

Another idea:
The input! macro, instead of reading from stdin, could read from Bevy.toml directly. This would allow for configuring all Bevy apps at runtime, instead of just extension codegen at compile time. For instance, editors could have settings in Bevy.toml:

[config]
theme = "Dark"
undo_limit = 128

[plugins]
something = {
  version = "0.3",
  config = { # just to show the relationship
    key = "value"
  }
}

This also means that plugins could have default configs instead of just options that evaluate to a default value when None (also allows for a default value of Some(T), that can be overridden to None by setting the value to {}, the only way to express it in TOML afk).
I wonder if the other macros could also use the file system instead of stdout/stderr (though I don't know the use cases for them outside of extension codegen yet).

[JonahPlusPlus]

The drawback is that Bevy.toml will have to be present, which might not be nice when publishing a game (maybe on debug builds the macros can load options at runtime, while release builds load options at compile time?)

[BlackPhlox]

In regards to codegen, I've made a small poc on this with bevy_cursed_editor which is similar to what you explained in some ways, though it generates the entire codebase from scratch. Might be useful for testing stuff out :)

[JonahPlusPlus]

Yeah, my worry with such a system is that it could make wrong assumptions. I think a good codegen system would generate very little code from scratch, relying on crates to provide the code needed. However, @Nilirad's reply makes it sound like my proposed system could be making assumptions on the interactions between crates, so I got to investigate that (and get caught up with the state of stageless).

[Nilirad]

Codegen will be a problem in the future. Right now, the scheduling of a plugin is completely self-contained. This however creates a problem that reduce plugin composability (See #2160). The issue will be solved by the upcoming stageless schedule, but as a result, the plugin is not scheduled by itself, but by user code. And that user code may vary depending on what other plugins are present.

[JonahPlusPlus]

I'm trying to understand the problem: would this affect all plugins or just plugins with special behavior? This codegen system could just be used for basic, self-contained plugins, then users can add other plugins that need more work to setup normally. Essentially, will the following not be possible anymore?

use bevy::prelude::*;

fn main() {
  App::new()
    .add_plugins(DefaultPlugins)
    .add_plugin(bevy_thing::ThingPlugin)
    .run();
}

[Nilirad]

I'm not confident about giving an accurate prediction, as nobody has any experience with the new pattern. However I can imagine that for most cases the codegen tool will be able to succeed, since the core part of the editor will already provide a consistent structure. I would not be surprised however if there will be cases where the default scheduling provided by the codegen tool will fail because of some strange interaction between plugins. In this case, special user intervention will be probably required to fix the problem.

[JonahPlusPlus]

Okay, after reviewing the stageless RFC, I think the codegen tool could integrate into the new API nicely:

use bevy::prelude::*;

fn main() {
  App::new()
    .add_plugins(DefaultPlugins)
    .include_plugins()
    .configure_set(
       bevy_some::SomeSet  // SomeSet and OtherSet are both plugins from include_plugins
         .after(bevy_other::OtherSet)
    )
    .run();
}

Though, I am still curious about how easy it would be to debug this. Maybe the tool could provide a command to create graphs for debugging the order of sets and systems? Just a thought.

[JonahPlusPlus]

Fleshing out the .bevy/config.toml format a bit more:

[editor]
profile = "basic" # set the current editor used
release = true    # runs the editor in release mode (see my previous reply about config evaluation time)

[alias] # creates aliases for other bevy commands
node = "editor --name node" # bevy node => bevy editor --name node

[codegen]
jobs = 0           # number of parallel jobs (0 evaluates to number of CPUs)
incremental = true # Writes the output of each installer to separate files before combining

[term]
quiet = false   # whether bevy output is quiet
verbose = false # whether bevy provides verbose output
color = 'auto'  # whether bevy colorizes output

Instead of the tool's source code allowing short versions of commands, there could be default aliases instead (this is the new default file):

[editor]
profile = "default"
release = true

[alias]
n = "new"
e = "editor"
a = "add"
r = "remove"

[codegen]
jobs = 0
incremental = true

[term]
quiet = false
verbose = false
color = 'auto'

Incremental codegen could work by generating files that have a hash of the config and the corresponding output:

b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9
@add_plugin(bevy_thing::ThingPlugin);

[nigrodev]

I hadn't read this issue yet, I found the idea very interesting

Yes, it's true that a lot of things related to Bevy have to be done manually. Just to make a functional and optimized piece of code, you need to spend a few minutes editing Cargo.toml, adding features and then editing the main rust file.

I made bevyinit, which is to minimize all this work in just a few seconds with some available templates and options: https://github.com/nigrodev/bevyinit

Currently, the use of cargo-binstall is very helpful for distributing the binary and making it easier to use

[alice-i-cecile]

I've written up some initial work scoping for this at https://hackmd.io/@bevy/rk2Qnf5i0/edit