Migrate engine to Schedule v3 (#7267)

Huge thanks to @maniwani, @devil-ira, @hymm, @cart, @superdump and @jakobhellermann for the help with this PR.

# Objective

- Followup #6587.
- Minimal integration for the Stageless Scheduling RFC: bevyengine/rfcs#45

## Solution

- [x]  Remove old scheduling module
- [x] Migrate new methods to no longer use extension methods
- [x] Fix compiler errors
- [x] Fix benchmarks
- [x] Fix examples
- [x] Fix docs
- [x] Fix tests

## Changelog

### Added

- a large number of methods on `App` to work with schedules ergonomically
- the `CoreSchedule` enum
- `App::add_extract_system` via the `RenderingAppExtension` trait extension method
- the private `prepare_view_uniforms` system now has a public system set for scheduling purposes, called `ViewSet::PrepareUniforms`

### Removed

- stages, and all code that mentions stages
- states have been dramatically simplified, and no longer use a stack
- `RunCriteriaLabel`
- `AsSystemLabel` trait
- `on_hierarchy_reports_enabled` run criteria (now just uses an ad hoc resource checking run condition)
- systems in `RenderSet/Stage::Extract` no longer warn when they do not read data from the main world
- `RunCriteriaLabel`
- `transform_propagate_system_set`: this was a nonstandard pattern that didn't actually provide enough control. The systems are already `pub`: the docs have been updated to ensure that the third-party usage is clear.

### Changed

- `System::default_labels` is now `System::default_system_sets`.
- `App::add_default_labels` is now `App::add_default_sets`
- `CoreStage` and `StartupStage` enums are now `CoreSet` and `StartupSet`
- `App::add_system_set` was renamed to `App::add_systems`
- The `StartupSchedule` label is now defined as part of the `CoreSchedules` enum
-  `.label(SystemLabel)` is now referred to as `.in_set(SystemSet)`
- `SystemLabel` trait was replaced by `SystemSet`
- `SystemTypeIdLabel<T>` was replaced by `SystemSetType<T>`
- The `ReportHierarchyIssue` resource now has a public constructor (`new`), and implements `PartialEq`
- Fixed time steps now use a schedule (`CoreSchedule::FixedTimeStep`) rather than a run criteria.
- Adding rendering extraction systems now panics rather than silently failing if no subapp with the `RenderApp` label is found.
- the `calculate_bounds` system, with the `CalculateBounds` label, is now in `CoreSet::Update`, rather than in `CoreSet::PostUpdate` before commands are applied. 
- `SceneSpawnerSystem` now runs under `CoreSet::Update`, rather than `CoreStage::PreUpdate.at_end()`.
- `bevy_pbr::add_clusters` is no longer an exclusive system
- the top level `bevy_ecs::schedule` module was replaced with `bevy_ecs::scheduling`
- `tick_global_task_pools_on_main_thread` is no longer run as an exclusive system. Instead, it has been replaced by `tick_global_task_pools`, which uses a `NonSend` resource to force running on the main thread.

## Migration Guide

- Calls to `.label(MyLabel)` should be replaced with `.in_set(MySet)`
- Stages have been removed. Replace these with system sets, and then add command flushes using the `apply_system_buffers` exclusive system where needed.
- The `CoreStage`, `StartupStage, `RenderStage` and `AssetStage`  enums have been replaced with `CoreSet`, `StartupSet, `RenderSet` and `AssetSet`. The same scheduling guarantees have been preserved.
  - Systems are no longer added to `CoreSet::Update` by default. Add systems manually if this behavior is needed, although you should consider adding your game logic systems to `CoreSchedule::FixedTimestep` instead for more reliable framerate-independent behavior.
  - Similarly, startup systems are no longer part of `StartupSet::Startup` by default. In most cases, this won't matter to you.
  - For example, `add_system_to_stage(CoreStage::PostUpdate, my_system)` should be replaced with 
  - `add_system(my_system.in_set(CoreSet::PostUpdate)`
- When testing systems or otherwise running them in a headless fashion, simply construct and run a schedule using `Schedule::new()` and `World::run_schedule` rather than constructing stages
- Run criteria have been renamed to run conditions. These can now be combined with each other and with states.
- Looping run criteria and state stacks have been removed. Use an exclusive system that runs a schedule if you need this level of control over system control flow.
- For app-level control flow over which schedules get run when (such as for rollback networking), create your own schedule and insert it under the `CoreSchedule::Outer` label.
- Fixed timesteps are now evaluated in a schedule, rather than controlled via run criteria. The `run_fixed_timestep` system runs this schedule between `CoreSet::First` and `CoreSet::PreUpdate` by default.
- Command flush points introduced by `AssetStage` have been removed. If you were relying on these, add them back manually.
- Adding extract systems is now typically done directly on the main app. Make sure the `RenderingAppExtension` trait is in scope, then call `app.add_extract_system(my_system)`.
- the `calculate_bounds` system, with the `CalculateBounds` label, is now in `CoreSet::Update`, rather than in `CoreSet::PostUpdate` before commands are applied. You may need to order your movement systems to occur before this system in order to avoid system order ambiguities in culling behavior.
- the `RenderLabel` `AppLabel` was renamed to `RenderApp` for clarity
- `App::add_state` now takes 0 arguments: the starting state is set based on the `Default` impl.
- Instead of creating `SystemSet` containers for systems that run in stages, simply use `.on_enter::<State::Variant>()` or its `on_exit` or `on_update` siblings.
- `SystemLabel` derives should be replaced with `SystemSet`. You will also need to add the `Debug`, `PartialEq`, `Eq`, and `Hash` traits to satisfy the new trait bounds.
- `with_run_criteria` has been renamed to `run_if`. Run criteria have been renamed to run conditions for clarity, and should now simply return a bool.
- States have been dramatically simplified: there is no longer a "state stack". To queue a transition to the next state, call `NextState::set`

## TODO

- [x] remove dead methods on App and World
- [x] add `App::add_system_to_schedule` and `App::add_systems_to_schedule`
- [x] avoid adding the default system set at inappropriate times
- [x] remove any accidental cycles in the default plugins schedule
- [x] migrate benchmarks
- [x] expose explicit labels for the built-in command flush points
- [x] migrate engine code
- [x] remove all mentions of stages from the docs
- [x] verify docs for States
- [x] fix uses of exclusive systems that use .end / .at_start / .before_commands
- [x] migrate RenderStage and AssetStage
- [x] migrate examples
- [x] ensure that transform propagation is exported in a sufficiently public way (the systems are already pub)
- [x] ensure that on_enter schedules are run at least once before the main app
- [x] re-enable opt-in to execution order ambiguities
- [x] revert change to `update_bounds` to ensure it runs in `PostUpdate`
- [x] test all examples
  - [x] unbreak directional lights
  - [x] unbreak shadows (see 3d_scene, 3d_shape, lighting, transparaency_3d examples)
  - [x] game menu example shows loading screen and menu simultaneously
  - [x] display settings menu is a blank screen
  - [x] `without_winit` example panics
- [x] ensure all tests pass
  - [x] SubApp doc test fails
  - [x] runs_spawn_local tasks fails
  - [x] [Fix panic_when_hierachy_cycle test hanging](alice-i-cecile#120)

## Points of Difficulty and Controversy

**Reviewers, please give feedback on these and look closely**

1.  Default sets, from the RFC, have been removed. These added a tremendous amount of implicit complexity and result in hard to debug scheduling errors. They're going to be tackled in the form of "base sets" by @cart in a followup.
2. The outer schedule controls which schedule is run when `App::update` is called.
3. I implemented `Label for `Box<dyn Label>` for our label types. This enables us to store schedule labels in concrete form, and then later run them. I ran into the same set of problems when working with one-shot systems. We've previously investigated this pattern in depth, and it does not appear to lead to extra indirection with nested boxes.
4. `SubApp::update` simply runs the default schedule once. This sucks, but this whole API is incomplete and this was the minimal changeset.
5. `time_system` and `tick_global_task_pools_on_main_thread` no longer use exclusive systems to attempt to force scheduling order
6. Implemetnation strategy for fixed timesteps
7. `AssetStage` was migrated to `AssetSet` without reintroducing command flush points. These did not appear to be used, and it's nice to remove these bottlenecks.
8. Migration of `bevy_render/lib.rs` and pipelined rendering. The logic here is unusually tricky, as we have complex scheduling requirements.

## Future Work (ideally before 0.10)

- Rename schedule_v3 module to schedule or scheduling
- Add a derive macro to states, and likely a `EnumIter` trait of some form
- Figure out what exactly to do with the "systems added should basically work by default" problem
- Improve ergonomics for working with fixed timesteps and states
- Polish FixedTime API to match Time
- Rebase and merge #7415
- Resolve all internal ambiguities (blocked on better tools, especially #7442)
- Add "base sets" to replace the removed default sets.

.github/contributing/example_style_guide.md

@@ -38,7 +38,7 @@ For more advice on writing examples, see the [relevant section](../../CONTRIBUTI
 4. In Queries, prefer `With<T>` filters over actually fetching unused data with `&T`.
 5. Prefer disjoint queries using `With` and `Without` over param sets when you need more than one query in a single system.
 6. Prefer structs with named fields over tuple structs except in the case of single-field wrapper types.
-7. Use enum-labels over string-labels for system / stage / etc. labels.
+7. Use enum-labels over string-labels for system / schedule / etc. labels.
 
 ## "Feature" examples
 

Cargo.toml

@@ -920,7 +920,7 @@ path = "examples/ecs/removal_detection.rs"
 
 [package.metadata.example.removal_detection]
 name = "Removal Detection"
-description = "Query for entities that had a specific component removed in a previous stage during the current frame"
+description = "Query for entities that had a specific component removed earlier in the current frame"
 category = "ECS (Entity Component System)"
 wasm = false
 
@@ -974,16 +974,6 @@ description = "Illustrates creating custom system parameters with `SystemParam`"
 category = "ECS (Entity Component System)"
 wasm = false
 
-[[example]]
-name = "system_sets"
-path = "examples/ecs/system_sets.rs"
-
-[package.metadata.example.system_sets]
-name = "System Sets"
-description = "Shows `SystemSet` use along with run criterion"
-category = "ECS (Entity Component System)"
-wasm = false
-
 [[example]]
 name = "timers"
 path = "examples/ecs/timers.rs"

benches/benches/bevy_ecs/components/archetype_updates.rs

@@ -1,22 +1,18 @@
-use bevy_ecs::{
-    component::Component,
-    schedule::{Stage, SystemStage},
-    world::World,
-};
+use bevy_ecs::{component::Component, schedule_v3::Schedule, world::World};
 use criterion::{BenchmarkId, Criterion};
 
 #[derive(Component)]
 struct A<const N: u16>(f32);
 
-fn setup(system_count: usize) -> (World, SystemStage) {
+fn setup(system_count: usize) -> (World, Schedule) {
     let mut world = World::new();
     fn empty() {}
-    let mut stage = SystemStage::parallel();
+    let mut schedule = Schedule::new();
     for _ in 0..system_count {
-        stage.add_system(empty);
+        schedule.add_system(empty);
     }
-    stage.run(&mut world);
-    (world, stage)
+    schedule.run(&mut world);
+    (world, schedule)
 }
 
 /// create `count` entities with distinct archetypes
@@ -78,13 +74,13 @@ pub fn no_archetypes(criterion: &mut Criterion) {
     let mut group = criterion.benchmark_group("no_archetypes");
     for i in 0..=5 {
         let system_count = i * 20;
-        let (mut world, mut stage) = setup(system_count);
+        let (mut world, mut schedule) = setup(system_count);
         group.bench_with_input(
             BenchmarkId::new("system_count", system_count),
             &system_count,
             |bencher, &_system_count| {
                 bencher.iter(|| {
-                    stage.run(&mut world);
+                    schedule.run(&mut world);
                 });
             },
         );
@@ -101,12 +97,12 @@ pub fn added_archetypes(criterion: &mut Criterion) {
             |bencher, &archetype_count| {
                 bencher.iter_batched(
                     || {
-                        let (mut world, stage) = setup(SYSTEM_COUNT);
+                        let (mut world, schedule) = setup(SYSTEM_COUNT);
                         add_archetypes(&mut world, archetype_count);
-                        (world, stage)
+                        (world, schedule)
                     },
-                    |(mut world, mut stage)| {
-                        stage.run(&mut world);
+                    |(mut world, mut schedule)| {
+                        schedule.run(&mut world);
                     },
                     criterion::BatchSize::LargeInput,
                 );

benches/benches/bevy_ecs/empty_archetypes.rs

@@ -1,9 +1,4 @@
-use bevy_ecs::{
-    component::Component,
-    prelude::*,
-    schedule::{Stage, SystemStage},
-    world::World,
-};
+use bevy_ecs::{component::Component, prelude::*, world::World};
 use bevy_tasks::{ComputeTaskPool, TaskPool};
 use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion};
 
@@ -80,14 +75,14 @@ fn par_for_each(
     });
 }
 
-fn setup(parallel: bool, setup: impl FnOnce(&mut SystemStage)) -> (World, SystemStage) {
+fn setup(parallel: bool, setup: impl FnOnce(&mut Schedule)) -> (World, Schedule) {
     let mut world = World::new();
-    let mut stage = SystemStage::parallel();
+    let mut schedule = Schedule::new();
     if parallel {
         world.insert_resource(ComputeTaskPool(TaskPool::default()));
     }
-    setup(&mut stage);
-    (world, stage)
+    setup(&mut schedule);
+    (world, schedule)
 }
 
 /// create `count` entities with distinct archetypes
@@ -158,8 +153,8 @@ fn add_archetypes(world: &mut World, count: u16) {
 fn empty_archetypes(criterion: &mut Criterion) {
     let mut group = criterion.benchmark_group("empty_archetypes");
     for archetype_count in [10, 100, 500, 1000, 2000, 5000, 10000] {
-        let (mut world, mut stage) = setup(true, |stage| {
-            stage.add_system(iter);
+        let (mut world, mut schedule) = setup(true, |schedule| {
+            schedule.add_system(iter);
         });
         add_archetypes(&mut world, archetype_count);
         world.clear_entities();
@@ -177,20 +172,20 @@ fn empty_archetypes(criterion: &mut Criterion) {
         e.insert(A::<10>(1.0));
         e.insert(A::<11>(1.0));
         e.insert(A::<12>(1.0));
-        stage.run(&mut world);
+        schedule.run(&mut world);
         group.bench_with_input(
             BenchmarkId::new("iter", archetype_count),
             &archetype_count,
             |bencher, &_| {
                 bencher.iter(|| {
-                    stage.run(&mut world);
+                    schedule.run(&mut world);
                 })
             },
         );
     }
     for archetype_count in [10, 100, 500, 1000, 2000, 5000, 10000] {
-        let (mut world, mut stage) = setup(true, |stage| {
-            stage.add_system(for_each);
+        let (mut world, mut schedule) = setup(true, |schedule| {
+            schedule.add_system(for_each);
         });
         add_archetypes(&mut world, archetype_count);
         world.clear_entities();
@@ -208,20 +203,20 @@ fn empty_archetypes(criterion: &mut Criterion) {
         e.insert(A::<10>(1.0));
         e.insert(A::<11>(1.0));
         e.insert(A::<12>(1.0));
-        stage.run(&mut world);
+        schedule.run(&mut world);
         group.bench_with_input(
             BenchmarkId::new("for_each", archetype_count),
             &archetype_count,
             |bencher, &_| {
                 bencher.iter(|| {
-                    stage.run(&mut world);
+                    schedule.run(&mut world);
                 })
             },
         );
     }
     for archetype_count in [10, 100, 500, 1000, 2000, 5000, 10000] {
-        let (mut world, mut stage) = setup(true, |stage| {
-            stage.add_system(par_for_each);
+        let (mut world, mut schedule) = setup(true, |schedule| {
+            schedule.add_system(par_for_each);
         });
         add_archetypes(&mut world, archetype_count);
         world.clear_entities();
@@ -239,13 +234,13 @@ fn empty_archetypes(criterion: &mut Criterion) {
         e.insert(A::<10>(1.0));
         e.insert(A::<11>(1.0));
         e.insert(A::<12>(1.0));
-        stage.run(&mut world);
+        schedule.run(&mut world);
         group.bench_with_input(
             BenchmarkId::new("par_for_each", archetype_count),
             &archetype_count,
             |bencher, &_| {
                 bencher.iter(|| {
-                    stage.run(&mut world);
+                    schedule.run(&mut world);
                 })
             },
         );

benches/benches/bevy_ecs/scheduling/mod.rs

@@ -1,19 +1,17 @@
 use criterion::criterion_group;
 
 mod run_criteria;
+mod running_systems;
 mod schedule;
-mod stages;
 
 use run_criteria::*;
+use running_systems::*;
 use schedule::*;
-use stages::*;
 
 criterion_group!(
     scheduling_benches,
     run_criteria_yes,
     run_criteria_no,
-    run_criteria_yes_with_labels,
-    run_criteria_no_with_labels,
     run_criteria_yes_with_query,
     run_criteria_yes_with_resource,
     empty_systems,