Tools 2.0

[alanjfs]

A discussion topic regarding the current implementation of the "tools" in the example application.

• Download for Windows

Goal

Overcome limitations of the current implementation.

1. Tools are called every frame, even when not "active"
2. Tools are free functions, to avoid issues with type when storing a _activeTool in the application
3. Tools do not support multiple inputs in parallel, e.g. wacom tablet + mouse movements
4. Tools need metadata, like a label and type, which are currently independent
5. Tools can only manipulate the next frame, i.e. does nothing (useful) during pause
6. Tools could be entities, but are not
7. Inputs are assigned to entities being manipulated, should maybe be assigned to the tool instead?

Overview of Current Implementation

The user interacts with the world using "tools".

• Tools.inl example
• Tool activation example
• Tool called here
• Current tool assigned here

A tool doesn't immediately modify any data, instead a tool generates events. Events are then interpreted by the application - via an "event handler" - which in turn modify your data. The goal is being able to play back the events and reproduce exactly what the user did with each tool.

Tool     Event    Application
 _    
| |        _
| |------>| |        _
| |       | |------>| |
|_|       | |       | |
          |_|       | |
                    | |
                    | |
                    |_|

The application has a notion of an "Active Tool" which is called once per frame.

sequentity/Example/Source/main.cpp

Lines 615 to 617 in f5196fb

	// Handle any input coming from the above drawScene()
	_activeTool.write();

The tool does nothing, unless there is an entity with a particular component called Active along with some "input".

sequentity/Example/Source/Tools.inl

Lines 140 to 144 in f5196fb

	static void TranslateTool() {
	// Handle press input of type: 2D range, relative anything with a position
	Registry.view<Name, Activated, InputPosition2D, Color, Position>().each([](
	auto entity,

The Active component is assigned by the UI layer, in this case ImGui, whenever an entity is clicked.

sequentity/Example/Source/main.cpp

Lines 571 to 583 in f5196fb

	if (ImGui::IsItemActivated()) {
	Registry.assign<Activated>(entity, sqty.current_time);
	Registry.assign<InputPosition2D>(entity, absolutePosition, relativePosition);
	}
	else if (ImGui::IsItemActive()) {
	Registry.assign<Active>(entity);
	Registry.assign<InputPosition2D>(entity, absolutePosition, relativePosition);
	}
	else if (ImGui::IsItemDeactivated()) {
	Registry.assign<Deactivated>(entity);
	}

These are the three states of any entity able to be manipulated with a Tool.

• Activated entity has transitioned from being passive to active, this happens once
• Active entity is activated, and being manipulated (e.g. dragged)
• Deactivated entity transitioned from active back to passive, happens once

Thoughts

Overall I'm looking for thoughts on the current system, I expect similar things have been done lots of times, perhaps with the exception of wanting to support (1) multiple inputs in parallel, e.g. two mice and (2) wanting the user to ultimately assign an arbitrary input to arbitrary tools, e.g. swap from the mouse affecting position to the Wii controller.

Ultimately, the application is meant to facilitate building of a physical "rig", where you have a number of physical input devices, each affecting some part of a character or world. Like a marionettist and her control bar.

Code wise, there are a few things I like about the current implementation, and some I dislike.

Likes

• Tristate I like the Activated, Active and Deactivated aspect; I borrowed that from ImGui which seem to work quite nicely and is quite general.
• Events rule I like that tools only have a single responsibility, which is to generate events. Which means I could generate events from anywhere, like from mouse move events directly, and it would integrate well with the sequencer and application behavior. =
• Encapsulation I also like that because events carry the final responsibility, manipulating events is straightforward and intuitive, and serialising those to disk is unambiguous.
• Generic inputs And I like how inputs are somewhat general, but I'll touch on inputs in another issue to keep this one focused on tools.

Dislikes

• UI and responsibility I don't like the disconnect between how Active is assigned from the UI layer
• Inputs on the wrong entities I don't like how inputs e.g. InputPosition2D are associated with entities like "hip", "leftLeg" etc. rather than a tool itself, which seems more intuitive.

[alanjfs]

Another concern with this method is that there is currently no notion or separation between input devices. Currently, ImGui decides whether or not an input qualifies as active, by whether an ImGui widget is made active or not. This currently can only come from mouse events I think, but I'd like to find room for detecting whether it's coming from an Xbox controller or a Wacom tablet.

So I was considering establishing a notion of a Device ID.

// Poll for all sorts of inputs
for (const auto& [device, input] : inputs) {
  if (device == DeviceID::Mouse) { do_something(input) }
  if (device == DeviceID::WacomPen) { ... }
  if (device == DeviceID::XBoxCtrl1) { ... }
}

But things get blurry real fast. :( Is there any reference to something of this nature somewhere?

[alanjfs]

One more thing that is made difficult with the current implementation.

When you hold K and click + drag, the "Scrub" tool is enabled to scrub the current time. But because a tool can only operate on entities with an Active component, it means you can only scrub the global time when you click + drag on one of the squares.

This problem extends to tools that would perform e.g. relative changes. For example, I'd like for middle-click drag to enable a RelativeTranslate tool which would move whichever entity has a Selected component. The goal being to avoid having to hit the object as it's moving around, which can be a bit of a challenge.

But that wouldn't work either, without dragging from one of the squares which defeats the purpose of the tool.

So these Active et. al. can't just be based on which entity ImGui says we're currently clicking on.