Add new MouseDrag plugin #2038

Henrique-BO · 2023-07-19T14:00:58Z

🎉 New feature

Closes #306

Summary

Adds a new MouseDrag plugin, which allows the user to exert forces and torques by dragging objects in the scene with the mouse cursor. It has two modes: translation and rotation.

The translation mode is activated by Ctrl+Right-clicking and holding a link in the scene. Dragging the mouse will then move the link towards the mouse position. There is an option on the interface to select whether the force should be applied to the link's center of mass or to the point where the mouse click occured. If center of mass is selected, only a force is applied, with a magnitude given by a constant stiffness and critical damping, scaled by the mass of the link. The force is always contained in a plane parallel to the camera and passing through the application point. If not, an additional torque is applied to account for the offset in the force application point. In this case, the rotation of the object is also slightly damped. The visualization (shown below) shows an arrow from the application point to the target position under the mouse cursor.

The rotation mode is activated by Ctrl+Left-clicking and holding a link in the scene. Dragging the mouse causes a pure torque to be applied, also with a constant stiffness and critical damping and proportional to the link's inertia. The torque is also contained in a plane parallel to the camera. The visualization displays a transparent red bounding box which corresponds to the target orientation of the link.

This approach is different from TransformControl's translation and rotation tools since it uses wrenches to move the objects, so they still interact with the physics of the simulation while being moved.

This plugin uses the /world/<world_name>/wrench topic provided by the ApplyLinkWrench system for applying wrenches. This system is automatically loaded by MouseDrag.

Test it

Open any test world.

gz sim src/gz-sim/examples/worlds/apply_link_wrench.sdf

Load the MouseDrag plugin from the plugin dropdown.

Drag the objects in the scene as you wish.

MouseDrag_demo.mp4

Checklist

Signed all commits for DCO
Added tests
Added example and/or tutorial
Updated documentation (as needed)
Updated migration guide (as needed)
Consider updating Python bindings (if the library has them)
codecheck passed (See contributing)
All tests passed (See test coverage)
While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

Signed-off-by: Henrique-BO <[email protected]>

…mode Signed-off-by: Henrique-BO <[email protected]>

Signed-off-by: Henrique-BO <[email protected]>

…_drag

Signed-off-by: Henrique-BO <[email protected]>

ahcorde · 2023-07-19T14:02:29Z

src/gui/plugins/mouse_drag/MouseDrag.cc

+    public: math::PID rotPid[3];
+
+    /// \brief Spring stiffness for translation, in (m/s²)/m
+    public: double posStiffness = 100;


Suggested change

public: double posStiffness = 100;

public: double posStiffness = 100.0;

ahcorde · 2023-07-19T14:02:35Z

src/gui/plugins/mouse_drag/MouseDrag.cc

+    public: double posStiffness = 100;
+
+    /// \brief Spring stiffness for rotation, in (rad/s²)/rad
+    public: double rotStiffness = 100;


Suggested change

public: double rotStiffness = 100;

public: double rotStiffness = 100.0;

ahcorde · 2023-07-19T14:02:45Z

src/gui/plugins/mouse_drag/MouseDrag.cc

+
+    /// \brief Arrow for visualizing force in translation mode.
+    /// This arrow goes from the application point to the target point.
+    public: rendering::ArrowVisualPtr arrowVisual;


Suggested change

public: rendering::ArrowVisualPtr arrowVisual;

public: rendering::ArrowVisualPtr arrowVisual{nullptr};

ahcorde · 2023-07-19T14:02:56Z

src/gui/plugins/mouse_drag/MouseDrag.cc

+    public: rendering::ArrowVisualPtr arrowVisual;
+
+    /// \brief Box for visualizing rotation mode
+    public: rendering::VisualPtr boxVisual;


Suggested change

public: rendering::VisualPtr boxVisual;

public: rendering::VisualPtr boxVisual{nullptr};

ahcorde · 2023-07-19T14:04:39Z

src/gui/plugins/mouse_drag/MouseDrag.hh

+    public: void Update(const UpdateInfo &_info,
+      EntityComponentManager &_ecm) override;
+
+    /// \brief Callback when echo button is pressed


Suggested change

/// \brief Callback when echo button is pressed

/// \brief Callback when echo button is pressed

/// \param[in] _checked ...

codecov · 2023-07-19T15:11:05Z

Codecov Report

Merging #2038 (0912d4f) into gz-sim7 (8352d3d) will decrease coverage by 0.01%.
The diff coverage is n/a.

❗ Current head 0912d4f differs from pull request most recent head 767e1e6. Consider uploading reports for the commit 767e1e6 to get more accurate results

@@             Coverage Diff             @@
##           gz-sim7    #2038      +/-   ##
===========================================
- Coverage    65.09%   65.09%   -0.01%     
===========================================
  Files          354      354              
  Lines        28733    28732       -1     
===========================================
- Hits         18704    18702       -2     
- Misses       10029    10030       +1

see 2 files with indirect coverage changes

Signed-off-by: Henrique-BO <[email protected]>

Henrique-BO · 2023-07-20T15:25:33Z

Addressed in 316f054

Signed-off-by: Henrique-BO <[email protected]>

azeey · 2023-07-26T20:43:21Z

Works great when the force is applied on the COM. I'm getting weird jitters with the default settings though

MouseDrag.webm

Does the point of application change while dragging? It seemed like it does, but I wasn't sure.

Signed-off-by: Henrique-BO <[email protected]>

Henrique-BO · 2023-08-01T15:13:09Z

The point of application shouldn't change. On my end, I think the problem was because of calculating angle differences close to $2\pi$. I modified the way wrenches are calculated in 965a9e0 and the motion seems better for me. Could you check if the jitters are still there @azeey? You were definetely getting more of it than I was.

azeey · 2023-08-01T16:31:17Z

The point of application shouldn't change. On my end, I think the problem was because of calculating angle differences close to 2π. I modified the way wrenches are calculated in 965a9e0 and the motion seems better for me. Could you check if the jitters are still there @azeey? You were definetely getting more of it than I was.

That made a big difference. It works much better now!!

Signed-off-by: Henrique-BO <[email protected]>

azeey

Works great! Just a few minor comments.

src/gui/plugins/mouse_drag/MouseDrag.cc

azeey · 2023-08-15T15:29:57Z

src/gui/plugins/mouse_drag/MouseDrag.cc

+
+    this->arrowVisual->SetVisible(true);
+    this->boxVisual->SetVisible(false);
+    this->planeVisual->SetVisible(true);


Can we make this configurable? After playing with it for a while, I'm not sure if it's helping a whole lot. What do you think?

I made the plane always visible in 78f86a5, do you think it's more intuitive? Honestly, I'm not loving it either and might just remove it

Yeah, I don't think it's helping. I'd be in favor of removing it.

Signed-off-by: Henrique-BO <[email protected]>

Henrique-BO · 2023-08-16T18:27:46Z

Changes addressed in 78f86a5

azeey · 2023-08-16T20:03:51Z

src/gui/plugins/mouse_drag/MouseDrag.cc

+    try
+    {
+      this->linkId =
+        std::get<uint64_t>(visual->Parent()->UserData("gazebo-entity"));


I got a segfault while testing this and I think it's because visual->Parent() was nullptr. Not sure how I got it to do that, but we should check first.

Here's the backtrace:

#0 __pthread_kill_implementation (no_tid=0, signo=11, threadid=139740024129088) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=11, threadid=139740024129088) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=139740024129088, signo=signo@entry=11) at ./nptl/pthread_kill.c:89 #3 0x00007f1a79642476 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26 #4 0x00007f1a79d86a30 in backward::SignalHandling::sig_handler(int, siginfo_t*, void*) (signo=<optimized out>, info=0x7f17c27faef0, _ctx=<optimized out>) at ~/ws/garden/src/gz-tools/src/Backward/backward.hpp:4268 #5 0x00007f1a79642520 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6 #6 gz::sim::MouseDragPrivate::HandleMouseEvents() (this=0x55d9814481c0) at ~/ws/garden/src/gz-sim/src/gui/plugins/mouse_drag/MouseDrag.cc:677 #7 0x00007f1a2c2cd267 in gz::sim::MouseDrag::eventFilter(QObject*, QEvent*) (this=0x55d980f019f0, _obj=0x55d97d425ab0, _event=0x7f17c27fb5a0) at ~/ws/garden/src/gz-sim/src/gui/plugins/mouse_drag/MouseDrag.cc:314 #8 0x00007f1a700b9b9a in QCoreApplicationPrivate::sendThroughObjectEventFilters(QObject*, QEvent*) () at /lib/x86_64-linux-gnu/libQt5Core.so.5 #9 0x00007f1a70d6c702 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () at /lib/x86_64-linux-gnu/libQt5Widgets.so.5 #10 0x00007f1a700b9e3a in QCoreApplication::notifyInternal2(QObject*, QEvent*) () at /lib/x86_64-linux-gnu/libQt5Core.so.5 #11 0x00007f1a40568f34 in gz::gui::plugins::GzRenderer::BroadcastMousePress() (this=0x55d97dff8bb0) at ~/ws/garden/src/gz-gui/src/plugins/minimal_scene/MinimalScene.cc:523 #12 0x00007f1a40568bbd in gz::gui::plugins::GzRenderer::HandleMouseEvent() (this=0x55d97dff8bb0) at ~/ws/garden/src/gz-gui/src/plugins/minimal_scene/MinimalScene.cc:388 #13 0x00007f1a40568695 in gz::gui::plugins::GzRenderer::Render(gz::gui::plugins::RenderSync*) (this=0x55d97dff8bb0, _renderSync=0x55d97dff8a50) at ~/ws/garden/src/gz-gui/src/plugins/minimal_scene/MinimalScene.cc:336 #14 0x00007f1a40590015 in gz::gui::plugins::RenderThreadRhiOpenGL::RenderNext(gz::gui::plugins::RenderSync*) (this=0x55d97dff8f50, _renderSync=0x55d97dff8a50) at ~/ws/garden/src/gz-gui/src/plugins/minimal_scene/MinimalSceneRhiOpenGL.cc:158 #15 0x00007f1a4056ba91 in gz::gui::plugins::RenderThread::RenderNext(gz::gui::plugins::RenderSync*) (this=0x55d97dff8b80, _renderSync=0x55d97dff8a50) at ~/ws/garden/src/gz-gui/src/plugins/minimal_scene/MinimalScene.cc:745 #16 0x00007f1a4058dc56 in QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<gz::gui::plugins::RenderSync*>, void, void (gz::gui::plugins::RenderThread::*)(gz::gui::plugins::RenderSync*)>::call(void (gz::gui::plugins::RenderThread::*)(gz::gui::plugins::RenderSync*), gz::gui::plugins::RenderThread*, void**) (f=(void (gz::gui::plugins::RenderThread::*)(gz::gui::plugins::RenderThread * const, gz::gui::plugins::RenderSync *)) 0x7f1a4056ba60 <gz::gui::plugins::RenderThread::RenderNext(gz::gui::plugins::RenderSync*) at ~/ws/garden/src/gz-gui/src/plugins/minimal_scene/MinimalScene.cc:744>, o=0x55d97dff8b80, arg=0x7f18a6521918) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:152 #17 0x00007f1a4058dbbd in QtPrivate::FunctionPointer<void (gz::gui::plugins::RenderThread::*)(gz::gui::plugins::RenderSync*)>::call<QtPrivate::List<gz::gui::plugins::RenderSync*>, void>(void (gz::gui::plugins::RenderThread::*)(gz::gui::plugins::RenderSync*), gz::gui::plugins::RenderThread*, void**) (f=(void (gz::gui::plugins::RenderThread::*)(gz::gui::plugins::RenderThread * const, gz::gui::plugins::RenderSync *)) 0x7f1a4056ba60 <gz::gui::plugins::RenderThread::RenderNext(gz::gui::plugins::RenderSync*) at ~/ws/garden/src/gz-gui/src/plugins/minimal_scene/MinimalScene.cc:744>, o=0x55d97dff8b80, arg=0x7f18a6521918) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:185 #18 0x00007f1a4058db02 in QtPrivate::QSlotObject<void (gz::gui::plugins::RenderThread::*)(gz::gui::plugins::RenderSync*), QtPrivate::List<gz::gui::plugins::RenderSync*>, void>::impl(int, QtPrivate::QSlotObjectBase*, QObject*, void**, bool*) (which=1, this_=0x7f18a4389090, r=0x55d97dff8b80, a=0x7f18a6521918, ret=0x0) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:418 #19 0x00007f1a700e741e in QObject::event(QEvent*) () at /lib/x86_64-linux-gnu/libQt5Core.so.5 #20 0x00007f1a70d6c713 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () at /lib/x86_64-linux-gnu/libQt5Widgets.so.5 #21 0x00007f1a700b9e3a in QCoreApplication::notifyInternal2(QObject*, QEvent*) () at /lib/x86_64-linux-gnu/libQt5Core.so.5 #22 0x00007f1a700bcf27 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () at /lib/x86_64-linux-gnu/libQt5Core.so.5 #23 0x00007f1a70113a67 in () at /lib/x86_64-linux-gnu/libQt5Core.so.5 #24 0x00007f1a6e935d3b in g_main_context_dispatch () at /lib/x86_64-linux-gnu/libglib-2.0.so.0 #25 0x00007f1a6e98b258 in () at /lib/x86_64-linux-gnu/libglib-2.0.so.0 #26 0x00007f1a6e9333e3 in g_main_context_iteration () at /lib/x86_64-linux-gnu/libglib-2.0.so.0 #27 0x00007f1a701130b8 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () at /lib/x86_64-linux-gnu/libQt5Core.so.5 #28 0x00007f1a700b875b in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () at /lib/x86_64-linux-gnu/libQt5Core.so.5 #29 0x00007f1a6fecbaf2 in QThread::exec() () at /lib/x86_64-linux-gnu/libQt5Core.so.5 #30 0x00007f1a6feccca1 in () at /lib/x86_64-linux-gnu/libQt5Core.so.5 #31 0x00007f1a79694b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #32 0x00007f1a79726a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Done in 3627480

azeey · 2023-08-16T20:04:39Z

src/gui/plugins/mouse_drag/MouseDrag.cc

+
+    this->arrowVisual->SetVisible(true);
+    this->boxVisual->SetVisible(false);
+    this->planeVisual->SetVisible(true);


Yeah, I don't think it's helping. I'd be in favor of removing it.

Signed-off-by: Henrique-BO <[email protected]>

azeey

Great work!

ahcorde

Some minor comments, but LGTM

ahcorde · 2023-08-16T21:53:16Z

src/gui/plugins/mouse_drag/MouseDrag.cc

+    public: math::Planed plane;
+
+    /// \brief Application point of the wrench in world coordinates
+    math::Vector3d applicationPoint;


nit:

Suggested change

math::Vector3d applicationPoint;

public: math::Vector3d applicationPoint;

ahcorde · 2023-08-16T21:53:38Z

src/gui/plugins/mouse_drag/MouseDrag.cc

+    public: math::Quaterniond initialRot;
+
+    /// \brief Point to which the link is dragged in translation mode
+    math::Vector3d target;


nit:

Suggested change

math::Vector3d target;

public: math::Vector3d target;

ahcorde · 2023-08-16T21:56:26Z

src/gui/plugins/mouse_drag/MouseDrag.cc

+      this->linkId =
+        std::get<uint64_t>(visual->Parent()->UserData("gazebo-entity"));
+    }
+    catch(std::bad_variant_access &e)


#include <variant>

ahcorde · 2023-08-16T21:56:59Z

src/gui/plugins/mouse_drag/MouseDrag.cc

+ *
+*/
+
+#include <mutex>


Suggested change

#include <mutex>

#include <chrono>

#include <mutex>

Done in 66b40af

Signed-off-by: Henrique-BO <[email protected]>

Signed-off-by: Addisu Z. Taddese <[email protected]>

Henrique-BO added 14 commits July 1, 2023 15:14

Create MouseDrag GUI plugin

142d5a1

Signed-off-by: Henrique-BO <[email protected]>

Track mouse position and the clicked link

9449842

Signed-off-by: Henrique-BO <[email protected]>

Implement calculation of plane of wrench application and translation …

508a0ef

…mode Signed-off-by: Henrique-BO <[email protected]>

Implement translation drag by arbitrary point in object

28919d6

Signed-off-by: Henrique-BO <[email protected]>

Implement rotation dragging

603e949

Signed-off-by: Henrique-BO <[email protected]>

Merge branch 'gazebosim:gz-sim7' into mouse_drag

eeade69

Improve rotation calculation and implement velocity damping

1383635

Signed-off-by: Henrique-BO <[email protected]>

Use PID for wrench calculation and implement visualization

0f68570

Signed-off-by: Henrique-BO <[email protected]>

Merge branch 'gazebosim:gz-sim7' into mouse_drag

f1179e8

Merge branch 'mouse_drag' of github.com:Henrique-BO/gz-sim into mouse…

009cdb2

…_drag

Merge branch 'gazebosim:gz-sim7' into mouse_drag

a4dcdff

Improve visualization for rotation

2b36d85

Signed-off-by: Henrique-BO <[email protected]>

Merge branch 'gazebosim:gz-sim7' into mouse_drag

8086225

Fix rotation mode

54a64c0

Signed-off-by: Henrique-BO <[email protected]>

github-actions bot added the 🌱 garden Ignition Garden label Jul 19, 2023

ahcorde requested changes Jul 19, 2023

View reviewed changes

Henrique-BO marked this pull request as ready for review July 19, 2023 14:21

Henrique-BO requested a review from mjcarroll as a code owner July 19, 2023 14:21

Make arrows always visible and minor changes

316f054

Signed-off-by: Henrique-BO <[email protected]>

Henrique-BO added 2 commits July 21, 2023 15:41

Automatically load ApplyLinkWrench system and improve calculations

e7f2ef2

Signed-off-by: Henrique-BO <[email protected]>

Merge branch 'gazebosim:gz-sim7' into mouse_drag

c31bec6

Merge branch 'gazebosim:gz-sim7' into mouse_drag

a639f5c

Henrique-BO mentioned this pull request Jul 27, 2023

Configurable stiffnesses in MouseDrag #2057

Merged

9 tasks

azeey added the beta Targeting beta release of upcoming collection label Jul 31, 2023

Change wrench calculation

965a9e0

Signed-off-by: Henrique-BO <[email protected]>

Henrique-BO added 5 commits August 1, 2023 17:46

Visualizing plane and optimize calculations

7c81964

Signed-off-by: Henrique-BO <[email protected]>

Merge branch 'gazebosim:gz-sim7' into mouse_drag

49de688

Merge branch 'gazebosim:gz-sim7' into mouse_drag

9bc7356

Merge branch 'gazebosim:gz-sim7' into mouse_drag

e6af6c7

Add mutex

f6920d3

Signed-off-by: Henrique-BO <[email protected]>

azeey reviewed Aug 15, 2023

View reviewed changes

Henrique-BO added 2 commits August 16, 2023 14:16

Merge branch 'gazebosim:gz-sim7' into mouse_drag

3fbb38f

Refactoring and minor changes

78f86a5

Signed-off-by: Henrique-BO <[email protected]>

azeey requested changes Aug 16, 2023

View reviewed changes

Remove plane visualization and check visual pointer

3627480

Signed-off-by: Henrique-BO <[email protected]>

azeey approved these changes Aug 16, 2023

View reviewed changes

azeey requested a review from ahcorde August 16, 2023 21:38

ahcorde approved these changes Aug 16, 2023

View reviewed changes

Henrique-BO and others added 2 commits August 16, 2023 19:13

Minor fixes

66b40af

Signed-off-by: Henrique-BO <[email protected]>

Fix compiler warning

767e1e6

Signed-off-by: Addisu Z. Taddese <[email protected]>

azeey enabled auto-merge (squash) August 17, 2023 01:52

azeey merged commit caefed8 into gazebosim:gz-sim7 Aug 17, 2023

Henrique-BO deleted the mouse_drag branch August 17, 2023 13:29

Henrique-BO mentioned this pull request Sep 24, 2023

Add pendulum for ApplyForceTorque and MouseDrag demo gazebosim/harmonic_demo#19

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new MouseDrag plugin #2038

Add new MouseDrag plugin #2038

Henrique-BO commented Jul 19, 2023 •

edited

Loading

ahcorde Jul 19, 2023

ahcorde Jul 19, 2023

ahcorde Jul 19, 2023

ahcorde Jul 19, 2023

ahcorde Jul 19, 2023

codecov bot commented Jul 19, 2023 •

edited

Loading

Henrique-BO commented Jul 20, 2023

azeey commented Jul 26, 2023

Henrique-BO commented Aug 1, 2023

azeey commented Aug 1, 2023

azeey left a comment

azeey Aug 15, 2023

Henrique-BO Aug 16, 2023

azeey Aug 16, 2023

Henrique-BO commented Aug 16, 2023

azeey Aug 16, 2023

Henrique-BO Aug 16, 2023

azeey Aug 16, 2023

azeey left a comment

ahcorde left a comment

ahcorde Aug 16, 2023

ahcorde Aug 16, 2023

ahcorde Aug 16, 2023

ahcorde Aug 16, 2023

Henrique-BO Aug 16, 2023

	public: double posStiffness = 100;
	public: double posStiffness = 100.0;

	public: double rotStiffness = 100;
	public: double rotStiffness = 100.0;

	public: rendering::ArrowVisualPtr arrowVisual;
	public: rendering::ArrowVisualPtr arrowVisual{nullptr};

	public: rendering::VisualPtr boxVisual;
	public: rendering::VisualPtr boxVisual{nullptr};

	/// \brief Callback when echo button is pressed
	/// \brief Callback when echo button is pressed
	/// \param[in] _checked ...

	math::Vector3d applicationPoint;
	public: math::Vector3d applicationPoint;

Add new MouseDrag plugin #2038

Add new MouseDrag plugin #2038

Conversation

Henrique-BO commented Jul 19, 2023 • edited Loading

🎉 New feature

Summary

Test it

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 19, 2023 • edited Loading

Codecov Report

Henrique-BO commented Jul 20, 2023

azeey commented Jul 26, 2023

Henrique-BO commented Aug 1, 2023

azeey commented Aug 1, 2023

azeey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Henrique-BO commented Aug 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

azeey left a comment

Choose a reason for hiding this comment

ahcorde left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Henrique-BO commented Jul 19, 2023 •

edited

Loading

codecov bot commented Jul 19, 2023 •

edited

Loading