Top usability issues from Fall 2022 manipulation projects

[RussTedrake]

I am combing back through the long list of discussions with students this project season, and wanted to call out the hottest hot spots all in one place. I will reference existing issues and am happy to triage these out into new issues, too. But I wanted to provide a wholistic view at least once.

• Getting object models into Drake.
  • Meshcat's support seemed sufficient (I believe it successfully loaded all objs... including even one that was semi-transparent! and it supports stl/dae as well), but since Drake only officially supprts obj, users had to convert to that format. => Document recommendations for converting mesh assets into drake-compatible formats #19479
  • RgbdSensor was a real bottleneck. One student said "i downloaded like a hundred objs from the internet, but less than 10% of them worked!". I'll follow up to get specific failure cases. Related to Rendering OBJ files with sophisticated materials #11949 .
  • We need to make it easier to run convex decomposition (https://github.com/gizatt/convex_decomp_to_sdf helped in many cases), if we cannot run it automatically. => Document recommendations for converting mesh assets into drake-compatible formats #19479
  • parser package paths were mysterious and loading error messages could be more helpful. "Since our project requires importing a scooper object, we were surprised by the difficulty of setting up the package.xml and other boilerplate necessary to import it into the simulation. It ended up being simple to put in, but we couldn’t figure it out on our own before asking for help." Update: the parser interface has come a long way now, with url support, etc.
• Lots of people authored SDF files with centers of mass outside the collision geometry. the results was that they thought the contact simulation was terrible, and started to lose faith in the simulator. But actually, there was e.g. a swinging mass far below the ground. (Visualize inertia ellipsoids #17662)
• ycb meshes not in the pip wheels really frustrated some folks. (pip install: Fetch platform-independent data  #15774)
• People still saw SystemExit: condition 'Delta >0' failed. (py mbp: Getting hard exit from "Failure at external/drake/multibody/plant/tamsi_solver.cc:217 in SolveQuadraticForTheSmallestPositiveRoot(): condition 'Delta > 0' failed."? #14590) . I thought Test cases for throwing exception if distal body translates and has zero/NaN mass or rotates and has zero/NaN rotational inertia (towards issue #17081). #17557 was supposed to help, but at least in RL examples it still appeared (and gives absolutely no help to the user). Update: hopefully resolved in [multibody] Improve error message for !(Delta > 0) #19407.
• Lots of people got confused by system wiring. [systems/framework] More verbose error message in DiagramBuilder #18446 is an attempt to fix. We should look for more places where the messages could be more helpful. The most common errors were trying to connect directly to a port in the manipulation station subdiagram from the parent diagram, or trying to connect to the controller_plant (which is built and passed to the InverseDynamicsController, etc, but never registered directly with a builder). I think javascript rendering of system diagrams #13874 could make a huge difference.
• RgbdSensor crashing when there is no display (RgbdSensor segfaults when there is no display #18445). To my knowledge, this only hit once, I think, but hit hard.
• Lots of request regarding wanting to add/remove bodies post-finalize (a well known issue). Others wanted to adjust diagram features after diagram.Build(); [primitives] Add TrajectorySource::UpdateTrajectory #18434 addressed one acute case.
• A few people got tripped up the scary pybind memory issue discussed in this slack thread -- we return a reference to the output cache's value which python can overwrite. That thread was closed with WON'T FIX. But it tripped a few people badly; specifically with the point cloud objects returned from DepthImageToPointCloud.
• People needed help finding Drake's sdfformat / urdf supported tags in doxygen. In particular, when they wanted to adjust friction, they tried standard urdf/sdf tags and saw no result.
• Effort/velocity limits are not respected in the default solver TAMSI. (MultibodyPlant::AddJointActuator() documentation should clarify effort limit is unused #14735)
• People saw RuntimeError: Error control wants to select step smaller than minimum allowed (1.737e-14) and didn't know why. Most often it was because their controller came close to singularity. That one is hard to fix in the general case, I think.
• Many people set data (e.g. multibody positions) in the wrong context; and generally don't know how to debug in the systems framework ([tutorials] Debugging a dynamical system #17249).
• A number of people wanted Python type hints. => Python type hints #19480.

Again, I will try to clean / curate these. But this is a first pass.

[buoyancy99]

We need more documentation. Having the documentations below shall answer 60% of the student questions.

• More tutorials explaining the concept in MultibodyPlants, including Diagram, Plant, Context, LeafSystem, Port etc
• Annotated example code of adding robot, objects, controller, cameras from scratch (AddManipulationStation seems to be a great one, but it's nicer to be made into a tutorial)
• Annotated example code of adding a few different controllers to robot and chain ports
• Annotated example of initialize object locations with SetFreeBodyPose after diagram is finalized, example of initialize robot joints such that end effector is at desired initial position after diagram is finalized
• Annotated example of chaining multiple Systems
• Annotated example code of adding a LeafSystem (e.g. Planner in clutter picking)
• When simulation stops due to stability, print links to debugging resources (make sim step smaller, add sphere to corner of cube etc.)
• High level APIs and documentation for manipulation such as "Planner" class in clutter clearing demo which takes in source and target end-effector positions (and maybe collision bodies in the scene) and output a trajectory (or joint command)
• After these examples are added, update drake tutorial on website such that we have some page pointing to all MultiBodyPlant/Robotic Manipulation resources

[ggould-tri]

Having spent a couple of days fighting with different obj issues in the meldis pipeline:

• The biggest issue I had was the lack of error feedback. obj -- especially when accompanied by mtl and png -- is hard to get right, doubly so when it interacts with bazel; when the only error is is Ignoring Meshcat http request -- or just a missing or all-black visualization -- then even when it's because of an invalid file or user error, the user can't remediate it regardless.
• The error messages are made harder to debug by the design of meldis: Lambdas hung off of hooks means error messages are nonlocal and lack stack traces to the actual logical point of origin (likely on the other end of a network connection). We're definitely missing some error return path or other.
• Even flagging what process+thread generated the error and what file it was processing when it did so would be helpful.

FYI: glTF can but does not necessarily fix this problem: It has the option for the external data to be included directly inline, but does not require it. Still, it would be an improvement.

[jwnimmer-tri]

People still saw SystemExit: condition 'Delta >0' failed #14590.

I thought #17557 was supposed to help ...

The #17557 was just a unit-test-only lemma. No user behavior changed.

[RussTedrake]

Within reason, I've connected all of the issues on my list above with feature requests or fixes. Will close this roll-up in favor of tracking the individual requests.