ign-gazebo3/main homebrew INTEGRATION test failures (segfaults)

[jacobperron]

Environment

• OS Version: macOS
• Source or binary build?
  3465bbe

Description

Three new test failures over the last couple days: https://build.osrfoundation.org/job/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/113/testReport/

The follow two started today:

INTEGRATION_each_new_removed (fails with exception)

        Start  67: INTEGRATION_each_new_removed
67: Test command: /Users/jenkins/workspace/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/build/bin/INTEGRATION_each_new_removed "--gtest_output=xml:/Users/jenkins/workspace/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/build/test_results/INTEGRATION_each_new_removed.xml"

67: Test timeout computed to be: 240

67: Running main() from /Users/jenkins/workspace/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/ign-gazebo/test/gtest/src/gtest_main.cc

67: [==========] Running 1 test from 1 test suite.

67: [----------] Global test environment set-up.

67: [----------] 1 test from EachNewRemovedFixture

67: [ RUN      ] EachNewRemovedFixture.EachNewEachRemovedInSystem

67/150 Test  #67: INTEGRATION_each_new_removed .........................Subprocess aborted***Exception:   1.08 sec

INTEGRATION_performer_detector (fails with segfault)

103: [ RUN      ] PerformerDetectorTest.HandlesRemovedParentModel
103: [Err] [PerformerDetector.cc:104] Performer detector[detector1] has an empty  with an associated  of [no_value]. This  will be ignored.
103: [Err] [PerformerDetector.cc:98] Performer detector[detector1] has an empty  with an associated  of [no_key]. This  will be ignored.
103/150 Test #103: INTEGRATION_performer_detector .......................***Exception: SegFault  1.85 sec

There are a two recent changes to the ign-gazebo3 branch, though I'm not sure if either of them is responsible for the recent failures:

• Add laser_retro support #603
• Fix pose of plane visual with non-default normal vector #574

This third failure started two days ago:

INTEGRATION_breadcrumbs (fails with segfault)

55: [ RUN      ] BreadcrumbsTest.FuelDeploy
55: [Msg] Loading SDF world file[/Users/jenkins/workspace/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/ign-gazebo/test/worlds/breadcrumbs.sdf].
55: Warning [Model.cc:212] Non-unique names detected in XML children of model with name[vehicle_blue].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [2251689575469537287] / [ign_gazebo_components.World].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [13994732549916512682] / [ign_gazebo_components.Name].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [12592746352568925681] / [ign_gazebo_components.Gravity].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [13224937992534617849] / [ign_gazebo_components.MagneticField].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [3630648173860223239] / [ign_gazebo_components.PhysicsEnginePlugin].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [17459188283658606303] / [ign_gazebo_components.RenderEngineServerPlugin].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [8705992680619689917] / [ign_gazebo_components.RenderEngineGuiPlugin].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [8753193699724811771] / [ign_gazebo_components.Wind].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [12173050716021724529] / [ign_gazebo_components.WorldLinearVelocity].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [15943768124495574352] / [ign_gazebo_components.WorldLinearVelocitySeed].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [3297509811873971798] / [ign_gazebo_components.ParentEntity].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [17100615127981600159] / [ign_gazebo_components.Scene].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [17605309075052480649] / [ign_gazebo_components.Atmosphere].
55: [Dbg] [LevelManager.cc:195] Reading performer info
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [4425034272867445569] / [ign_gazebo_components.Performer].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [11311330372155309806] / [ign_gazebo_components.PerformerLevels].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [17121648710877364109] / [ign_gazebo_components.Geometry].
55: [Msg] Created performer [3 / perf1]
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [8064491505919932473] / [ign_gazebo_components.Level].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [2668898242563798256] / [ign_gazebo_components.DefaultLevel].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [11371360182141354106] / [ign_gazebo_components.LevelEntityNames].
55: [Dbg] [Physics.cc:600] Loaded [ignition::physics::dartsim::Plugin] from library [/usr/local/Cellar/ignition-physics2/2.3.0_2/lib/ign-physics-2/engine-plugins/libignition-physics-dartsim-plugin.dylib]
55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Physics] for entity [1]
55: [Msg] Create service on [/world/breadcrumbs/create]
55: [Msg] Remove service on [/world/breadcrumbs/remove]
55: [Msg] Pose service on [/world/breadcrumbs/set_pose]
55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::UserCommands] for entity [1]
55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::SceneBroadcaster] for entity [1]
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [4981278897826323946] / [ign_gazebo_components.WorldSdf].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [6687176221774458630] / [ign_gazebo_components.Model].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [6612894081701502240] / [ign_gazebo_components.Pose].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [8546580419506082455] / [ign_gazebo_components.Static].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [9712747055438129860] / [ign_gazebo_components.WindMode].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [5661073481138181711] / [ign_gazebo_components.SelfCollide].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [11683062252779233161] / [ign_gazebo_components.SourceFilePath].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [5081358965268446661] / [ign_gazebo_components.Link].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [8112400427272910195] / [ign_gazebo_components.Inertial].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [16454635107327670381] / [ign_gazebo_components.Visual].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [13011964647677164955] / [ign_gazebo_components.CastShadows].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [13440282432131634483] / [ign_gazebo_components.Transparency].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [9853217982010720764] / [ign_gazebo_components.Material].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [17938588655714334139] / [ign_gazebo_components.Collision].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [9225962031573086509] / [ign_gazebo_components.CollisionElement].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [10522242218202596205] / [ign_gazebo_components.CanonicalLink].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [360894313363162387] / [ign_gazebo_components.Joint].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [4514298940308304267] / [ign_gazebo_components.JointType].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [16752981946841796574] / [ign_gazebo_components.JointAxis].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [6791213906290467105] / [ign_gazebo_components.ThreadPitch].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [14863055383328686534] / [ign_gazebo_components.ParentLinkName].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [3888602368311486678] / [ign_gazebo_components.ChildLinkName].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [11536476718181283925] / [ign_gazebo_components.ModelSdf].
55: [Msg] DiffDrive subscribing to twist messages on [/model/vehicle_blue/cmd_vel]
55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::DiffDrive] for entity [5]
55: [Msg] Breadcrumbs subscribing to deploy messages on [/model/vehicle_blue/breadcrumbs/B1/deploy]
55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5]
55: [Msg] Breadcrumbs subscribing to deploy messages on [/model/vehicle_blue/breadcrumbs/B2/deploy]
55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5]
55: [Msg] Breadcrumbs subscribing to deploy messages on [/fuel_deploy]
55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5]
55: [Msg] Breadcrumbs subscribing to deploy messages on [/model/vehicle_blue/breadcrumbs/B1_perf/deploy]
55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5]
55: [Msg] Breadcrumbs subscribing to deploy messages on [/model/vehicle_blue/breadcrumbs/B1_perf_large_volume/deploy]
55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5]
55: [Msg] Breadcrumbs subscribing to deploy messages on [/no_rename_deploy]
55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5]
55: [Msg] Breadcrumbs subscribing to deploy messages on [/rename_deploy]
55: [Dbg] [SimulationRunner.cc:775] Loaded system [ignition::gazebo::systems::Breadcrumbs] for entity [5]
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [3866641186784191835] / [ign_gazebo_components.Light].
55: [Msg] Loaded level [4]
55: [Msg] Serving world controls on [/world/breadcrumbs/control] and [/world/breadcrumbs/playback/control]
55: [Msg] Serving GUI information on [/world/breadcrumbs/gui/info]
55: [Msg] World [breadcrumbs] initialized with [1ms] physics profile.
55: [Msg] Serving world SDF generation service on [/world/breadcrumbs/generate_world_sdf]
55: [Msg] Serving world names on [/gazebo/worlds]
55: [Msg] Resource path add service on [/gazebo/resource_paths/add].
55: [Msg] Resource path get service on [/gazebo/resource_paths/get].
55: [Msg] Resource paths published on [/gazebo/resource_paths].
55: [Msg] Found no publishers on /stats, adding root stats topic
55: [Msg] Found no publishers on /clock, adding root clock topic
55: [Dbg] [SimulationRunner.cc:422] Creating PostUpdate worker threads: 4
55: [Dbg] [SimulationRunner.cc:435] Creating postupdate worker thread (0)
55: [Dbg] [SimulationRunner.cc:435] Creating postupdate worker thread (1)
55: [Dbg] [SimulationRunner.cc:435] Creating postupdate worker thread (2)
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [14423792621074650228] / [ign_gazebo_components.JointVelocityCmd].
55: [Dbg] [EntityComponentManager.cc:632] Using components of type [9436740692853731264] / [ign_gazebo_components.JointPosition].
55: [Msg] Serving scene information on [/world/breadcrumbs/scene/info]
55: [Msg] Serving graph information on [/world/breadcrumbs/scene/graph]
55: [Msg] Serving full state on [/world/breadcrumbs/state]
55: [Msg] Serving full state (async) on [/world/breadcrumbs/state_async]
55: [Msg] Publishing scene information on [/world/breadcrumbs/scene/info]
55: [Msg] Publishing entity deletions on [/world/breadcrumbs/scene/deletion]
55: [Msg] Publishing state changes on [/world/breadcrumbs/state]
55: [Msg] Publishing pose messages on [/world/breadcrumbs/pose/info]
55: [Msg] Publishing dynamic pose messages on [/world/breadcrumbs/dynamic_pose/info]
55: [Msg] Deploying B2_0 at -2.2 2 5.525 -0 0 0
55: [Msg] Deploying B2_1 at -2.2 2 5.525 -0 0 0
55: [Msg] Deploying B2_2 at -2.2 2 5.525 -0 0 0
55: [Msg] Deploying B2_3 at -2.2 2 5.525 -0 0 -0�[1
 55/150 Test  #55: INTEGRATION_breadcrumbs ..............................***Exception: SegFault  4.58 sec

Correlates to #609, but I'm just speculating that they may be related.

[j-rivero]

I've detected some more today:

The following tests FAILED:
	 61 - INTEGRATION_breadcrumbs (SEGFAULT)
	 75 - INTEGRATION_each_new_removed (SEGFAULT)
	143 - INTEGRATION_user_commands (SEGFAULT)

[j-rivero]

Current list of segfaults observed in main branch:

The following tests FAILED:
	 61 - INTEGRATION_breadcrumbs (SEGFAULT)
	 73 - INTEGRATION_diff_drive_system (SEGFAULT)
	103 - INTEGRATION_level_manager (SEGFAULT)
	141 - INTEGRATION_scene_broadcaster_system (SEGFAULT)
	151 - INTEGRATION_user_commands (SEGFAULT)

[chapulina]

#807 has a potential solution

[jacobperron]

More segfaulting tests from yesterday

	 59 - INTEGRATION_ackermann_steering_system (SEGFAULT)
	119 - INTEGRATION_logical_audio_sensor_plugin (SEGFAULT)
	159 - INTEGRATION_user_commands (SEGFAULT)

https://build.osrfoundation.org/job/ignition_gazebo-ci-main-homebrew-amd64/79

[Blast545]

Grepping our last 30 days jobs at https://build.osrfoundation.org/job/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/ we find the following Integration tests flakily segfaulting on MacOS:

ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|221|INTEGRATION_pose_publisher_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|219|INTEGRATION_diff_drive_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|219|INTEGRATION_velocity_control_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|212|INTEGRATION_pose_publisher_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|212|INTEGRATION_touch_plugin
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|211|INTEGRATION_altimeter_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|209|INTEGRATION_level_manager
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|208|INTEGRATION_diff_drive_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|208|INTEGRATION_level_manager
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|207|INTEGRATION_network_handshake
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|206|INTEGRATION_pose_publisher_system
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|205|INTEGRATION_logical_audio_sensor_plugin
ignition_gazebo-ci-ign-gazebo3-homebrew-amd64|205|INTEGRATION_performer_detector

I'm opening a PR to disable them on MacOS for now.

[azeey]

I think most of these segfaults have been fixed by #1836. I'll go ahead and close this.