Video update rate and cuda are not working.

[IuliuNovac]

Bug report

Issue details

I am changing the camera settings for the simulation but enabling cuda and changing update rate are not working. When enabeling cuda, <use_cuda>true</use_cuda>, there is no more video output. When changing the <update_rate>20</update_rate>, there is no change in FPS. The image resolution changes without issues.

I am using an NVIDIA L4 gpu, with NVIDIA L4 VRAM. There is barely any usage.

This is the configuration file.

  <sensor name="camera" type="camera">
    <pose>0 0 0 -1.57 -1.57 0</pose>
    <camera>
      <horizontal_fov>2.0</horizontal_fov>
      <image>
        <width>32</width>
        <height>24</height>
      </image>
      <clip>
        <near>0.05</near>
        <far>15000</far>
      </clip>
    </camera>
    <always_on>1</always_on>
    <update_rate>20</update_rate>
    <visualize>1</visualize>

    <plugin filename="CameraZoomPlugin"
        name="CameraZoomPlugin">
      <max_zoom>125.0</max_zoom>
      <slew_rate>0.42514285714</slew_rate>
    </plugin>
    
    <plugin name="GstCameraPlugin"
        filename="GstCameraPlugin">
      <udp_host>127.0.0.1</udp_host>
      <udp_port>5600</udp_port>
      <use_basic_pipeline>true</use_basic_pipeline>
      <use_cuda>true</use_cuda>
    </plugin>

  </sensor>

Gsteamer command

gst-launch-1.0 -v udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! rtph264depay ! avdec_h264 ! videoconvert ! fpsdisplaysink video-sink=autovideosink sync=false

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Got context from element 'autovideosink0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage/GstGLImageSink:sink: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage: sync = false
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
/GstPipeline:pipeline0/GstUDPSrc:udpsrc0.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
New clock: GstSystemClock
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264

Logs

[Msg] GstCameraPlugin: image topic [world/iris_runway/model/iris_with_gimbal/model/gimbal/link/pitch_link/sensor/camera/image]
[Msg] GstCameraPlugin: enable topic [world/iris_runway/model/iris_with_gimbal/model/gimbal/link/pitch_link/sensor/camera/image/enable_streaming]
[Dbg] [CameraSensor.cc:480] Enabling camera sensor: 'iris_with_gimbal::gimbal::pitch_link::camera' data generation.
[Msg] GstCameraPlugin:: streaming: started
[Dbg] [GstCameraPlugin.cc:407] GstCameraPlugin: creating generic pipeline
[Dbg] [GstCameraPlugin.cc:477] Using Cuda
[Msg] GstCameraPlugin: GStreamer element set state returned: 2
[Msg] GstCameraPlugin: starting GStreamer main loop

Version
Master branch

Platform
[ ] All
[ ] AntennaTracker
[X] Copter
[ ] Plane
[ ] Rover
[ ] Submarine

[IuliuNovac]

The solution to increase the fps is running with higher frequency with -z 1000.

The cuda pipeline seems to be odd. Could someone explain how to set the right pipeline for cuda?

GST_DEBUG=3 gst-launch-1.0 -v udpsrc port=5600 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! rtph264depay ! h264parse ! identity silent=false ! avdec_h264 ! videoconvert ! nvh264enc bitrate=800 preset=1 ! fpsdisplaysink video-sink=autovideosink sync=false

0:00:03.143219509 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.152016330 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.161007796 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.169893667 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.178655136 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.186627007 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.195297441 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.205585473 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.214132605 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.222478166 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.240257989 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.251110211 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.259886638 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.268951976 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.277225666 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.285682907 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.295426409 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.305170705 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.314307800 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.323525971 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.332682051 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.341781994 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.350402596 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.359266195 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.368698366 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.377041382 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
0:00:03.385405786 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped
^Chandling interrupt.
0:00:03.394794017 18601 0x5e52a3941860 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 367 will be dropped

[prodrone-liu]

I am also running the same command with use_cuda set to true. But GstCameraPlugin failed in my case. May I ask about your configurations of cuda?
Logs:

[Msg] GstCameraPlugin:: streaming: started
[Dbg] [GstCameraPlugin.cc:407] GstCameraPlugin: creating generic pipeline
[Dbg] [GstCameraPlugin.cc:477] Using Cuda
(gz sim server:8009): GLib-GObject-CRITICAL **: 11:26:59.773: g_object_set: assertion 'G_IS_OBJECT (object)' failed
[Err] [GstCameraPlugin.cc:419] GstCameraPlugin: failed to create GStreamer elements
[Msg] GstCameraPlugin: starting GStreamer main loop