Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No Frames Received & D455 go offline frequently #11515

Closed
Wilson-shuo opened this issue Mar 3, 2023 · 30 comments
Closed

No Frames Received & D455 go offline frequently #11515

Wilson-shuo opened this issue Mar 3, 2023 · 30 comments

Comments

@Wilson-shuo
Copy link

  • Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)


Required Info
Camera Model D400
Firmware Version 05.13.00.50
Operating System & Version Linux (Ubuntu 18.04)
Kernel Version (Linux Only) JETPACK 4.5.1
Platform NVIDIA Jetson
SDK Version 2.50
Language C++
Segment 3D measure

Issue Description

THIS CAMERA ARE DRIVING ME MAD QAQ.....................

I got 3 x realsense D455 connect to nvidia jetson nano via usb hub.

First of all, the usb hub got individual power support, I am sure that every camera got good power support.

Second, My project open D455 one by one, always close this one before next open streaming. So the situation that three cameras are turned on at the same time does not exist.

Everything seems ok but when I capture from three cameras one by one,something goes wrong.

It have two outcomes.

1.All realsense cameras "offline",It means u can not see them both via realsense-viewer or "lsusb"

2.You can see them online, also u can find them in realsense-viewer, but color frame goes wrong and "no frames arrived"

2775d51cafcae0c772caa75bc34ac9f

So in my programme i will got rs2::error no device connect or rs2::error frames time out.

These fatal errors have made my programme unusable.

How to avoid frames time out or camera offline?????Need help!

@Wilson-shuo
Copy link
Author

if realsense-viewer can not receive color frame,it pop out tons error like this:

04/03 03:01:08,552 ERROR [548435464208] (librealsense-exception.h:52) xioctl(VIDIOC_G_CTRL) failed Last Error: Invalid argument
04/03 03:01:08,583 ERROR [548435464208] (librealsense-exception.h:52) xioctl(VIDIOC_G_CTRL) failed Last Error: Invalid argument
04/03 03:01:08,617 ERROR [548435464208] (librealsense-exception.h:52) xioctl(VIDIOC_G_CTRL) failed Last Error: Invalid argument
04/03 03:01:08,648 ERROR [548435464208] (librealsense-exception.h:52) xioctl(VIDIOC_G_CTRL) failed Last Error: Invalid argument
04/03 03:01:08,715 ERROR [548435464208] (librealsense-exception.h:52) xioctl(VIDIOC_G_CTRL) failed Last Error: Invalid argument
04/03 03:01:08,750 ERROR [548435464208] (librealsense-exception.h:52) xioctl(VIDIOC_G_CTRL) failed Last Error: Invalid argument
04/03 03:03:24,716 ERROR [548435464208] (librealsense-exception.h:52) xioctl(VIDIOC_STREAMON) failed for buf_type=1 Last Error: Cannot allocate memory
04/03 03:03:31,792 WARNING [548120019392] (ds5-timestamp.cpp:69) UVC metadata payloads not available. Please refer to the installation chapter for details.
04/03 03:03:34,009 ERROR [548435464208] (librealsense-exception.h:52) xioctl(VIDIOC_STREAMON) failed for buf_type=1 Last Error: Cannot allocate memory
04/03 03:03:50,770 ERROR [548435464208] (librealsense-exception.h:52) xioctl(VIDIOC_STREAMON) failed for buf_type=1 Last Error: Cannot allocate memory
04/03 03:03:54,238 ERROR [548435464208] (librealsense-exception.h:52) xioctl(VIDIOC_STREAMON) failed for buf_type=1 Last Error: Cannot allocate memory

@MartyG-RealSense
Copy link
Collaborator

Hi @Wilson-shuo The xioctl(VIDIOC_G_CTRL) errors indicate that there may be a conflict with the kernel. What method did you use to install the librealsense SDK on your Jetson, please?

@Wilson-shuo
Copy link
Author

I installed SDK by clone source code from github, makeinstall after cmake.

@MartyG-RealSense
Copy link
Collaborator

Installation of the SDK on Jetson boards from source code requires the application of a dedicated kernel patch for Jetson by running a script called ./scripts/patch-realsense-ubuntu-L4T.sh

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md

If the SDK is built from source code with CMake using the RSUSB Backend installation method detailed at #6964 (comment) then the kernel is bypassed and it is not necessary to apply the patch.

@Wilson-shuo
Copy link
Author

Installation of the SDK on Jetson boards from source code requires the application of a dedicated kernel patch for Jetson by running a script called ./scripts/patch-realsense-ubuntu-L4T.sh

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md

If the SDK is built from source code with CMake using the RSUSB Backend installation method detailed at #6964 (comment) then the kernel is bypassed and it is not necessary to apply the patch.

Hi! Deeply thanks for u reply.

I have tried this shell script, after that follow u instruction to install SDK again.

I found compile with “CUDA ON” do help with latency(I noticed that timestamp no more go deep red in realsense-viewer)

But the realsense camera still keeps go offline....................

Most of the situations is color frame do not arrive in 5s.

Also if one camera"s color frame time out, It will keep time out all the time no matter you reboot jetson how many times.

We could not plug out camera everytime, cause our device set on the ceiling.

Even use "handware reset" can not solve the problem.

Any other suggestions?

@Wilson-shuo
Copy link
Author

Size 214016 out of 814335 bytes (26%)
06/03 13:50:17,553 WARNING [547004346816] (backend-v4l2.cpp:1062) Incomplete frame received: Incomplete video frame detected!
Size 459776 out of 814335 bytes (56%)
06/03 13:50:17,621 WARNING [547004346816] (backend-v4l2.cpp:1062) Incomplete frame received: Incomplete video frame detected!
Size 590848 out of 814335 bytes (72%)
06/03 13:50:17,623 WARNING [547264389568] (backend-v4l2.cpp:1062) Incomplete frame received: Incomplete video frame detected!
Size 279552 out of 814335 bytes (34%)
06/03 13:50:17,685 WARNING [547004346816] (backend-v4l2.cpp:1062) Incomplete frame received: Incomplete video frame detected!
Size 525312 out of 814335 bytes (64%)
06/03 13:50:17,686 WARNING [547247604160] (backend-v4l2.cpp:1062) Incomplete frame received: Incomplete video frame detected!
Size 214016 out of 814335 bytes (26%)
06/03 13:50:17,794 WARNING [547255996864] (backend-v4l2.cpp:1062) Incomplete frame received: Incomplete video frame detected!
Size 246784 out of 814335 bytes (30%)
06/03 13:50:17,818 WARNING [546987561408] (backend-v4l2.cpp:1062) Incomplete frame received: Incomplete video frame detected!
Size 279552 out of 814335 bytes (34%)
06/03 13:50:25,879 WARNING [547264389568] (backend-v4l2.cpp:1127) Frames didn't arrived within 5 seconds
06/03 13:50:25,904 WARNING [547222426048] (backend-v4l2.cpp:1127) Frames didn't arrived within 5 seconds
06/03 13:50:25,904 WARNING [547004346816] (backend-v4l2.cpp:1127) Frames didn't arrived within 5 seconds
06/03 13:50:25,909 WARNING [547247604160] (backend-v4l2.cpp:1127) Frames didn't arrived within 5 seconds
06/03 13:50:25,929 WARNING [546987561408] (backend-v4l2.cpp:1127) Frames didn't arrived within 5 seconds
06/03 13:50:27,018 ERROR [546979168704] (librealsense-exception.h:52) set_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,018 ERROR [546979168704] (global_timestamp_reader.cpp:239) Error during time_diff_keeper polling: set_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,031 ERROR [546953990592] (librealsense-exception.h:52) get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,032 ERROR [546953990592] (error-handling.cpp:78) Error during polling error handler: get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,032 ERROR [546870129088] (librealsense-exception.h:52) get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,054 ERROR [546870129088] (ds5-thermal-monitor.cpp:94) Error during thermal compensation handling: get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,135 ERROR [546735911360] (librealsense-exception.h:52) get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,136 ERROR [546735911360] (error-handling.cpp:78) Error during polling error handler: get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,136 ERROR [547272782272] (librealsense-exception.h:52) set_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,137 ERROR [547272782272] (global_timestamp_reader.cpp:239) Error during time_diff_keeper polling: set_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,142 ERROR [546727518656] (librealsense-exception.h:52) get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,142 ERROR [546727518656] (ds5-thermal-monitor.cpp:94) Error during thermal compensation handling: get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,267 ERROR [547502670272] (librealsense-exception.h:52) get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,267 ERROR [547502670272] (ds5-thermal-monitor.cpp:94) Error during thermal compensation handling: get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,268 ERROR [546685555136] (librealsense-exception.h:52) set_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,269 ERROR [546685555136] (global_timestamp_reader.cpp:239) Error during time_diff_keeper polling: set_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device
06/03 13:50:27,269 ERROR [547373429184] (librealsense-exception.h:52) get_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device

@MartyG-RealSense
Copy link
Collaborator

The Incomplete video frame detected! warning is highly rare but more likely to occur on Jetson boards such as the one that you are using. #10379 offers advice for resolving it.

@Wilson-shuo
Copy link
Author

The Incomplete video frame detected! warning is highly rare but more likely to occur on Jetson boards such as the one that you are using. #10379 offers advice for resolving it.

Ok thanks, any more idea about color frame do not arrive in 5 seconds? we tried the kernel patch but it does not work.

@Wilson-shuo
Copy link
Author

The Incomplete video frame detected! warning is highly rare but more likely to occur on Jetson boards such as the one that you are using. #10379 offers advice for resolving it.

For the "Incomplete frame", I open 3 x depth streams (848x480,5FPS) and 3 x color streams(848x480,5FPS) ,after a while all camera will offline and I could not find them both in "lsusb" and realsense-viewer. I will try u suggestion.

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Mar 7, 2023

If the cameras behave normally at first but time-out or disconnect later, it is possible that repeated stop-start of the three cameras may be causing a memory leak. This is where the available memory capacity is progressively consumed until a program becomes unstable.

You can test for a memory leak in Ubuntu by using a system monitoring tool such as htop and watching the available memory in real-time whilst the program is running. If the amount of memory is declining then this would be an indication that a memory leak is occurring on each stop of the camera and using up a further chunk of memory, thus destabilizing the program.

@Wilson-shuo
Copy link
Author

If the cameras behave normally at first but time-out or disconnect later, it is possible that repeated stop-start of the three cameras may be causing a memory leak. This is where the available memory capacity is progressively consumed until a program becomes unstable.

You can test for a memory leak in Ubuntu by using a system monitoring tool such as htop and watching the available memory in real-time whilst the program is running. If the amount of memory is declining then this would be an indication that a memory leak is occurring on each stop of the camera and using up a further chunk of memory, thus destabilizing the program.

thank you! I will pay attention to memory leaks. However, the current problem is that if the camera is turned on once and turned off, the color frame will time out the next time when the camera is turned on. If it is a memory leak problem, it should be OK to turn on the camera after ”reboot“ jetson. But the fact is that the color frame still times out after ”reboot“.

@MartyG-RealSense
Copy link
Collaborator

Are all three cameras sharing the same pipeline and you have not created multiple pipelines (one for each camera)?

If you are using the same pipeline for all three cameras, are you identifying the cameras by their unique serial number so that the program knows which camera it is activating when one has been stopped and another of the three is started?

For example, after a Stop() instruction, use a cfg configuration instruction to define the serial number of the next camera to be activated and then Start() with that configuration, making sure to place the config / cfg name in the brackets so that the script knows to use the defined configuration.

pipe.stop();

cfg.enable_device(serial);

pipe.start(cfg);

Where 'serial' contains the serial number of the camera to be activated next, or you can hard-code the serial number into the brackets.

cfg.enable_device('123456789');

@Wilson-shuo
Copy link
Author

Are all three cameras sharing the same pipeline and you have not created multiple pipelines (one for each camera)?

If you are using the same pipeline for all three cameras, are you identifying the cameras by their unique serial number so that the program knows which camera it is activating when one has been stopped and another of the three is started?

For example, after a Stop() instruction, use a cfg configuration instruction to define the serial number of the next camera to be activated and then Start() with that configuration, making sure to place the config / cfg name in the brackets so that the script knows to use the defined configuration.

pipe.stop();

cfg.enable_device(serial);

pipe.start(cfg);

Where 'serial' contains the serial number of the camera to be activated next, or you can hard-code the serial number into the brackets.

cfg.enable_device('123456789');

My code under blow:(c++)

`m_pipeline_cfg.enable_device(SN_Number_1) // use cfg to open one camera
m_pipeline_cfg.enable_stream(RS2_STREAM_COLOR, 848, 480, RS2_FORMAT_BGR8, 5);
m_pipeline_cfg.enable_stream(RS2_STREAM_DEPTH, 848, 480, RS2_FORMAT_Z16, 5);

m_profile = m_pipeline.start(m_pipeline_cfg);
frameset frames = m_pipeline.wait_for_frames();
depth_frame depth = frames.get_depth_frame();
frame color = frames.get_color_frame();

//do some process to color frame and depth frame.......................

m_pipeline_cfg.disable_all_streams();
m_pipeline.stop();

//stop this camera then open the next.................

m_pipeline_cfg.enable_device(SN_Number_2) // use cfg to open one camera
m_pipeline_cfg.enable_stream(RS2_STREAM_COLOR, 848, 480, RS2_FORMAT_BGR8, 5);
m_pipeline_cfg.enable_stream(RS2_STREAM_DEPTH, 848, 480, RS2_FORMAT_Z16, 5);
//etc...............................`

@Wilson-shuo
Copy link
Author

I uploaded my D455 device time-out video to youtube. I hope you could have a glance.
Here is the video link:https://youtu.be/VVEhKYpY8pQ
It is totally unusable,Even I reboot jetson.

@MartyG-RealSense
Copy link
Collaborator

Thanks very much for the code listing.

You should not need to use m_pipeline_cfg.disable_all_streams(); as the pipeline stop instruction will close all the streams on that camera anyway.

The No Frames Received problem that your video demonstrates can occur if another program that is using the camera streams is already running. For example, if you run a program that you have created yourself and then afterwards open realsense-viewer and try to enable the depth or color stream then No Frames Received is displayed. If the first program is closed and then a stream in the realsense-viewer is activated, the stream enables.

The reverse is also true. If you first enable streams in realsense-viewer and then launch another program, the realsense-viewer streams will work but the streams in the secondly-launched program will not.

Below, this effect is demonstrated by launching rs-capture first and then secondly trying to enable RGB in realsense-viewer.

image

The reason why this occurs is that by the operating rules of the RealSense SDK, once a stream is enabled on a particular camera then the program that enabled it places a 'claim' on that stream. This prevents another program from accessing the same stream on that camera until the stream is disabled in the first program and the claim is released, allowing another program to use it.

Can you confirm whether you have no other RealSense program running when you run your C++ program or run realsense-viewer, please?

@Wilson-shuo
Copy link
Author

Thanks very much for the code listing.

You should not need to use m_pipeline_cfg.disable_all_streams(); as the pipeline stop instruction will close all the streams on that camera anyway.

The No Frames Received problem that your video demonstrates can occur if another program that is using the camera streams is already running. For example, if you run a program that you have created yourself and then afterwards open realsense-viewer and try to enable the depth or color stream then No Frames Received is displayed. If the first program is closed and then a stream in the realsense-viewer is activated, the stream enables.

The reverse is also true. If you first enable streams in realsense-viewer and then launch another program, the realsense-viewer streams will work but the streams in the secondly-launched program will not.

Below, this effect is demonstrated by launching rs-capture first and then secondly trying to enable RGB in realsense-viewer.

image

The reason why this occurs is that by the operating rules of the RealSense SDK, once a stream is enabled on a particular camera then the program that enabled it places a 'claim' on that stream. This prevents another program from accessing the same stream on that camera until the stream is disabled in the first program and the claim is released, allowing another program to use it.

Can you confirm whether you have no other RealSense program running when you run your C++ program or run realsense-viewer, please?

I am sure that only one program is using realsense-sdk. My program will start at system boot up, but it will not open camera or do any operation until 20 mins later. My program is managed by a 20 min timer. So in my youtube viedo I use "ctl + c" killed my program immediately after reboot,then immediately launched realsense-viewer,So it is impossible to get multiple program use one sdk api at the sametime.

@Wilson-shuo
Copy link
Author

bug.mp4

I could show u this video. you can see that only realsense-viewer itself control all streams, after a while suddenly color frames go time out but depth frames go well as normal.

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Mar 9, 2023

Looking at your YouTube video again and its warning message about the V4L2 backend, and also your mention of using the L4T shell script during installation, it looks as though you are installing the SDK from source code and building the SDK in V4L2 backend mode. This mode relies on the Jetson kernel to operate correctly.

When performance with a kernel-based SDK build is unstable, building from source code with the RSUSB backend mode can often resolve the problem as it bypasses the kernel and so is not dependent on Linux versions or kernel versions. Whilst RSUSB is not as ideally suited to multiple camera applications as the V4L2 backend, it can work with multicam. And as you are starting and stopping individual cameras rather than planning to have three cameras active simultaneously, RSUSB backend should be fine.

You can build in RSUSB backend mode by using the CMake build flag -DFORCE_RSUSB_BACKEND=true and not using the L4T shell script (as you do not need to patch the kernel with an RSUSB build).


Also, if using a powered USB hub would be inconvenient for your project, you could first try using a passive USB hub (one that draws its power from the computer instead of the mains electricity socket). A USB hub should have a different brand of USB controller to the ones that the built-in ports use, so this can help to confirm whether a USB instability on built-in ports is caused by the particular brand of USB controller that the computer's ports use.

@Wilson-shuo
Copy link
Author

Looking at your YouTube video again and its warning message about the V4L2 backend, and also your mention of using the L4T shell script during installation, it looks as though you are installing the SDK from source code and building the SDK in V4L2 backend mode. This mode relies on the Jetson kernel to operate correctly.

When performance with a kernel-based SDK build is unstable, building from source code with the RSUSB backend mode can often resolve the problem as it bypasses the kernel and so is not dependent on Linux versions or kernel versions. Whilst RSUSB is not as ideally suited to multiple camera applications as the V4L2 backend, it can work with multicam. And as you are starting and stopping individual cameras rather than planning to have three cameras active simultaneously, RSUSB backend should be fine.

You can build in RSUSB backend mode by using the CMake build flag -DFORCE_RSUSB_BACKEND=true and not using the L4T shell script (as you do not need to patch the kernel with an RSUSB build).

Also, if using a powered USB hub would be inconvenient for your project, you could first try using a passive USB hub (one that draws its power from the computer instead of the mains electricity socket). A USB hub should have a different brand of USB controller to the ones that the built-in ports use, so this can help to confirm whether a USB instability on built-in ports is caused by the particular brand of USB controller that the computer's ports use.

eeda87ba8c928f4efc9102f822db0643.mp4

Change USB backend seems helpless.......

Still get no frames arrived......

1678605362287

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Mar 12, 2023

Looking at your latest video, the log error Failed to read busnum/devnum is a highly rare error. On the occasions in the past when it has been reported, it has almost always been an issue that occurs on very small-sized single-board devices and is related to insufficient power being provided to the camera by the USB connection.

If the ports on the Nano are working correctly then it could also suggest a possible problem with the USB cable being used with the camera. Are you using the official 1 meter cable supplied with the camera or a longer USB cable of your own choice, please?

As you have a problem with three cameras rather than one, a damaged USB cable is unlikely to be the cause otherwise the issue would occur with only one camera.

@Wilson-shuo
Copy link
Author

We connect camera by USB-fibre-optical device. Our camera is connected to the USB-to-fiber module with independent power supply. The network signal is transmitted to the Jetson nano 50m away through the fiber, and then converted to USB.

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Mar 12, 2023

It is possible to have a RealSense camera up to 100 meters away from the computer when using fiber-optic cable, as demonstrated by the premium-quality USB equipment supplier Newnex in the video below, by using an optical repeater device (a Firenex 5000H in the video).

https://www.youtube.com/watch?v=GLQgR1jT04M&t=129s

The Firenex 5000H product details are here:

https://www.newnex.com/usb-3-1-optical-fiber-extender-repeater.php

Is your USB-to-fiber module an optical repeater, please?

@Wilson-shuo
Copy link
Author

254709a05e02807661af338a6f95a57

Yes,We use a fiber module to convert usb signal to light signal, so it could transport more than 50m. then convert light signal back to usb signal and plug into jetson nano.

@Wilson-shuo
Copy link
Author

On RUSB backend,We found that "Handware reset" on realsense-viewer could remove the “timeout status” of the realsense device. Could I use Cpp code to “Handware reset” a specific device(by SN number)?

@MartyG-RealSense
Copy link
Collaborator

#9287 (comment) has a C++ script for looping through all attached RealSense cameras and performing hardware_reset() on them.

There is only a Python example available at #5428 (comment) regarding resetting a specific device by serial number, though the C++ equivalent should be similar.

@Wilson-shuo
Copy link
Author

Hi we found a new bug.
When use RUSB backend, one of the three cameras will go offline. can not find it via realsense-viewer.
Through "lsusb" we can see 3x VIA Lab device,but only 2x Intel device. Reboot does not work.

1678946253718

@Wilson-shuo
Copy link
Author

We think that might be a random error,until both jetson nano goes wrong.

1678948061508

@MartyG-RealSense
Copy link
Collaborator

Does the missing camera come back if you perform a full shutdown (not reset) and then boot-up with the power button?

@MartyG-RealSense
Copy link
Collaborator

Hi @Wilson-shuo Do you require further assistance with this case, please? Thanks!

@MartyG-RealSense
Copy link
Collaborator

Case closed due to no further comments received.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants