libusb and libuvc errors when using multiple devices on Linux

[spschul]

Describe the bug
When streaming with multiple devices on Linux, when the program calls start_cameras the program will fail with errors such as an error for libusb_submit_transfer and similar-looking errors. Diving deeper, it becomes clear that a function within libusb is failing, returning ENOMEM indicating that the kernel is out of memory. The problem is that there's a setting (readable at /sys/module/usbcore/parameters/usbfs_memory_mb) that limits the amount of memory available for USB IO. However, some applications that require more intensive usage of that memory often use more than that. As documented at OpenKinect/libfreenect2#97, there is a fix for this: allocate more memory. I tried it with 32mb instead of 16 and it seems to be working (I ran into issues the first time I tested it but they seemed unrelated; documenting in case someone else runs into this).

@wes-b I'd like to discuss how to handle this; right now I'm leaning towards having a section on it in the docs soon and somewhere early, perhaps even recommending that users preemptively add more memory.
Do we want to attempt to make this fix a part of the package? I think that is too much, and also we don't know how much memory people will have to spare, but also don't know how much memory they'll need. Using many cameras will require more memory, etc.

To fix on AMD64, using 32 mb instead of 16,

1. Edit /etc/default/grub, replacing the line that says GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" with GRUB_CMDLINE_LINUX_DEFAULT="quiet splash usbcore.usbfs_memory_mb=32"
2. Run sudo update-grub
3. Restart the computer

To fix on ARM64 (Jetson Nano)

1. Edit /boot/extlinux/extlinux.conf
2. Change APPEND ${cbootargs} quiet to APPEND ${cbootargs} usbcore.usbfs_memory_mb=128 usbcore.autosuspend=-1. (It is probably fine to leave quiet and remove usbcore.autosuspend=-1 but that was not tested.)
3. Save changes & reboot
4. confirm changes with cat /sys/module/usbcore/parameters/usbfs_memory_mb

To Reproduce
Stream from 2 devices on Linux

Expected behavior
Programs do not crash in this case

Desktop (please complete the following information):

• OS with Version: Ubuntu 18.04
• SDK Version: 1.1.1
• Firmware version: n/a

[spschul]

After discussion with @wes-b here's a few considerations.

There is a maximum on memory allowed for the libusb pool, specified with the environment variable K4A_MAX_LIBUSB_POOL. For Windows, K4A_MAX_LIBUSB_POOL=80000000 but for Linux, K4A_MAX_LIBUSB_POOL=1000000. This limit was chosen for Windows to have enough memory to not drop frames, but on Linux it needed to be smaller because of the default size of usbfs_memory_mb. After finding this fix, it's clear that we can support doing larger libusb_pool sizes.

If anyone has issues with dropping frames, this could potentially be a solution.

Currently, we know that 16MB of memory for usbfs_memory_mb is sufficient with 1 camera on Linux with K4A_MAX_LIBUSB_POOL=1000000. So, we can conclude that as an upper bound 16MB * (number of cameras) allocated for usbfs_memory_mb should be sufficient.

However, if you go ahead and set K4A_MAX_LIBUSB_POOL=8000000, which will probably help with not dropping frames, you should instead set usbfs_memory_mb to 16MB * 8 * (number of cameras) = 128MB * (number of cameras).

Note that these are upper bounds; they are probably overkill. It would be nice to find a more precise specification.

We should also add checks to the project on relevant calls to libusb and libuvc to check for these considerations.

[somacoder]

@spschul and @wes-b this appears to eliminate the libusb errors for me. Great catch! However, I then stumble into another error. Are you getting the following error after as well? This is from the master Kinect after setting up the subordinate. If not, I can provide more details. I'm having trouble pinpointing it and whether or not it's related, and am hesitant to open a separate issue for this yet.

[2019-07-12 17:09:57.789] [error] [t=2996] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/color/uvc_camerareader.cpp (153): Start(). Failed to start streaming: Unknown error
[2019-07-12 17:09:57.789] [error] [t=2996] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/color/color.cpp (194): color->m_spCameraReader->Start(width, height, fps, config->color_format, &color_capture_available, color) returned failure in color_start()
[2019-07-12 17:09:57.789] [error] [t=2996] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/sdk/k4a.c (725): color_start(device->color, config) returned failure in k4a_device_start_cameras()
Runtime error: k4a_device_start_cameras(device, device_config) returned 1

[spschul]

This looks unrelated. If you create a new issue, we'd be happy to take a look!

[ruoyuwangeel4930]

@spschul Hi Steve, thanks for helping out! I have changed /etc/default/grub according to your instruction. Now my /etc/default/grub looks like this:

However, I still cannot use two devices at the same time. Here is the error message:

My system is Ubuntu18.04 and I used apt-get to install the SDK.
Both devices are good when I test them separately. Do you have any idea about the possible cause to this problem? Thanks!

[spschul]

Ok so after running sudo update-grub and restarting, you're still having issues? It's hard to say without more info

Is it possible to set the K4A_LOG_LEVEL environment variable to "i", try again, and post the output? This should enable more logging so the errors are more specific.

Also, are you running k4aviewer to get these results? If not, try running two instances of k4aviewer at once.

Hope this helps get us down the path to a solution!

[ruoyuwangeel4930]

@spschul I was trying to run two k4aviewers. The first one runs normally. For the second one, the device can be detected, but cannot be started:

After I set K4A_LOG_LEVEL=i, I saw this information in the terminal:

[pfagomez]

I got a similar issue when using only one device:

k4aviewer libusb: error [_get_usbfs_fd] libusb couldn't open USB device /dev/bus/usb/004/017: Permission denied libusb: error [_get_usbfs_fd] libusb requires write access to USB device nodes. libusb: error [_get_usbfs_fd] libusb couldn't open USB device /dev/bus/usb/004/017: Permission denied libusb: error [_get_usbfs_fd] libusb requires write access to USB device nodes.

after running
sudo chmod -R 777 /dev/bus/usb/004
the error disappeared, however it reappears everytime I reconnect the device.

[wes-b]

@ruoyuwangeel4930 is seems like something may be using that device. Can you open them individually? Streaming LED turns on when the camera is running, so you can us that to confirm both cameras are working as expected. The error you shared is a failure to claim the USB interface. This is strange because you have been able to open the handle to the device at this point

[ruoyuwangeel4930]

@pfagomez I think your problem can be addressed by doing this:

https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/develop/docs/usage.md#linux-device-setup

[ruoyuwangeel4930]

@wes-b I opened the two devices individually, and they both work well. This is what I got from the log:

[2019-07-26 16:27:43.680] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/usbcommand/usbcommand.c (285): find_libusb_device(). Container ID found: {338cf61a-bca9-455c-08b1-8a6fc81ff9cd} 
[2019-07-26 16:27:43.680] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/usbcommand/usbcommand.c (185): populate_serialnumber(). Serial Number found 000073492312
[2019-07-26 16:27:43.779] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/usbcommand/usbcommand.c (285): find_libusb_device(). Container ID found: {338cf61a-bca9-455c-08b1-8a6fc81ff9cd} 
[2019-07-26 16:27:43.779] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/usbcommand/usbcommand.c (185): populate_serialnumber(). Serial Number found 000073492312
[2019-07-26 16:27:43.807] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (243): ******************** Device Info ********************
[2019-07-26 16:27:43.807] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (244): K4A SDK version:     1.1.1
[2019-07-26 16:27:43.832] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (250): Serial Number:       000073492312
[2019-07-26 16:27:43.832] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (254): RGB Sensor Version:  1.6.98
[2019-07-26 16:27:43.832] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (257): Depth Sensor Version:1.6.70
[2019-07-26 16:27:43.832] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (260): Mic Array Version:   1.6.14
[2019-07-26 16:27:43.832] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (263): Sensor Config:       6109.7
[2019-07-26 16:27:43.832] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (266): Build type:          Release
[2019-07-26 16:27:43.832] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (271): Signature type:      MSFT
[2019-07-26 16:27:43.832] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (277): ****************************************************
[2019-07-26 16:27:43.920] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/usbcommand/usbcommand.c (285): find_libusb_device(). Container ID found: {41a7bbea-b1d3-4658-e794-9687d66a8f2a} 
[2019-07-26 16:27:43.921] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/usbcommand/usbcommand.c (185): populate_serialnumber(). Serial Number found 000148192412
[2019-07-26 16:27:44.019] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/usbcommand/usbcommand.c (285): find_libusb_device(). Container ID found: {41a7bbea-b1d3-4658-e794-9687d66a8f2a} 
[2019-07-26 16:27:44.019] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/usbcommand/usbcommand.c (185): populate_serialnumber(). Serial Number found 000148192412
[2019-07-26 16:27:44.047] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (243): ******************** Device Info ********************
[2019-07-26 16:27:44.047] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (244): K4A SDK version:     1.1.1
[2019-07-26 16:27:44.072] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (250): Serial Number:       000148192412
[2019-07-26 16:27:44.072] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (254): RGB Sensor Version:  1.6.98
[2019-07-26 16:27:44.072] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (257): Depth Sensor Version:1.6.70
[2019-07-26 16:27:44.072] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (260): Mic Array Version:   1.6.14
[2019-07-26 16:27:44.072] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (263): Sensor Config:       6109.7
[2019-07-26 16:27:44.072] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (266): Build type:          Release
[2019-07-26 16:27:44.072] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (271): Signature type:      MSFT
[2019-07-26 16:27:44.072] [info] [t=18130] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (277): ****************************************************
malloc_consolidate(): invalid chunk size
Aborted (core dumped)

It threw "malloc_consolidate(): invalid chunk size", after I closed the k4aviewer.

[spschul]

It's possible although not certain that the problem when you closed the viewer could be related to #353, but I don't think that it's your issue.

Could you provide a log like the above log, but of the failing k4aviewer when you run two k4aviewer instances and one of them fails?

Thanks!

[ichlubna]

Using Arch Linux I had the same problem. The grub fix solved the problem of program crashing when using two devices but I am still getting this message:

libusb: error [submit_bulk_transfer] submiturb failed error -1 errno=12
[2019-08-26 15:01:54.396] [error] [t=4782] ../src/usbcommand/usbstreaming.c (223): usb_cmd_lib_usb_thread(). No libusb transfers could not be submitted, error:LIBUSB_ERROR_IO
[2019-08-26 15:01:56.051] [error] [t=4783] ../src/capturesync/capturesync.c (141): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:    184422 type:Color
[2019-08-26 15:01:56.118] [error] [t=4783] ../src/capturesync/capturesync.c (141): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:    251055 type:Color
[2019-08-26 15:01:56.182] [error] [t=4783] ../src/capturesync/capturesync.c (141): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:    317722 type:Color
....

The usb error is coming after calling start_cameras on the second device. (disappears if I call stop_cameras on the first device before that)

The first Kinect streams fine but the second returns empty image.

BTW running two k4aviewer instances works fine.

EDIT: My mistake, as @spschul wrote, 32 mb might not be enough. Used 256 and the problem is gone!

[wes-b]

Added Arm64 instructions to the issue information.