Real-time application with two K4A devices has too much latency per frame

[StevenButner]

I am developing real-time robotics application hosted on Ubuntu Linux 18.04LTS.
I am currently using the Azure Kinect SDK K4A library version 1.2.0 with matching 1.2 firmware loaded on both devices.

I created a screen display for my host computer with millisecond-resolution system timestamp. Then I pointed both cameras so that they can simultaneously capture an RGB image of the system display. The two cameras are hardware synchronized and I've developed a method that, once calibrated, can produce a host timestamp that approximately matches the K4A's frame timestamp with my host's realtime timestamp. In developing this method, I discovered that there is approximately 140 milliseconds of latency somewhere within the K4A acquisition pipeline. This is more latency than we'd like to have in our real-time control system.

My system can capture depath and color or IR image frames at 5Hz (nominally) or at 15Hz for testing. It's interesting that we see the same latency when capturing at 5Hz than we do when capturing at 15 Hz .... even though the inter-frame times for 15Hz are significantly less than this latency.

My question is whether there are any parameters of my system or within the K4A library that might help to reduce the 140msec frame latency that I'm observing? Below is the output of the Azure Kinect DK Firmware Tool ... showing my device details and firmware versions.

== Azure Kinect DK Firmware Tool ==
Device Serial Number: 000261592512
Current Firmware Versions:
RGB camera firmware: 1.6.102
Depth camera firmware: 1.6.75
Depth config file: 6109.7
Audio firmware: 1.6.14
Build Config: Production
Certificate Type: Microsoft

Device Serial Number: 000441592912
Current Firmware Versions:
RGB camera firmware: 1.6.102
Depth camera firmware: 1.6.75
Depth config file: 6109.7
Audio firmware: 1.6.14
Build Config: Production
Certificate Type: Microsoft

Any pointers to additional documentation or advice for reducing latency would be greatly appreciated.
Steve Butner

[wes-b]

I would suggest

• Make sure you are using device timestamp, not system. Device timestamp is captured at a hardware level and synchronized across multiple devices. This is start of exposure when multiple devices are used, so this time calculation will include the sensors raw read time.
• Test with the depth camera off. This will use less USB bandwidth and reduce its impact on the issue.
• Use the smallest RGB resolution. This too will use less USB bandwidth and reduce its impact on the issue.
• If you need to use both color and depth, ensure synchronized_images_only is false so images are not held by the sdk.
• Try 30 fps – agree it is strange for the latency to be the same at 15 & 5.

[wes-b]

Also do latency testing with a single device. Please see #530 to issues with multi sync we working through.

In single device mode, the device timestamp is center of exposure rather than beginning when using multi device settings.

[StevenButner]

Thanks for the response and for your second update about #530 and other issues relating to sync. I sent this comment originally via reply email ... but since it didn't appear here on GitHub, I'm now cutting and pasting it here. Apologies if you are getting it multiple times.

My ultimate system will have two cameras and it will be important to run them in a synchronized mode. I am able to try various other scenarios (for development/debug), but for the moment, my 2 cameras do seem to be nicely sync'd. Because my test setup has a millisecond-level timestamp displayed in real-time on a monitor with both of the Kinect devices viewing that same monitor, I can tell if the images they capture are from different times or not. I'm consistently seeing very good agreement in the captured images.

Some of your suggestions are relatively easy to try and some are not so easy. Because depth is our system's primary need, we always must be capturing that. Additionally, my test setup uses the captured RGB image to visually verify the synchronization as well as the timestamping. So, ..... turning the depth off (which I have tried) is easy, but it cannot be that way in our final system. With 720P RGB images at 15Hz and 512x512 depth info (wide field of view mode) and using the K4A's hardware timestamps, I still see latencies on the order of approximately 135 milliseconds.

When I turn off acquisition of depth frames as you suggest, I get very similar latency results(!). From this I conclude that the USB bandwidth likely isn't the limiting factor that is determining the latency(?). You also recommended using the smallest RBG resolution .... and I have been doing that. We are using the 720P (1280x720) color mode.

As to the decision on which Kinect frame timestamp to use (i.e., "system" or "hardware"), naturally I'd like to use the hardware one for better accuracy. For our ultimate robotic system, however, the cameras can be running as long as a couple of hours at a time. In such a scenario, we are concerned about timing drift since there are differing time bases in the Kinect devices as opposed to our host's Intel-Core-i7 CPU. I don't know the amount of drift in the current system but I do know that there will be a timing drift in any asynchronous system like this. As such, we will almost certainly have to use the Kinect's "system" frame timestamp since it will derive its timing from the same CPU timebase that the rest of our system uses. I took a look at some statistics on the two frame timestamps and found that the "system" timestamp has about +/-660 microseconds of jitter, but the average frame inter-arrival times otherwise agree strongly with the "hardware" frame timestamps.

The timestamps that I'm putting on the final depth and/or RGB frames are computed in my driver-adaptation layer. The computation is simply the addition of a constant offset that represents the difference in timing epoch between each Kinect device and our host's system time. The code that I've developed to adapt the Kinects to our robotic system tries very hard (shortly after the cameras have started streaming and have synchronized) to call k4a_device_get_capture() in a loop with carefully chosen timeouts to ensure that all available queued-up captures have been acquired. Once we know that the capture queue is dry, we request one more capture with a longer timestamp (set a bit larger than the frame-to-frame timing). When that capture completes successfully, we immediately do the timestamp basis computation and thereafter the timing basis is never changed. As each frame comes in, its timestamp (we can use either the "system" or the "hardware" frame timestamps via a compile-time choice) is grabbed and that particular Kinect's time basis is added to it to yield the final timestamp for that given frame for my system. Each Kinect device has its own unique timestamp basis number (computed as described above).

For testing, I've been logging all of the timestamps and then checking that the time displayed in the captured image matches as closely as possible with the "computed" timestamp. The results are quite good (provided I use a fast enough monitor!), but the mystery that precipitated my original inquiry still remains: Can you speculate as to what factor(s) cause the latency of captured frames (based on the captured image frame's timestamp) to be nearly 140 milliseconds earlier than the time when we actually get the capture? And, why does this 140 msec latency seem to be present when we operate the cameras at 5Hz as well as at 15Hz? Unfortunately, due to limitations of my test system, we cannot run a test at 30 fps without missing lots of frames.

You also suggested turning off "synchronized_images_only", i.e. setting it false. I did that and it appears that the (formerly 140 msec) latency changes to approximately 100 msec. I'd like to understand a bit more about the consequences of setting synchronized_images_only = false. Why did you expect that setting it false would improve the latency? And, with it set to false, what exactly are we allowing to happen? For example, are we saying that there can be cycles where either the depth or the color image in a given Kinect device capture could be missing?

[wes-b]

To deliver a synchornized color and depth image, the SDK must hold images in queus until both images are ready so they can be presented as a single capture. Setting synchronized_images_only to false means we will not hold any images, instead instead they will be delivered to the user as soon as they are received. This means you have to be careful to ensure that captures have a color or depth & IR image. Depth slows latency down because once it has been received by the SDK it needs to convert depth raw data to an image, which happens on the GPU. This amount of time veries based on GPU performance and load.

[StevenButner]

Very useful to know. It seems like a latency on the order of 100msec is about the best we can do on my system. We will make that work. I will close this ticket now. Thanks so much.

[Chris45215]

wes-b and StevenButner, I have encountered the same problem - the sensors have a very high latency no matter what settings are used, and regardless of how many are used. I posted instructions for a simple, repeatable test that anyone can perform in the post at #514 (comment). I measured the latency at roughly 0.2 seconds in that test (and 0.25 seconds when using body tracking), which includes additional process overhead, so that is a good match to your measurement of 140ms. I tried in several configurations and the results are consistent and point to the same conclusion.

No matter what configuration, sensor, and settings are used, the sensors have a very high latency compared to similar sensors and even the Kinect For XBox One. The leaked specs for the for the Kinect for XBox one show a latency of 0.06 seconds, which matches tests I have done.

StevenButner, I have been impressed by the Orbbec Astra sensors, and the Embedded S model has a much longer depth range than its specs claim. Intel's RealSense sensors also have better latency, though the output is messy compared to the Orbbec and Azure Kinect. The ROS wiki includes a list of popular compatible depth cameras and their latencies at https://rosindustrial.org/3d-camera-survey. If Microsoft cannot improve the latency of the Azure Kinect, I would suggest trying some of those, as that's what I've been doing.