Can speed up scene rendering?

[Nickvot]

Hi, I'm using the latest version of the AirSim and I tried to get stereo images.
Getting a stereo image pair usually takes 90-110ms (about 10 FPS) on nVidia GTX1070.
Adapting Unreal engine quality didn't help.
Is there any way to speed up scene rendering?

[saihv]

For me, the most effective trick was to reduce the resolution of the cameras.

[sytelus]

@saihv - thanks for your contribution to make things jitter free :). A quick question I'd was about RHICmdList.ReadSurfaceData. It seems that this call must be made from game thread. I'm wondering how you are making sure of that... Right now Unreal crashes randomly when making lots of get image API calls. I'm trying to investigate the reason for cause.

I'm working on specifying resolution in settings.json. Hopefully will get it done in 1-2 days.

[saihv]

@sytelus Do you see crashes when getImage is being called too frequently, but normal performance otherwise?

IIRC, RHICmdList.ReadSurfaceData should be called from the render thread. In my original implementation of the camera recording, you can notice that the ReadSurfaceData() call was inside an ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER specification: which has an underlying task graph and queues up the read command into the render thread, and might also possibly take care of 'too many' requests. Upon a quick look, it looks like ReadSurfaceData() is being called directly from RenderRequest.cpp without ENQUEUE: perhaps this is causing problems?

[sytelus]

Ah... looks like someone modified your original code and probably introduced bug. I'll investigate this and update this thread.

Meanwhile, I was wondering if you have experimented or know about another method to get screenshot. This is more "official way" of doing and it and one advantage I see is that it doesn't require another SceneCapture2D component which means it doesn't need to re-render scene again. In this case, you only get screenshot and assuming you have set FPV view, you can get fast mono images. According to docs, you first call FScreenshotRequest::RequestScreenshot(false) and then wait until image is recieved in ViewportClient->OnScreenshotCaptured event handler. I'm wondering if this might be the fastest way to get FPV mono images...

[saihv]

Interesting! I hadn't looked at FScreenshot earlier: when I was initially looking into options for screenshots, most of the native functionality seemed to involve blocking the game thread, thereby physics.

After a quick look, it looks like even FScreenshotRequest::RequestScreenshot does something similar. (reference). It seems to be calling GetViewportScreenshot(), which in turn calls ReadPixels() which leaves us with the same conundrum as AirSim's very first version of screenshots. But on the other hand, the guys at UnrealCV seem to have found an alternate method for this, by using RequestScreenshot() as an async operation. (reference). Perhaps it would be good to benchmark how fast UnrealCV logs screenshots.

[sytelus]

Some good news... I was able to track down crashes to few bugs that wasn't setting RenderTarget properly. Now I can run for fairly long time without encountering any crashes! I haven't done the change for ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER as I found there is also a check for not to run on render thread. I still don't completely understand that part so until I see problem, I would hold off on this change. I'm planning to stress test everything leaving to run for whole night.

I've also added ability to set resolution, FOV and few other params using settings.json. Here's the doc that describes it.

[saihv]

@sytelus

Just wanted to add a little bit to our past discussion here.

With the new recording method (and possibly the UE version upgrade?), I am seeing a bit of an issue. Essentially, there's an expensive flush operation being performed along with the recording, which is coming from the RHICmdList.ReadSurfaceData() function. This is weird, because it did not happen in UE 4.15 and the old recording code: but based on my quick look at the UE repository, this function hasn't really changed from 4.15 onwards. Also, the new method is theoretically similar to the old one, just with some extra abstraction etc. (correct me if I am wrong).

In 4.15 and the old method of recording, once the sub-views were turned on, and the capture components started running, there was no additional overhead involved with ReadSurfaceData() or the data saving. But in the new versions, there is an additional slowdown which seems to be coming from this strange flush operation. Because of this flush, even the game thread hangs for a little bit in order to sync the frame. Attaching a screenshot from UE profiler here. (notice the continuous expensive spikes coming from ReadSurfaceData_Flush..) Unfortunately, this 'on-off' kind of jitter doesn't go away even if I limit the max FPS of the game. In this picture, the frame rate is still decent because there was only one vehicle, but because I regularly work with 3-5 drones in game, this problem becomes more apparent.

I was just wondering if you have any thoughts about this. Also, is it possible at all to run the new AirSim code in UE 4.15? Or is 4.16/17 deeply integrated with the new features? Thanks for your time!

[saihv]

@sytelus

I found a fix for the jitters. HDR is not being disabled in the render target in the new code, which is why the flush is taking a long time (if you remember, in the old Render Target uasset files, HDR box was checked off) (please see below). If smoother recording is desired, this line will have to be added to APIPCamera::updateCaptureComponentSettings

render_target->bHDR = false;

With this addition, the flush command takes less than a millisecond for the same scene, compared to >10 ms with HDR on. Perhaps it would be good to add this to the settings file as well so users can choose between quality and performance!

EDIT: Additional observation - Actually, I don't think the flush was taking a long time just because HDR was on: it's because Unreal was allocating way too much memory. A HDR image is twice the size of a non-HDR image: so for the same resolution, a HDR image should not take 14 ms to flush compared to 1 ms for a non HDR image. It looks like if we don't specify HDR or no HDR, Unreal is allocating too much memory to the render target in general. If the HDR true/false specification is done AFTER the resolution is set at the line

render_target->InitAutoFormat(settings.width, settings.height);

Looks like then, Unreal allocates exactly how much memory is needed for that image size, and the jitter goes away even with HDR on. I am not 100% sure how Unreal is managing these things underneath, but just based on my experiments. For now, I am disabling HDR entirely, and I have jitter-free recording for around 5 vehicles.

[nickwalton]

Hello. I'm currently facing the same issue on an Unreal project. Were you able to solve it by changing bHDR to false? I'm working on unreal version 4.17 and even after doing these changes I'm still facing the same low fps issue. Do you have any recommendations?