Performance impact after simGetImages() finished

[petergerten]

Hi,

I have a camera with 640x480 resolution configured and use simGetImages() in a loop getting ~40fps.
All good.
Now I added a second camera with all the same characteristics but a very high resolution: 5000x4000.
This is to mimic a still image function.

After I call simGetImages() on the high res camera once, the main camera loop framerate drops from ~40 fps to ~1fps.
While a drop during high-res image capture is expected - it never recovers from this and stays at ~1 fps.

Does anybody has an idea what is happening here?

Peter

[saihv]

Just to confirm, you've stopped the simGetImages loop and the game still runs at 1 FPS?

[petergerten]

@saihv basically I have one loop that captures 640x480 images. There is one condition if it is met it gets one high res image from the other camera. After that the condition is not triggered and the loop should run normal. But it does not.

[petergerten]

@saihv I have created a minimal example to reproduce the issue.
It runs a low resolution capture in a loop from a 640x480 camera and every 500 frames takes one frame from a high res camera (5000x4000).

import airsim
import sys
from datetime import datetime

client = airsim.MultirotorClient(ip=sys.argv[1])
framecounter = 1

prevtimestamp = datetime.now()

while(True):
    if framecounter%500 == 0:
        client.simGetImages([airsim.ImageRequest("highres", airsim.ImageType.Scene, False, False)])
        print("High resolution image captured.")
    if framecounter%30 == 0:
        now = datetime.now()
        print("Time spent for 30 frames: " + str(now-prevtimestamp))
        prevtimestamp = now
    client.simGetImages([airsim.ImageRequest("front_center", airsim.ImageType.Scene, False, False)])
    framecounter += 1

To be clear: the high res capture does work and produces a correct image - I just can't explain the performance impact afterwards:

bash-4.4# python3 demo.py 172.17.0.2
Time spent for 30 frames: 0:00:01.392797
Time spent for 30 frames: 0:00:01.276568
Time spent for 30 frames: 0:00:01.205635
Time spent for 30 frames: 0:00:01.127614
Time spent for 30 frames: 0:00:01.115183
Time spent for 30 frames: 0:00:01.011492
Time spent for 30 frames: 0:00:00.998835
Time spent for 30 frames: 0:00:01.000715
Time spent for 30 frames: 0:00:01.001668
Time spent for 30 frames: 0:00:01.004924
Time spent for 30 frames: 0:00:01.017626
Time spent for 30 frames: 0:00:01.032733
Time spent for 30 frames: 0:00:01.029245
Time spent for 30 frames: 0:00:01.022289
Time spent for 30 frames: 0:00:01.030059
Time spent for 30 frames: 0:00:01.032360
High resolution image captured.
Time spent for 30 frames: 0:00:07.714133
Time spent for 30 frames: 0:00:12.827132
Time spent for 30 frames: 0:00:12.839705
Time spent for 30 frames: 0:00:12.885909
Time spent for 30 frames: 0:00:12.815402
Time spent for 30 frames: 0:00:12.830131

[petergerten]

What I have noticed is this message on the airsim side:

[2020.01.21-04.50.07:427][695]LogRenderer: Reallocating scene render targets to support 5000x4000 Format 10 NumSamples 1 (Frame:2695).

I guess that results in some configuration that might cause the performance impact due to not being changed back?

[petergerten]

@saihv this seems to be a general issue that the performance will always be degraded in relation to the highest resolution used via the API.

[petergerten]

@saihv do you have any pointers on how to solve this?

[ironclownfish]

I have seen behavior like this due to the camera being activated upon capturing an image, but then never being deactivated.
You can work around this by toggling the PIP window for the camera on and then off to deactivate it. (i.e. press the keys 1 2 3, then 1 2 3 again. Frame rate should return to normal)

[petergerten]

@ironclownfish I actually only use the API and run AirSim headless. There is no window to toggle something. I never seen anything in the API referring to the notion of activating/deactivating a camera. Do you have any pointers to more information on that?

[petergerten]

@ironclownfish If not, could you please remove the 'fix proposed' tag? - that would increase the likelihood that someone takes a look at the issue :)

[petergerten]

@saihv do you have any insight on how to enable/disable a camera via API (see @ironclownfish response)?

[ironclownfish]

I've looked through the code and it seems there's no way to disable a camera via the API in headless mode.
If you want to do it yourself via C++, all you have to do is call PIPCamera::setCameraTypeEnabled(ImageType type, bool enabled)

[petergerten]

@ironclownfish thanks. That sounds promising! Currently I would not know how (where) to implement that though.

Currently it seems that the overall performance degrades to the highest resolution camera used in any client application. I am a bit baffled that nobody else seem to have a performance problem.

[rajat2004]

It runs a low resolution capture in a loop from a 640x480 camera and every 500 frames takes one frame from a high res camera (5000x4000).

import airsim
import sys
from datetime import datetime

client = airsim.MultirotorClient(ip=sys.argv[1])
framecounter = 1

prevtimestamp = datetime.now()

while(True):
    if framecounter%500 == 0:
        client.simGetImages([airsim.ImageRequest("highres", airsim.ImageType.Scene, False, False)])
        print("High resolution image captured.")
    if framecounter%30 == 0:
        now = datetime.now()
        print("Time spent for 30 frames: " + str(now-prevtimestamp))
        prevtimestamp = now
    client.simGetImages([airsim.ImageRequest("front_center", airsim.ImageType.Scene, False, False)])
    framecounter += 1

@petergerten Do you have a settings file to go alongwith this? Will try to reproduce and then maybe add API for disabling the camera

[rajat2004]

Opened #2465 which closes this