-
Notifications
You must be signed in to change notification settings - Fork 43
Frequently Asked Questions
Most of the issues with DReyeVR installation and usage are listed here. If this document is not up to date, check out our GitHub issues page and create a new issue if you have a new issue.
Show answer
Like Carla, we use the notion of "ego-vehicle" to differentiate the AI-controlled vehicles from the player-driven vehicle. When you first start DReyeVR, you spawn in the world in a vehicle, this is the EgoVehicle.
In addition to the EgoVehicle, we use a "DReyeVR-sensor" which we call the EgoSensor that inherits from a Carla sensor but with features specific for DReyeVR. Our Ego-sensor tracks data such as user-inputs (keyboard/mouse/steering wheel), VR HMD orientation and location (6 dof), first-person camera frame captures, and of course the eye-tracker sensor. The underlying data structures used throughout the EgoSensor are defined in DReyeVRData.h, specifically in struct DReyeVR::AggregateData.
TL;DR: The EgoVehicle is the player-controlled vehicle, the EgoSensor is a sensor that contains all the core DReyeVR data we care about tracking in real time.
Show answer
Since the DReyeVR EgoVehicle inherits from an ACarlaWheeledVehicle, most of the standard CARLA vehicle features work out of the box. We needed to modify the recorder/replayer so all features of our EgoSensor are accurately recorder/replayed (such as HMD location/rotation, eye tracker data, user inputs, etc.).
However, main assumption used throughout DReyeVR is that that there is only ever one EgoVehicle in the world at a time. This is because each EgoVehicle will compete for the VR HMD system, eye tracker, and is a highly expensive Actor (eg. real-time mirrors). Therefore, we do not permit spawning new EgoVehicle actors from the PythonAPI, if you want to access the EgoVehicle, we built a mechanism for you to do so with DReyeVR_utils.py as follows:
from DReyeVR_utils import find_ego_vehicle
...
world = client.get_world()
...
DReyeVR_vehicle = find_ego_vehicle(world)Summary of supported CARLA features.
| Feature | Supported? | Notes |
|---|---|---|
| Keyboard manual control | ✔️ | |
| Wheel manual control | ✔️ | |
| World record/replay | ✔️ | |
| EgoSensor record/replay | ✔️ | Specific for DReyeVR |
| EgoSensor PythonAPI streaming | ✔️ | Specific for DReyeVR |
| Switching maps at runtime | ✔️ | |
| Changing weather | ✔️ | |
| Spawning EgoVehicle | ❌ | Supposed to be already-spawned in map. Persistence. |
| Destroying EgoVehicle | ❌ | Supposed to always be alive in map. Persistence. |
| EgoVehicle Carla AI Autopilot | ✔️ | |
ScenarioRunner --route
|
✔️ | The way we've been using ScenarioRunner so far |
ScenarioRunner --scenario
|
❌ | Not implemented yet |
ScenarioRunner --agent
|
❌ | Not implemented yet |
Show answer
The following PythonAPI scripts have been tested and should work with DReyeVR:
(:heavy_check_mark: = YES, :white_check_mark: = sort of, :grey_question: = Unsure, :x: = NO)
| Script | Supported? | Notes |
|---|---|---|
DReyeVR_AI.py |
✔️ | Example with the DReyeVR autopilot |
DReyeVR_logging.py |
✔️ | Log the DReyeVR EgoSensor data in real time |
schematic_mode.py |
✔️ | Like no_rendering_mode.py but with EyeTracker visualization |
schematic_mode.py |
✔️ | Like no_rendering_mode.py but with EyeTracker visualization |
scenario_runner.py --route |
✔️ | |
scenario_runner.py --scenario |
❌ | Not implemented |
scenario_runner.py --agent |
❌ | Not implemented |
show_recorder_collisions.py |
✔️ | |
show_recorder_file_info.py |
✔️ | |
start_recording.py |
✔️ | |
start_replaying.py |
✔️ | |
dynamic_weather.py |
✔️ | |
no_rendering_mode.py |
✅ | should instead use schematic_mode.py which is our DReyeVR wrapper around no_rendering_mode.py
|
show_recorder_actors_blocked.py |
✅ | |
automatic_control.py |
❌ | Use DReyeVR_AI.py
|
client_bounding_boxes.py |
❔ | |
draw_skeleton.py |
❔ | |
generate_traffic.py |
❔ | |
lidar_to_camera.py |
❔ | |
manual_control.py |
❌ | should never need this, manual control is done through the server |
manual_control_carsim.py |
❌ | |
manual_control_chrono.py |
❌ | |
manual_control_steeringwheel.py |
❌ | |
open3d_lidar.py |
❔ | |
sensor_synchronization.py |
❔ | |
synchronous_mode.py |
❔ | |
tutorial.py |
❌ | Spawns another EgoVehicle |
vehicle_gallery.py |
❌ | |
vehicle_physics.py |
❔ | |
visualize_multiple_sensors.py |
❔ |
Show answer
Currently we use the BP_TeslaM3 mesh provided by Carla for our EgoVehicle because the mesh has highly detailed internals and has good proportions for our use case.
However there is no reason for the EgoVehicle to have to be a Tesla, in fact it does not even need to be a Car, it could be a (motor)cyclist, firetruck, or other Actor inheriting from ACarlaWheeledVehicle.
We do not currently have updated documentation for changing the EgoVehicle mesh, but it should be fairly self-explanatory from some earlier guides where we first designed the EgoVehicle and decided on a mesh. Since the AEgoVehicle is a blueprint class in CPP, any Carla blueprint can be fitted to reparent this class and inherit all its functionality.
However, you will need to modify some soft-coded defaults that we tuned for our particular mesh. For instance, the locations of the driver's seat head position, steering wheel, dashboard, mirrors, etc. will all be relative to your mesh, so you will need to adjust these accordingly.
NOTE: Unlike those earlier tutorials, don't worry about adding the UCameraComponent to the Blueprint, since this is handled with the DReyeVR_Pawn.
Show answer
Once DReyeVR is installed (see make install), it lives in your CARLA directory as source files that work nicely with CARLA & UE4.
You can find where the files in this DReyeVR repo are placed in the CARLA repo by reading our correspondences files.
For example, if I wanted to edit the EgoVehicle.cpp file, I would do so in /PATH/TO/CARLA/Unreal/CarlaUE4/Source/CarlaUE4/DReyeVR/EgoVehicle.cpp. Ignore any intermediate or generated files as those are effectively cached and will be regenerated.
This is the same procedure for any files, including the DReyeVR params, PythonAPI files, ScenarioRunner, etc.
Another option is simply to modify all the DReyeVR files in the DReyeVR repo itself, then run make install again to overwrite the old files.
Show answer
First, you should verify that your logitech racing hardware is supported by the LogitechWheelPlugin we use (Supported devices denoted in the README).
Now that you've verified you have supposedly compatible hardware, we recommend using the Logitech G Hub and ensuring your hardware is detected by the OS.
If your hardware is doing something wrong when in DReyeVR (or seemingly nothing at all) then try enabling the Logitech-verbose-logging mode in the UE4 editor by modifying the param LogUpdates to True (in the [Hardware] section of the config file).
With this verbose-logging enabled, run make launch again to open Carla in the UE4-editor. Open Window -> Developer Tools -> Output Log to see all the logs from the level. Run the level and ensure one of these strings are printed:
# is detected
Found a Logitech device (XYZ) connected on input 0
# not detected
Could not find Logitech device connected on input 0
You should also read the logs which now will print on every tick whether or not it detects a state change such as a button press, axis rotation, or pedal depression. If those are still detected but the system is not responding, make an issue and let us know :)
Show answer
This happens when running DReyeVR-specific PythonAPI scripts and your Python install can't find DReyeVR_utils.py
We can fix this by adding PythonAPI/examples to the PYTHONPATH as follows:
# on Linux/MacOS
export CARLA_ROOT=/PATH/TO/carla/
export SCENARIO_RUNNER_ROOT=/PATH/TO/scenario_runner/
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla/dist/carla-0.9.13-py3.7-linux-x86_64.egg
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla/agents
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/examples # <-- make sure examples is present
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI
# on Windows x64 Visual C++ Toolset
set CARLA_ROOT=C:PATH\TO\carla\
set SCENARIO_RUNNER_ROOT=C:PATH\TO\scenario_runner\
set PYTHONPATH=%PYTHONPATH%;%CARLA_ROOT%\PythonAPI\carla\dist\carla-0.9.13-py3.7-win-amd64.egg
set PYTHONPATH=%PYTHONPATH%;%CARLA_ROOT%\PythonAPI\carla\agents
set PYTHONPATH=%PYTHONPATH%;%CARLA_ROOT%\PythonAPI\examples # <-- make sure examples is present
set PYTHONPATH=%PYTHONPATH%;%CARLA_ROOT%\PythonAPI\carla
set PYTHONPATH=%PYTHONPATH%;%CARLA_ROOT%\PythonAPINote that adding these paths to the PYTHONPATH might be different if you're using conda. If so, check out Install.md#using-conda-for-pythonapi.
Related issue(s): #16
Show answer
This is likely the case if you are running make install from inside the Carla directory.
The make install command is only defined when invoked from the DReyeVR directory.
Related issue(s): #7.
Show answer
This might show up when you are running a PythonAPI script after installing DReyeVR if you haven't re-run the make PythonAPI command.
After installing DReyeVR to Carla, you also need to rebuild the PythonAPI (which also rebuilds LibCarla) for all the DReyeVR-PythonAPI compatibility to work.
# after installing DReyeVR to carla
cd /PATH/TO/DReyeVR
make install CARLA=/PATH/TO/CARLA SR=/PATH/TO/SR
# now back in Carla, need to rebuild both PythonAPI & CarlaUE4
make PythonAPI && make launchRelated issues(s): #12
Show answer
This occurs when you are building PythonAPI and the output contains:
error: invalid command 'bdist_wheel'
-[BuildPythonAPI]: Carla lib for python has been successfully installed in "..."!
Although the log [BuildPythonAPI] says "successfuly installed", this is a (carla) bug and the installation actually failed as described with error above. You should make sure no error messages have occured in your build log.
(Thanks to @SteadyBits for the answer!) You should ensure wheel is installed via pip install wheel. You can check the PYTHONPATH to see if the wheel package is installed.
Then rebuilding the PythonAPI (make PythonAPI) should succeed without the error: invalid command message.
Related issue(s): #19