Author's implementation of Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments (ICCV 2023)
We checked code works in Ubuntu 20.04.
All dependencies can be installed at once by the command below.
install_total.shNote: The script includes sudo commands.
This script download sources on c_env and py_env and installs on the same directory. It will take some time to install libraries on the c_env folder.
You can build via:
./run_cmake.sh
cd build
make -j8- Download bvh files for the motion synthesizer from the link, and place into
data/motionfolder. - From the link above, download
autoencoder.pklfile and place intoautoencoder/output/cnn_pretrained/model/folder. - For the scenes, download 3D scene scans (*.ply files) from the PROX dataset, and place into
data/scene/proxfolder. As the 3D scene meshes have a lot of vertices/faces, rendering the meshes could be a bit slow. You can download downsampled meshes from the PROX dataset or decimate through MeshLab.
The inputs (action cue, 3D scene, initial starting point) are defined in env/env_config/prox_example.xml file.
/env
├── env_config // includes 3D scene, action cue, initial starting point
/data
├── character
│ └── object // includes xml files for scene meshes (Dummy xml files to format 3D scenes as 6Dof node in DART library)
│ skel_mxm.xml // skeleton
├── scene
│ └── prox // includes .ply files of prox 3D scene scans
└── motion // includes motion database
/result
You can adjust hyperparameters/configurations defined in env/EnvConfigurations.h file.
You can render the results by executing render file in build/render.
Sample motions and pretrained policies are included in this repository.
cd build
./render/render --type=action_control --env=examples/SCENE_NAME/input${i}.xml --ppo=examples/SCENE_NAME_input${i}/network-0 --dir=SAVE_DIR_NAME-
Optimizing to fit into chairs (Sec 3.6, Fig. 10): add
--optimizeflag to run optimization, and the results would be saved in theresults/SAVE_DIR_NAMEfolder. -
Or, you can run optimization and save the results via the UI. Results would be saved in
/results/SAVE_DIR_NAME/.
cd build
# for action
./render/render --type=pp_record --dir=SAVE_DIR_NAME
# for action + manip
./render/render --type=manip_record --dir=MANIP_EXAMPLE Foot contact and penetration evaluation (Sec. 4.1) are included in type=pp_record renderer.
Note that we cannot provide object meshes used for manipulation, so to visualize human motion only, use type=pp_record instead of type=manip_record.
cd build
./render/render --type=bvh --bvh=walk/BVH_FILENAME.bvh cd build
./render/render --type=int_gen --env=examples/SCENE_NAME/input1.xml # only the 3D scene information are read cd network
python3 ppo.py --test_name="POLICY_NAME" --env="examples/SCENE_NAME/input${i}" --ntimesteps=350 --nslave=8 # adjust the parameters freelyIf you find the repo useful, please cite:
@inproceedings{lee2023lama,
title = {Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments},
author = {Lee, Jiye and Joo, Hanbyul},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
year = {2023}
}