iTeach: Interactive Teaching for Robot Perception using Mixed Reality 🤖🌐

Jishnu Jaykumar P      Cole Salvato      Vinaya Bomnale      Jikai Wang      Yu Xiang

Project Webpage | 🤗 DH-YOLO Demo

We introduce iTeach, a Mixed Reality (MR) framework designed to enhance robot perception through real-time interactive teaching. By enabling human instructors to dynamically label robot RGB data, iTeach improves both the accuracy and adaptability of robot perception to new scenarios. The framework supports on-the-fly data collection and labeling, enhancing model performance and generalization. Applied to door and handle detection for household tasks, iTeach integrates a HoloLens app with an interactive YOLO model. Furthermore, we introduce the IRVLUTD DoorHandle dataset. DH-YOLO, our efficient detection model, significantly boosts the accuracy and efficiency of door and handle detection, showcasing the potential of MR to make robotic systems more capable and adaptive in real-world environments.

Getting Started in only 3 steps 🚀 !!!

The iTeach system can be started in just 3 simple steps:

Step-1: Build and install the iTeachLabeller app on the HoloLens 2.

• App Download Link 📱
• App Install Video 🛠️

Step-2: Navigate to the src directory and follow the setup instructions in its README file 📚.
Step-3: Start interacting with the app—navigate the robot, collect faulty samples, label them, and fine-tune the model. Real World Demo 🤖

✨ We show a demo of setting up the experiment hardware, network, and scripts to be run in this video 🎦. For detailed steps, refer to the video description 📋.

Directory Structure 📁

To begin working with the codebase, first navigate to the relevant directory and explore the files and subdirectories. Each directory includes its own README file with specific instructions on how to use the code.

• src: Contains the primary experiment files. 🧪
• toolkit: Source code for the iTeach toolkit. 🛠️
• hololens_app: Source code for the iTeachLabeller application. 📱
• dataloader: PyTorch dataloader for the IRVLUTD DoorHandle dataset. 🗃️
• hf_demo: Source code for the DHYOLO Hugging Face space. 🤗

Note: Click to show more 💡 (For PyPI)

For the toolkit and dataloader, execute the following commands with each new PyPI build:

rm -rf build/ dist/ # Also remove the corresponding .egg-info directory
python setup.py sdist bdist_wheel # Make sure to change the version in setup.py before running this
twine upload dist/* # Ensure you have the pypi-token

BibTex 📚

Please cite iTeach if it helps your research 🙌:

@misc{padalunkal2024iteach,
    title={iTeach: Interactive Teaching for Robot Perception using Mixed Reality},
    author={Jishnu Jaykumar P and Cole Salvato and Vinaya Bomnale and Jikai Wang and Yu Xiang},
    year={2024},
    eprint={2410.09072},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

Contact 📬

For any clarification, comments, or suggestions, you can choose from the following options:

• Join the discussion forum. 💬
• Report an issue. 🛠️
• Contact Jishnu. 📧

Acknowledgements 🙏

This work was supported by the DARPA Perceptually-enabled Task Guidance (PTG) Program under contract number HR00112220005, the Sony Research Award Program, and the National Science Foundation (NSF) under Grant No.2346528. We thank Sai Haneesh Allu for his assistance with the real-world experiments. 🙌