Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping

In CoRL 2020 [arXiv] [Robot Video] [project page] [pdf]

Adithya Murali, Weiyu Liu, Kenneth Marino, Sonia Chernova, Abhinav Gupta

Carnegie Mellon University Robotics Institute, Georgia Institute of Technology, Facebook AI Research

This is a pytorch-based implementation for our CoRL 2020 paper on task-oriented grasping. If you find this work useful in your research, please cite:

@inproceedings{murali2020taskgrasp,
  title={Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping},
  author={Murali, Adithyavairavan and Liu, Weiyu and Marino, Kenneth and Chernova, Sonia and Gupta, Abhinav},
  booktitle={Conference on Robot Learning},
  year={2020}
}

The code has been tested on Ubuntu 16.04 and with CUDA 10.0.

Installation

1. Create a virtual env or conda environment with python3

conda create --name taskgrasp python=3.6
conda activate taskgrasp

2. Make a workspace, clone the repo

mkdir ~/taskgrasp_ws && cd ~/taskgrasp_ws
git clone https://github.com/adithyamurali/TaskGrasp.git

3. Install dependencies

cd TaskGrasp
conda install pytorch=1.4.0 torchvision cudatoolkit=10.0 -c pytorch
pip install -r requirements.txt

4. Compile and install PointNet ops

cd ~/taskgrasp_ws
git clone https://github.com/erikwijmans/Pointnet2_PyTorch.git
cd Pointnet2_PyTorch
pip install -r requirements.txt
pip install -e .

5. Install Pytorch Geometric (only tested on v1.5.0)

pip install torch-scatter==latest+cu100 -f https://pytorch-geometric.com/whl/torch-1.4.0.html && pip install torch-sparse==latest+cu100 -f https://pytorch-geometric.com/whl/torch-1.4.0.html && pip install torch-cluster==latest+cu100 -f https://pytorch-geometric.com/whl/torch-1.4.0.html && pip install torch-spline-conv==latest+cu100 -f https://pytorch-geometric.com/whl/torch-1.4.0.html && pip install torch-geometric==1.5.0

Dataset

The dataset (5 GB) could be downloaded here and place it in the data folder as shown below:

cd ~/taskgrasp_ws/TaskGrasp
unzip ~/Downloads/data.zip -d ./
rm ~/Downloads/data.zip

To run any of the demo scripts explained below, do the same with the pretrained models (589 MB) and config files (10 KB) and put them in the checkpoints and cfg folder respectively.

Coming Soon: We are trying to release mesh models for the objects as well.

Usage

NOTE: The stable grasps were sampled from the point cloud using the agile_grasp repo from GT-RAIL.

Point Cloud	Stable grasps

To visualize the point cloud data and stable grasps:

python visualize.py --data_and_grasps --obj_name 124_paint_roller

Add the --obj_path argument with the absolute path to the dataset if you have placed it somewhere other than the default location (data/taskgrasp). The object can be specified with the --obj_name argument and the full list of objects can be found here.

To visualize the labelled task-oriented grasps in the TaskGrasp dataset:

python visualize.py --visualize_labels  --visualize_labels_blacklist_object 124_paint_roller

To visualize a specific grasp:

python visualize.py --obj_name 124_paint_roller --visualize_grasp --grasp_id 10

To run any of the training or evaluation scripts, download the config files from here and put them in the cfg folder. To train a model:

python gcngrasp/train.py --cfg_file cfg/train/gcngrasp/gcngrasp_split_mode_t_split_idx_3_.yml

To run pretrained models, download the models from here and unzip them into the checkpoints folder. To evaluate trained model on test set:

python gcngrasp/eval.py cfg/eval/gcngrasp/gcngrasp_split_mode_t_split_idx_3_.yml --save

Here's how to run evaluation on a single point cloud (e.g. sample data of an unknown object captured from a depth sensor) and assuming you already have sampled stable grasps. Make sure you have downloaded the pretrained models and data from the previous step, and run the following:

python gcngrasp/infer.py cfg/eval/gcngrasp/gcngrasp_split_mode_t_split_idx_3_.yml --obj_name pan --obj_class pan.n.01 --task pour