Skip to content

Latest commit

 

History

History
167 lines (133 loc) · 19.1 KB

README.md

File metadata and controls

167 lines (133 loc) · 19.1 KB
Raw

BiGym

PRs

BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmark
Nikita Cherniadev*, Nicholas Backshall*, Xiao Ma*, Yunfan Lu, Younggyo Seo, Stephen James

BiGym is a new benchmark and learning environment for mobile bi-manual demo-driven robotic manipulation. BiGym features 40 diverse tasks set in home environments, ranging from simple target reaching to complex kitchen cleaning. To capture the real-world performance accurately, we provide human-collected demonstrations for each task, reflecting the diverse modalities found in real-world robot trajectories. BiGym supports a variety of observations, including proprioceptive data and visual inputs such as RGB, and depth from 3 camera views.

For latest updates, check our project page: https://chernyadev.github.io/bigym/

Table of Contents

  1. Install
  2. Tasks
  3. Usage
  4. Contributing

Install

pip install .

Tasks

Task Description Preview
ReachTarget Reach the target with either left or right wrist.
ReachTargetSingle Reach the target with specific wrist.
ReachTargetDual Reach 2 targets, one with each arm.
StackBlocks Move blocks across the table, and stack them in the target area.
MovePlate Move the plate between two draining racks.
MoveTwoPlates Move two plates simultaneously from one draining rack to the other.
FlipCup Flip the cup, initially positioned upside down on the table, to an upright position.
FlipCutlery Take the cutlery from the static holder, flip it, and place it back into the holder.
DishwasherOpen Open the dishwasher door and pull out all trays.
DishwasherClose Push back all trays and close the door of the dishwasher.
DishwasherOpenTrays Pull out the dishwasher’s trays with the door initially open.
DishwasherCloseTrays Push the dishwasher’s trays back with the door initially open.
DishwasherLoadPlates Move plates from the rack to the lower tray of the dishwasher.
DishwasherLoadCups Move cups from the table to the upper tray of the dishwasher.
DishwasherLoadCutlery Move cutlery from the table holder to the dishwasher’s cutlery basket.
DishwasherUnloadPlates Move plates from the tray of the dishwasher to a table rack.
DishwasherUnloadCups Move cups from the upper tray of the dishwasher to the table.
DishwasherUnloadCutlery Move cutlery from the cutlery basket to a tray on the table.
DishwasherUnloadPlatesLong A full task of unloading a plate: picking up the plate from dishwasher, placing this plate into the rack located in the closed wall cabinet, and closing the dishwasher and cupboard.
DishwasherUnloadCupsLong A full task of unloading a cup: picking up the cup, placing it inside the closed wall cabinet, and closing the dishwasher and cupboard.
DishwasherUnloadCutleryLong A full task of unloading a cutlery: picking up a cutlery, placing it into the cutlery tray inside the closed drawer, and closing the dishwasher and drawer.
DrawerTopOpen Open the top drawer of the kitchen cabinet.
DrawerTopClose Close the top drawer of the kitchen cabinet.
DrawersAllOpen Open all sliding drawers of the kitchen cabinet.
DrawersAllClose Close all sliding drawers of the kitchen cabinet.
WallCupboardOpen Open doors of the wall cabinet.
WallCupboardClose Close doors of the wall cabinet.
CupboardsOpenAll Open all drawers and doors of the kitchen set.
CupboardsCloseAll Close all drawers and doors of the kitchen set.
PutCups Pick up cups from the table and put them into the closed wall cabinet.
TakeCups Take two cups out from the closed wall cabinet and put them on the table.
PickBox Pick up a large box from the floor and place it on the counter.
StoreBox Move a large box from the counter to the shelf in the cabinet below.
SaucepanToHob Take the saucepan from the closed cabinet and place it on the hob.
StoreKitchenware Take all items from the hob and place them in the cabinet below.
ToastSandwich Use the spatula to put the sandwich on the frying pan.
FlipSandwich Flip the sandwich in the frying pan using the spatula.
RemoveSandwich Take the sandwich out of the frying pan.
GroceriesStoreLower Place a random set of groceries in the cabinets below the counter.
GroceriesStoreUpper Place a random set of groceries in cabinets and shelves on the wall.

Usage

Directly instantiate the task of interest. Tasks are located in bigym/envs/.

from bigym.action_modes import TorqueActionMode
from bigym.envs.reach_target import ReachTarget
from bigym.utils.observation_config import ObservationConfig, CameraConfig

env = ReachTarget(
    action_mode=TorqueActionMode(floating_base=True),
    observation_config=ObservationConfig(
        cameras=[
            CameraConfig(
                name="head",
                rgb=True,
                depth=False,
                resolution=(128, 128),
            )
        ],
    ),
    render_mode=None,
)

Use ActionModes to parameterise how you want to control your robot.

Working with demonstrations

Demo Store

Please see simple example here: examples/replay_demo.py.

Demonstrations are automatically downloaded from GitHub releases. When demos are requested by calling DemoStore.get_demos(). The current dataset will be cached locally at $HOME/.bigym/v0.0.0. Demonstrations with custom observations and frequency are also cached to $HOME/.bigym/v0.0.0.

⚠️ Warning: We are working on the dataset. Not all demonstrations will result in successful completion of the task. Please validate before use.

⚠️ Warning: The current dataset includes demonstrations for the following action modes only:

  • JointPositionActionMode(floating_base=True, absolute=True)
  • JointPositionActionMode(floating_base=True, absolute=False)

Demo Player

Replay existing demos using GUI player.

python tools/demo_player/main.py

VR Demo Recorder

Record new demos in VR. Follow VR README to configure docker container to run this tool.

python tools/demo_recorder/main.py

Contributing

On each PR, please ensure to bump the:

  • Major version if you alter the existing interface in any way.
  • Minor version if you have added new features (which didn't break the existing interface)
  • Patch version for bug fixes.

Please ensure that you pass pre-commits before opening a PR: pre-commit run --all-files and that you pass all tests: pytest tests/ --run-slow.

Licenses

Citation

If you find our work helpful, please kindly cite us

@article{chernyadev2024bigym,
  title={BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmark},
  author={Chernyadev, Nikita and Backshall, Nicholas and Ma, Xiao and Lu, Yunfan and Seo, Younggyo and James, Stephen},
  journal={arXiv preprint arXiv:2407.07788},
  year={2024}
}