xfgao / vrkitchen

VRKitchen: an Interactive 3D Environment for Learning Real Life Cooking Tasks. Visit the project site for more information: https://sites.google.com/view/vr-kitchen/.

VRKitchen

VRKitchen is an interactive 3D virtual kitchen environment which provides a platform for training and evaluating various learning and planning algorithms in a variety of cooking tasks. Two kinds of tasks are available:

1. Tool Use: requires an agent to continuously control its hands to make use of a tool.

   E.g. cutting a carrot into pieces, peeling a kiwi, pouring water from one cup to another.

2. Preparing dishes: agents must perform a series of atomic actions in the right order to achieve a compositional goal.

   E.g. making fruit juice, beef stew and sandwiches.

Requirements

• OS: Ubuntu 16.04
• Python 2.7.12

Getting Started

1. Clone the repo and install the requirements.

git clone https://github.com/xfgao/VRKitchen.git
pip install -r requirements.txt

2. Download the zip file and unzip it into the /Binaries folder.

• [Download] https://github.com/xfgao/VRKitchen/releases/download/v0.1.0/VRKitchen-v0.1.0.zip

3. To reproduce the experiment results, run the Script/example_dish.py (for preparing dishes) and Script/example_tool.py (for tool use).

python Script/example_dish.py
python Script/example_tool.py

Have fun!

Dataset

Dataset for the dish preparation task: https://drive.google.com/file/d/1QTAvO6Uwm_paGAfGMveKHCMxwAxGAyi_/view?usp=sharing

Citation

@article{VRKitchen,
  author    = {Xiaofeng Gao and
               Ran Gong and
               Tianmin Shu and
               Xu Xie and
               Shu Wang and
               Song{-}Chun Zhu},
  title     = {VRKitchen: an Interactive 3D Virtual Environment for Task-oriented
               Learning},
  journal   = {arXiv},
  volume    = {abs/1903.05757},
  year      = {2019},
}

vrkitchen's Issues

Controlling the agent and recording data

Dear VRKitchen development group,

greetings from the Vision for Robotics group from the University of Technology Vienna, Austria (https://www.acin.tuwien.ac.at/en/vision-for-robotics/)!

We came across your VRKitchen project and it looks very interesting to us. More specifically, We would use the "prepare dish" scenario with atomic actions to create traces of object positions / object-to-object contacts that I use as input to our own plan recognition algorithm. Your environment seems very suited to this task and would save me a lot of time.

Therefore we would kindly ask you to clarify the usage of your system to us. We tried to run it, with some success:

• The simulation starts and a window on the screen appears, but there was no support for HTC Vive.
• If we don't comment out the call to simulate the steps of a plan, there is random movement and an error message that data is missing.
• We didn't find the relevant piece of code that would allow us to act in the simulation through the VR controller input ourselves.

We would be very thankful for some clarification and further insights into your project so we can use it to create test data for our plan recognition project.

Best regards,

Michael Koller