alters-mit / magnebot

High-level API for the TDW Magnebot

Magnebot

Magnebot is a high-level robotics-like API for TDW. The Magnebot can move around the scene and manipulate objects by picking them up with "magnets". The simulation is entirely driven by physics.

The Magnebot can be loaded into a wide variety of scenes populated by interactable objects.

At a low level, the Magnebot is driven by robotics commands such as set_revolute_target, which will turn a revolute drive. The high-level API combines the low-level commands into "actions", such as grasp(target_object) or move_by(distance).

The Magnebot API supports both single-agent and multi-agent simulations.

Requirements

See TDW requirements.

Installation

1. pip3 install magnebot
2. (Linux servers only): Download the latest TDW build and unzip it. For more information on setting up a TDW build on a server, read this. On a personal computer, the build will be downloaded and launched automatically.

Test if your installation was successful

1. Run this controller:

from magnebot import MagnebotController

c = MagnebotController() # On a server, change this to MagnebotController(launch_build=False)

c.init_scene()
c.move_by(2)
print(c.magnebot.dynamic.transform.position)
c.end()

2. (Linux servers only): Launch the TDW build. On a personal computer, the build will launch automatically.

Update an existing installation

1. pip3 install tdw -U
2. pip3 install magnebot -U
3. (Linux servers only): Download the latest TDW build and unzip it. On a personal computer, the build will automatically be upgraded the next time you create a TDW controller.

If you are upgrading from Magnebot 1.3.2 or earlier, be aware that there are many changes to the API in Magnebot 2.0.0 and newer. Read the changelog for more information.

Manual

General

• Changelog
• Troubleshooting and debugging
• Performance benchmark

TDW Documentation

Before using Magnebot, we recommend you read TDW's documentation to familiarize yourself with some of the underlying concepts in this API:

• Setup
• Core Concepts
• Objects and scenes
• Robots
• Multi-agent simulations

MagnebotController (single-agent, high-level API)

The MagnebotController offers a simplified API for single-agent simulations. Actions are non-interruptible; self.move_by(2) will simulate motion until the action ends (i.e. when the Magnebot has moved forward by 2 meters). This API mode has been optimized for ease of use and simulation speed.

• Overview
• Scene setup
• Output data
• Actions
• Moving, turning, and collision detection
• Arm articulation
• Grasp action
• Camera rotation
• Third-person cameras
• Occupancy maps

Magnebot (n-agent, lower-level API)

Magnebot is a TDW add-on that must be added to a TDW controller to be usable. Magnebot can be used in multi-agent simulations, but it requires a more extensive understanding of TDW than MagnebotController.

• Overview
• Scene setup
• Output data
• Actions
• Moving, turning, and collision detection
• Arm articulation
• Grasp action
• Camera
• Third-person cameras
• Occupancy maps
• Multi-agent simulations

Actions

It is possible to define custom Magnebot actions by extending the Action class.

• Overview
• Move and turn actions
• Arm articulation actions
• Inverse kinematics (IK) actions
• Camera actions

---

API

• MagnebotController
• Magnebot
• Other API documentation

---

Examples

Example controllers show actual examples for an actual use-case.

Other controllers in this repo:

• Promo controllers are meant to be use to generate promo videos or images; they include low-level TDW commands that you won't need to ordinarily use.
• Test controllers load the Magnebot into an empty room and test basic functionality.

---

Higher-level APIs

The Magnebot API relies on the tdw Python module. Every action in this API uses combinations of low-level TDW commands and output data, typically across multiple simulation steps.

This API is designed to be used as-is or as the base for an API with higher-level actions, such as the Transport Challenge.

API	Description
Transport Challenge	Transport objects from room to room using containers as tools.
Multimodal Challenge	Perceive objects in the scene using visual and audio input.

magnebot's Issues

Cannot continue with the example

I have been following the example here

from magnebot import Magnebot, Arm
m = Magnebot()
status = m.init_floorplan_scene(scene="2a", layout=1)

Now I got an error message below.

>>> status = m.init_floorplan_scene(scene="2a", layout=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mai/jroh/miniconda3/envs/tdw/lib/python3.8/site-packages/magnebot/magnebot_controller.py", line 480, in init_floorplan_scene
    return self._init_scene(scene=scene_commands,
  File "/home/mai/jroh/miniconda3/envs/tdw/lib/python3.8/site-packages/magnebot/magnebot_controller.py", line 2364, in _init_scene
    status = self._do_arm_motion()
  File "/home/mai/jroh/miniconda3/envs/tdw/lib/python3.8/site-packages/magnebot/magnebot_controller.py", line 1853, in _do_arm_motion
    for i in range(len(state_0.joint_angles[a_id])):
KeyError: -1262053853

I have installed tdw==1.8.21 and magnebot==1.2.2.

self.sock.recv() causes hang

I am trying to install using pip on my Mac. After running the controller I always get a hang after m=Controller(). The imports seems to work fine however. I hit control+C to quit and always get the following output which seems like the self.socket.recv() line is making the program hang for me:

line 50, in __init__
    self.socket.recv()
  File "zmq/backend/cython/socket.pyx", line 781, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 817, in zmq.backend.cython.socket.Socket.recv
  File "zmq/backend/cython/socket.pyx", line 186, in zmq.backend.cython.socket._recv_copy
  File "zmq/backend/cython/checkrc.pxd", line 13, in zmq.backend.cython.checkrc._check_rc

Any help would be appreciated. Thank you.

`m.init_scene(scene="1a", layout=0, room=1)` takes forever

My Latitude-5580 with 4-cored i7-7600U CPU @ 2.80GHz repeatedly blocks on m.init_scene(scene="1a", layout=0, room=1). What can I do to speed this process up?

my gpu info:

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce 930MX       On   | 00000000:03:00.0 Off |                  N/A |
| N/A   51C    P5    N/A /  N/A |    329MiB /  2004MiB |     11%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1012      G   /usr/lib/xorg/Xorg                 38MiB |
|    0   N/A  N/A      1718      G   /usr/lib/xorg/Xorg                162MiB |
|    0   N/A  N/A      1857      G   /usr/bin/gnome-shell               32MiB |
|    0   N/A  N/A      2257      G   ...gAAAAAAAAA --shared-files       86MiB |
+-----------------------------------------------------------------------------+

'NoneType' object has no attribute 'transform'

After running the provided code example, I get this error : 'NoneType' object has no attribute 'transform'

from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from magnebot import Magnebot, ActionStatus

c = Controller()  # On a server, change this to Controller(launch_build=False)
magnebot = Magnebot()
c.communicate(TDWUtils.create_empty_room(12, 12))
magnebot.move_by(2)
while magnebot.action.status == ActionStatus.ongoing:
    c.communicate([])
c.communicate([])
print(magnebot.dynamic.transform.position)
c.communicate({"$type": "terminate"})

custom_api.py doesn't work

Disable Physics for Magnebot?

Hey @alters-mit, I would like to disable the physics for magnebot and I'm wondering if there are commands in the tdw API that would help me achieve that. Firstly, if I would like to move/rotate the robot I would want to teleport it to the destination without physics. This should also teleport the objects held by the robot. Similarly, in order to grasp some objects I was wondering if I could disable the physics and attach objects to one of the magnebot's arms based on object id. Thanks!

Destroying magnebot

Hello, is there a way of destroying a magnebot at runtime?

I have tried with the next code, but with no success:

import numpy as np
from tdw.controller import Controller
from tdw.tdw_utils import TDWUtils
from magnebot import Magnebot, ImageFrequency
from magnebot.util import get_default_post_processing_commands
from tdw.add_ons.first_person_avatar import FirstPersonAvatar
import pdb

width = 256
height = 256


class ChaseBall(Controller):


    def __init__(self, port: int = 1071, check_version: bool = True, launch_build: bool = True):
        super().__init__(port=port, check_version=check_version, launch_build=launch_build)



        # Add the avatar and the Magnebot.

        
        self.av = FirstPersonAvatar()

        self.magnebot: Magnebot = Magnebot(robot_id=self.get_unique_id(),   position={"x": -1.4, "y": 0, "z": -1.1},
                                           image_frequency=ImageFrequency.never)

      
        self.add_ons.extend([self.av, self.magnebot])


        # Create the scene.

        commands = [ 
                    TDWUtils.create_empty_room(20, 20),
                    
                    {"$type": "set_screen_size",
                     "width": width, #640,
                     "height": height}, #480},
                    {"$type": "create_interior_walls", "walls": [{"x": 6, "y": 1}, {"x": 6, "y": 2},{"x": 6, "y": 3},{"x": 6, "y": 4},{"x": 6, "y": 5},{"x": 1, "y": 6},{"x": 2, "y": 6},{"x": 3, "y": 6},{"x": 4, "y": 6},{"x": 5, "y": 6}]},
                    {"$type": "create_interior_walls", "walls": [{"x": 14, "y": 1}, {"x": 14, "y": 2},{"x": 14, "y": 3},{"x": 14, "y": 4},{"x": 14, "y": 5},{"x": 19, "y": 6},{"x": 18, "y": 6},{"x": 17, "y": 6},{"x": 16, "y": 6},{"x": 15, "y": 6}]},   
                    {"$type": "create_interior_walls", "walls": [{"x": 6, "y": 19}, {"x": 6, "y": 18},{"x": 6, "y": 17},{"x": 6, "y": 16},{"x": 6, "y": 15},{"x": 1, "y": 14},{"x": 2, "y": 14},{"x": 3, "y": 14},{"x": 4, "y": 14},{"x": 5, "y": 14}]},
                    {"$type": "create_interior_walls", "walls": [{"x": 14, "y": 19}, {"x": 14, "y": 18},{"x": 14, "y": 17},{"x": 14, "y": 16},{"x": 14, "y": 15},{"x": 19, "y": 14},{"x": 18, "y": 14},{"x": 17, "y": 14},{"x": 16, "y": 14},{"x": 15, "y": 14}]}
                    ]


        


        # Add post-processing.
        commands.extend(get_default_post_processing_commands())
        

        self.communicate(commands)


    


    
    def run(self):

        commands = []
      
        timer = 100
        print(self.magnebot.robot_id)
        
        while True:

           
            resp = self.communicate(commands)
            commands.clear()
            #pdb.set_trace()

            all_images = []
            

            if not timer:
                print("Destroying robot")
                commands.append({"$type": "destroy_robot", "id": self.magnebot.robot_id}) #Destroy magnebot after certain time
                timer -= 1
            elif timer > 0:
                timer -= 1




        self.communicate({"$type": "terminate"})


    


if __name__ == "__main__":
    c = ChaseBall()
    print('hello')
    c.run()

After trying to destroy the robot using the destroy_robot command, the simulation crashes with the next output:

IndexOutOfRangeException: Index was outside the bounds of the array.
  at UnityEngine.ArticulationReducedSpace.get_Item (System.Int32 i) [0x00018] in <654f6f6960084f7f9e01df4721ae83bf>:0 
  at TDWInput.SendDynamicRobots.GetData (FlatBuffers.FlatBufferBuilder fbb) [0x00278] in <99d63b1e506f4bb8be3f49a715cbd4fa>:0 
  at TDWInput.SendSingleDataCommand`1[T].GetOutputData (NetMQ.NetMQMessage& data) [0x00007] in <99d63b1e506f4bb8be3f49a715cbd4fa>:0 
  at TDWInput.SendDataCommand`1[T].Send (NetMQ.NetMQMessage& data) [0x00014] in <99d63b1e506f4bb8be3f49a715cbd4fa>:0 
  at (wrapper delegate-invoke) <Module>.invoke_void_NetMQMessage&(NetMQ.NetMQMessage&)
  at NetworkManager.GetOutput (NetMQ.NetMQMessage& output) [0x0005c] in <99d63b1e506f4bb8be3f49a715cbd4fa>:0 
  at Req.Update () [0x00038] in <99d63b1e506f4bb8be3f49a715cbd4fa>:0 
  at NetworkManager.Update () [0x0000e] in <99d63b1e506f4bb8be3f49a715cbd4fa>:0 

I also have tried with the destroy_object and destroy_avatar commands with no success.

push.py doesn't work

Magnebot position when turning and moving varies between machines