dtorre38 / mujoco_opencv

Integrating opencv with mujoco.

All of the following functions are in python, but are being ported to C. Also, check out model/box.xml to find out how to get a working ArUco marker in mujoco!

cart_pendulum.py is an implementation of setting a subcreen for the cart pole. The subscreen shows the top view of the pole looking down at the cart.

diff_drive_tracking.py is similar to opencv_tracking.py except there is a differential drive car that is being controlled to track a moving object.

mujoco_tracking.py is an implementation of the mujoco camera tracking feature on another body in the subscreen.

opencv_tracking.py tracks a moving object with opencv integration for applying a bounding box on the item being tracked.

utils/render_insetscreen.py is a function call for setting subscreens in the main mujoco simulation window. Add cameras as needed in xml files.

utils/get_frame.py is a function call for obtaining the pixels in a frame that will be passed to detect_and_draw_bound.py for further processing.

utils/detect_and_draw_bound.py is a function call that draws a bounding box on (yellow) colored objects. Requires the frame obtained by get_frame.py. The color of the object can be specified.

mujoco_opencv's Issues

Depth Rendering

Hi,

Thanks for integrating mujoco with opencv and sharing the same. I am facing one issue with respect to depth rendering, Earlier, I was using mujoco-viewer library to capture the depth frame. Now I am trying to extract the same with your code.

I am calling

depth, offscreen_viewport = get_frame(model, data, opt, scene, context, 'chaser_cam', loc_x=viewport_width - frame_width, loc_y=viewport_height - frame_height, width=frame_width, height=frame_height)

in my simulation loop. The get_frame is modified as follows.

def get_frame(model, data, opt, scene, context, camera_name, loc_x, loc_y, width=640, height=480):

    # 1. Create a rectangular viewport in the upper right corner for example.
   offscreen_viewport = mj.MjrRect(int(loc_x), int(loc_y), width, height)    
    
    # 2. Specify a different camera view by updating the scene with mjv_updateScene.
    # Set the camera to the specified view
    camera_id = mj.mj_name2id(model, mj.mjtObj.mjOBJ_CAMERA, camera_name)
    offscreen_cam = mj.MjvCamera()
    offscreen_cam.type = mj.mjtCamera.mjCAMERA_FIXED
    offscreen_cam.fixedcamid = camera_id

    mj.mjv_updateScene(model, data, opt, None, offscreen_cam, mj.mjtCatBit.mjCAT_ALL.value, scene)
    
    # 3.Render the scene in the offscreen buffer with mjr_render.
    depth = np.zeros((height*width*3, 1), dtype=np.uint8)  # Placeholder for depth data
    mj.mjr_render(offscreen_viewport, scene, context)

    # 4. Read the depth with mjr_readPixels.
    mj.mjr_readPixels(None, depth, offscreen_viewport, context)
        
    # 5. Depth processing 
    # Shift nearest values to the origin.
    depth-= depth.min()
    # Scale by 2 mean distances of near rays.
    depth= depth/ 2*depth[depth<= 1].mean()
    # Scale to [0, 255]
    depth= 255*np.clip(depth, 0, 1)
          
    # 6. Call mjr_drawPixels using the rectangular viewport you created in step 1.
    # mj.mjr_drawPixels(depth, None, offscreen_viewport, context)
    
    return frame, offscreen_viewport

But depth is not getting rendered. Can you please help with this ?

Thanks and Regards
Karthik R