j96w / mimicplay Goto Github PK

windows system
error：
ERROR: Failed building wheel for egl_probe
Running setup.py clean for egl_probe
Failed to build egl_probe
ERROR: Could not build wheels for egl_probe, which is required to install pyproject.toml-based projects

Hello, this is really impressive work.
I have been able to replicate the simulation part of training the high and low level planner using play data. I am currently focusing on training a high-level latent plan using human play data. To this end, I have successfully generated an HDF5 file containing all the relevant information from the human play data extracted from the provided videos. However, I have encountered a hurdle in adapting the existing code for training the high level planner to utilize my generated HDF5 file. The current training code appears to be tailored for simulation data, leading to errors when I attempt to integrate my dataset. Any guidance or direction you could offer would be immensely helpful.
Thank you!

Hi, I am trying to obtain a dataset with images from another camera pov by replaying the provided playdata/demo.hdf5 dataset through dataset_states_to_obs.py. According to information from the robosuite documentation, I should be able to do this by modifying the agentview camera params in _setup_camera() in LIBERO/libero/libero/envs/problems/libero_kitchen_tabletop_manipulation.py. It doesn't seem to work as the camera view stays the same when I visualize the converted dataset with playback_robomimic_dataset.py. However, after I modified the said function, the rollout videos saved while training the low-level policy do follow the new camera configurations.

Does anyone have more insight into this or tried to do so themselves? Thanks a lot!

Hi,
Here https://mimic-play.github.io/ it seems you are using an iPhone to control the robotic arm. Could you please let me know how you did so? Is there any APP or open-source code to do so? Thanks.

Hi,what a brilliant work of mimicplay!
In fact,when implementing training lowlevel controller with my real-world playdata,aftering training my highlevel human playdata, error occured like this:

Traceback (most recent call last):
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/scripts/train.py", line 379, in main
train(config, device=device)
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/scripts/train.py", line 197, in train
step_log = TrainUtils.run_epoch(model=model, data_loader=train_loader, epoch=epoch, num_steps=train_num_steps)
File "/home/kiriyamagk/桌面/robomimic/robomimic/utils/train_utils.py", line 559, in run_epoch
info = model.train_on_batch(input_batch, epoch, validate=validate)
File "/home/kiriyamagk/桌面/robomimic/robomimic/algo/bc.py", line 137, in train_on_batch
predictions = self._forward_training(batch)
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/algo/mimicplay.py", line 328, in _forward_training
_, mlp_feature = self.human_nets.policy._get_latent_plan(batch['obs'], batch["goal_obs"])
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/algo/mimicplay.py", line 137, in _get_latent_plan
dists, enc_out, mlp_out = self.nets["policy"].forward_train( #调用策略网络
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/models/policy_nets.py", line 226, in forward_train
out, enc_out, mlp_out = MIMO_MLP.forward(self, return_latent=return_latent, obs=obs_dict, goal=goal_dict)
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/models/obs_nets.py", line 594, in forward
mlp_out = self.nets"mlp"
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/kiriyamagk/桌面/robomimic/robomimic/models/base_nets.py", line 286, in forward
return self._model(inputs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 118, in forward
return F.linear(input, self.weight, self.bias)

However,error disappeared after I converted the script in demo_mp4.py like this:
all_hand_loc = np.concatenate((hand_loc_1[:, :, -2:], hand_loc[:, :, -2:]), axis=2)
,setting ac_dim=30 in the highlevel_human.json and finally retraining highlevel planner and lowlevel controller.Also,it seems no use setting ac_dim=40 in the lowlevel.json to train lowlevel controller successfully without changing all_hand_loc in demo_mp4.py.

Hello! @j96w It's an excellent work of robot learning.
When I tried to process the data downloaded from the google drive using
python scripts/dataset_states_to_obs.py --dataset 'datasets/playdata/demo_modified.hdf5' --done_mode 0 --camera_names agentview robot0_eye_in_hand --camera_height 84 --camera_width 84 --output_name image_demo_local.hdf5 --exclude-next-obs,
I got the render error about OpenGL
OpenGL.raw.EGL._errors.EGLError: EGLError( err = EGL_NOT_INITIALIZED, baseOperation = eglMakeCurrent, cArguments = ( <OpenGL._opaque.EGLDisplay_pointer object at 0x7fe287ca7dc0>, <OpenGL._opaque.EGLSurface_pointer object at 0x7fe2921ab840>, <OpenGL._opaque.EGLSurface_pointer object at 0x7fe2921ab840>, <OpenGL._opaque.EGLContext_pointer object at 0x7fe2921ab7c0>, ), result = 0 ).
Have you ever met this problem? Or I would like to know if the version of the mujoco causes this matter. My mujoco version is 3.1.1.

Environment Libero_Kitchen_Tabletop_Manipulation not found. Make sure it is a registered environment among: Lift, Stack, NutAssembly, NutAssemblySingle, NutAssemblySquare, NutAssemblyRound, PickPlace, PickPlaceSingle, PickPlaceMilk, PickPlaceBread, PickPlaceCereal, PickPlaceCan, Door, Wipe, ToolHang, TwoArmLift, TwoArmPegInHole, TwoArmHandover, TwoArmTransport

I am very interested in your work. I encountered an error while running run_trained_agent.py. What could be the possible reasons for this? Do you have any solutions for fixing the bug? Thanks!

Hello, I would like to study the wrist-mounted camera part in STL format in your article. Can you provide me with a file?

i do not have panda robotic arm?Can i realize the mimic play in UR robotic arm?If yes, how to make my own datasets for the specific task?

using obs modality: low_dim with keys: ['robot0_eef_pos', 'robot0_gripper_qpos', 'robot0_eef_quat']
using obs modality: rgb with keys: ['agentview_image', 'robot0_eye_in_hand_image']
using obs modality: depth with keys: []
using obs modality: scan with keys: []

============= Loaded Environment Metadata =============
obs key agentview_image with shape (120, 120, 3)
obs key robot0_eef_pos with shape (1, 4)
run failed with error:
"Unable to synchronously open object (object 'robot0_eef_quat' doesn't exist)"

Traceback (most recent call last):
File "scripts/train.py", line 371, in main
train(config, device=device)
File "scripts/train.py", line 78, in train
shape_meta = FileUtils.get_shape_metadata_from_dataset(
File "/home/lhy/robomimic/robomimic/utils/file_utils.py", line 144, in get_shape_metadata_from_dataset
initial_shape = demo["obs/{}".format(k)].shape[1:]
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/home/lhy/anaconda3/envs/mimicplay/lib/python3.8/site-packages/h5py/hl/group.py", line 357, in getitem
oid = h5o.open(self.id, self.e(name), lapl=self.lapl)
File "h5py/objects.pyx", line 54, in h5py.objects.with_phil.wrapper
File "h5py/objects.pyx", line 55, in h5py.objects.with_phil.wrapper
File "h5py/h5o.pyx", line 241, in h5py.h5o.open
KeyError: "Unable to synchronously open object (object 'robot0_eef_quat' doesn't exist)"
Hello, when I used the real-world data example you provided for training, I was able to successfully train the high-level planner. However, when I used the trained high-level planner to train the low-level controller, I encountered the above error. The explanation is that the hdf5 dataset file obtained using the video data you provided is missing the "robot0_eef_quat" key. The command I used for training is "python scripts/trainpy -- config config/low-level. json -- dataset'datasets/playdata/demo hand oc_1_new. hdv5 '- bddl_2' scripts/bddl_2/KITCHEN-SCENE9 eval task-1_turn_on \ Stove_put on \ ntove on \ nshelf. bddl '- videosprompt'datasets/eval task-1_turn_on \ Stove ut on \ s_on \ nshelf. bddl' tove_put-bowl-on_sthelf/image_demo. hdf5 '". Do you know how to solve it?