j96w / mimicplay Goto Github PK
View Code? Open in Web Editor NEW"MimicPlay: Long-Horizon Imitation Learning by Watching Human Play" code repository
License: MIT License
"MimicPlay: Long-Horizon Imitation Learning by Watching Human Play" code repository
License: MIT License
windows system
error:
ERROR: Failed building wheel for egl_probe
Running setup.py clean for egl_probe
Failed to build egl_probe
ERROR: Could not build wheels for egl_probe, which is required to install pyproject.toml-based projects
Hello, this is really impressive work.
I have been able to replicate the simulation part of training the high and low level planner using play data. I am currently focusing on training a high-level latent plan using human play data. To this end, I have successfully generated an HDF5 file containing all the relevant information from the human play data extracted from the provided videos. However, I have encountered a hurdle in adapting the existing code for training the high level planner to utilize my generated HDF5 file. The current training code appears to be tailored for simulation data, leading to errors when I attempt to integrate my dataset. Any guidance or direction you could offer would be immensely helpful.
Thank you!
Hi, I am trying to obtain a dataset with images from another camera pov by replaying the provided playdata/demo.hdf5
dataset through dataset_states_to_obs.py
. According to information from the robosuite documentation, I should be able to do this by modifying the agentview
camera params in _setup_camera()
in LIBERO/libero/libero/envs/problems/libero_kitchen_tabletop_manipulation.py
. It doesn't seem to work as the camera view stays the same when I visualize the converted dataset with playback_robomimic_dataset.py
. However, after I modified the said function, the rollout videos saved while training the low-level policy do follow the new camera configurations.
Does anyone have more insight into this or tried to do so themselves? Thanks a lot!
Hi,
Here https://mimic-play.github.io/ it seems you are using an iPhone to control the robotic arm. Could you please let me know how you did so? Is there any APP or open-source code to do so? Thanks.
Hi,what a brilliant work of mimicplay!
In fact,when implementing training lowlevel controller with my real-world playdata,aftering training my highlevel human playdata, error occured like this:
Traceback (most recent call last):
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/scripts/train.py", line 379, in main
train(config, device=device)
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/scripts/train.py", line 197, in train
step_log = TrainUtils.run_epoch(model=model, data_loader=train_loader, epoch=epoch, num_steps=train_num_steps)
File "/home/kiriyamagk/桌面/robomimic/robomimic/utils/train_utils.py", line 559, in run_epoch
info = model.train_on_batch(input_batch, epoch, validate=validate)
File "/home/kiriyamagk/桌面/robomimic/robomimic/algo/bc.py", line 137, in train_on_batch
predictions = self._forward_training(batch)
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/algo/mimicplay.py", line 328, in _forward_training
_, mlp_feature = self.human_nets.policy._get_latent_plan(batch['obs'], batch["goal_obs"])
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/algo/mimicplay.py", line 137, in _get_latent_plan
dists, enc_out, mlp_out = self.nets["policy"].forward_train( #调用策略网络
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/models/policy_nets.py", line 226, in forward_train
out, enc_out, mlp_out = MIMO_MLP.forward(self, return_latent=return_latent, obs=obs_dict, goal=goal_dict)
File "/home/kiriyamagk/桌面/MimicPlay/mimicplay/models/obs_nets.py", line 594, in forward
mlp_out = self.nets"mlp"
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/kiriyamagk/桌面/robomimic/robomimic/models/base_nets.py", line 286, in forward
return self._model(inputs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/kiriyamagk/anaconda3/envs/mimicplay/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 118, in forward
return F.linear(input, self.weight, self.bias)
However,error disappeared after I converted the script in demo_mp4.py like this:
all_hand_loc = np.concatenate((hand_loc_1[:, :, -2:], hand_loc[:, :, -2:]), axis=2)
,setting ac_dim=30
in the highlevel_human.json and finally retraining highlevel planner and lowlevel controller.Also,it seems no use setting ac_dim=40
in the lowlevel.json to train lowlevel controller successfully without changing all_hand_loc
in demo_mp4.py.
Hello! @j96w It's an excellent work of robot learning.
When I tried to process the data downloaded from the google drive using
python scripts/dataset_states_to_obs.py --dataset 'datasets/playdata/demo_modified.hdf5' --done_mode 0 --camera_names agentview robot0_eye_in_hand --camera_height 84 --camera_width 84 --output_name image_demo_local.hdf5 --exclude-next-obs
,
I got the render error about OpenGL
OpenGL.raw.EGL._errors.EGLError: EGLError( err = EGL_NOT_INITIALIZED, baseOperation = eglMakeCurrent, cArguments = ( <OpenGL._opaque.EGLDisplay_pointer object at 0x7fe287ca7dc0>, <OpenGL._opaque.EGLSurface_pointer object at 0x7fe2921ab840>, <OpenGL._opaque.EGLSurface_pointer object at 0x7fe2921ab840>, <OpenGL._opaque.EGLContext_pointer object at 0x7fe2921ab7c0>, ), result = 0 )
.
Have you ever met this problem? Or I would like to know if the version of the mujoco causes this matter. My mujoco version is 3.1.1.
Environment Libero_Kitchen_Tabletop_Manipulation not found. Make sure it is a registered environment among: Lift, Stack, NutAssembly, NutAssemblySingle, NutAssemblySquare, NutAssemblyRound, PickPlace, PickPlaceSingle, PickPlaceMilk, PickPlaceBread, PickPlaceCereal, PickPlaceCan, Door, Wipe, ToolHang, TwoArmLift, TwoArmPegInHole, TwoArmHandover, TwoArmTransport
I am very interested in your work. I encountered an error while running run_trained_agent.py. What could be the possible reasons for this? Do you have any solutions for fixing the bug? Thanks!
Hello, I would like to study the wrist-mounted camera part in STL format in your article. Can you provide me with a file?
i do not have panda robotic arm?Can i realize the mimic play in UR robotic arm?If yes, how to make my own datasets for the specific task?
using obs modality: low_dim with keys: ['robot0_eef_pos', 'robot0_gripper_qpos', 'robot0_eef_quat']
using obs modality: rgb with keys: ['agentview_image', 'robot0_eye_in_hand_image']
using obs modality: depth with keys: []
using obs modality: scan with keys: []
============= Loaded Environment Metadata =============
obs key agentview_image with shape (120, 120, 3)
obs key robot0_eef_pos with shape (1, 4)
run failed with error:
"Unable to synchronously open object (object 'robot0_eef_quat' doesn't exist)"
Traceback (most recent call last):
File "scripts/train.py", line 371, in main
train(config, device=device)
File "scripts/train.py", line 78, in train
shape_meta = FileUtils.get_shape_metadata_from_dataset(
File "/home/lhy/robomimic/robomimic/utils/file_utils.py", line 144, in get_shape_metadata_from_dataset
initial_shape = demo["obs/{}".format(k)].shape[1:]
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/home/lhy/anaconda3/envs/mimicplay/lib/python3.8/site-packages/h5py/hl/group.py", line 357, in getitem
oid = h5o.open(self.id, self.e(name), lapl=self.lapl)
File "h5py/objects.pyx", line 54, in h5py.objects.with_phil.wrapper
File "h5py/objects.pyx", line 55, in h5py.objects.with_phil.wrapper
File "h5py/h5o.pyx", line 241, in h5py.h5o.open
KeyError: "Unable to synchronously open object (object 'robot0_eef_quat' doesn't exist)"
Hello, when I used the real-world data example you provided for training, I was able to successfully train the high-level planner. However, when I used the trained high-level planner to train the low-level controller, I encountered the above error. The explanation is that the hdf5 dataset file obtained using the video data you provided is missing the "robot0_eef_quat" key. The command I used for training is "python scripts/trainpy -- config config/low-level. json -- dataset'datasets/playdata/demo hand oc_1_new. hdv5 '- bddl_2' scripts/bddl_2/KITCHEN-SCENE9 eval task-1_turn_on \ Stove_put on \ ntove on \ nshelf. bddl '- videosprompt'datasets/eval task-1_turn_on \ Stove ut on \ s_on \ nshelf. bddl' tove_put-bowl-on_sthelf/image_demo. hdf5 '". Do you know how to solve it?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.