siddhanthaldar / baku Goto Github PK

hi, thank you for your great contribution to multi task policy learning.
any suggestion for improvement on shaking and shivering especially for long horizonal task.
this one
https://baku-robot.github.io/#:~:text=BAKU%20on%20Long%20Horizon%20Tasks

$python train.py agent=baku suite=libero dataloader=libero suite/task=libero_90 suite.hidden_dim=256
when run above command, after all training steps are done and all snapshots are saved, I got below error. I did a bit debugging, and it seems that all BAKU python code are correctly run, but got below error.

Anybody knows why? (I paste all my pip packages at the end of this post)

Exception ignored in: <function MjRenderContext.del at 0x149b885b9160>
Traceback (most recent call last):
File "/home/zhangfaen/miniconda3/envs/baku/lib/python3.9/site-packages/robosuite/utils/binding_utils.py", line 199, in del
self.gl_ctx.free()
File "/home/zhangfaen/miniconda3/envs/baku/lib/python3.9/site-packages/robosuite/renderers/context/egl_context.py", line 149, in free
EGL.eglMakeCurrent(EGL_DISPLAY, EGL.EGL_NO_SURFACE, EGL.EGL_NO_SURFACE, EGL.EGL_NO_CONTEXT)
File "src/errorchecker.pyx", line 58, in OpenGL_accelerate.errorchecker._ErrorChecker.glCheckError
OpenGL.raw.EGL._errors.EGLError: EGLError(
err = EGL_NOT_INITIALIZED,
baseOperation = eglMakeCurrent,
cArguments = (
<OpenGL._opaque.EGLDisplay_pointer object at 0x149b3da80c40>,
<OpenGL._opaque.EGLSurface_pointer object at 0x149b886a18c0>,
<OpenGL._opaque.EGLSurface_pointer object at 0x149b886a18c0>,
<OpenGL._opaque.EGLContext_pointer object at 0x149b886a17c0>,
),
result = 0
)
Exception ignored in: <function EGLGLContext.del at 0x149b885adf70>
Traceback (most recent call last):
File "/home/zhangfaen/miniconda3/envs/baku/lib/python3.9/site-packages/robosuite/renderers/context/egl_context.py", line 155, in del
self.free()
File "/home/zhangfaen/miniconda3/envs/baku/lib/python3.9/site-packages/robosuite/renderers/context/egl_context.py", line 149, in free
EGL.eglMakeCurrent(EGL_DISPLAY, EGL.EGL_NO_SURFACE, EGL.EGL_NO_SURFACE, EGL.EGL_NO_CONTEXT)
File "src/errorchecker.pyx", line 58, in OpenGL_accelerate.errorchecker._ErrorChecker.glCheckError
OpenGL.raw.EGL._errors.EGLError: EGLError(
err = EGL_NOT_INITIALIZED,
baseOperation = eglMakeCurrent,
cArguments = (
<OpenGL._opaque.EGLDisplay_pointer object at 0x149b3da80c40>,
<OpenGL._opaque.EGLSurface_pointer object at 0x149b886a18c0>,
<OpenGL._opaque.EGLSurface_pointer object at 0x149b886a18c0>,
<OpenGL._opaque.EGLContext_pointer object at 0x149b886a17c0>,
),
result = 0
)

$pip freeze
%pip freeze
absl-py @ file:///C:/ci/absl-py_1624299085608/work
antlr4-python3-runtime==4.8
anyio @ file:///tmp/build/80754af9/anyio_1644463572971/work/dist
argon2-cffi @ file:///opt/conda/conda-bld/argon2-cffi_1645000214183/work
argon2-cffi-bindings @ file:///tmp/build/80754af9/argon2-cffi-bindings_1644569679365/work
asttokens @ file:///opt/conda/conda-bld/asttokens_1646925590279/work
attrs @ file:///croot/attrs_1695717823297/work
Babel @ file:///croot/babel_1671781930836/work
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
bddl==3.5.0
beautifulsoup4 @ file:///croot/beautifulsoup4-split_1718029820055/work
black==24.4.2
bleach @ file:///opt/conda/conda-bld/bleach_1641577558959/work
Brotli @ file:///croot/brotli-split_1714483155106/work
certifi @ file:///croot/certifi_1717618050233/work/certifi
cffi @ file:///croot/cffi_1714483155441/work
cfgv==3.4.0
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
click @ file:///croot/click_1698129812380/work
cloudpickle @ file:///croot/cloudpickle_1683040006038/work
comm @ file:///croot/comm_1709322850197/work
contourpy @ file:///croot/contourpy_1700583582875/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
cytoolz @ file:///croot/cytoolz_1701723583781/work
dask @ file:///croot/dask-core_1715838601404/work
debugpy @ file:///croot/debugpy_1690905042057/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
decord==0.6.0
defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work
diffusers==0.11.1
distlib==0.3.8
dm-env==1.6
dm-tree==0.1.8
dm_control==1.0.20
easydict==1.13
egl_probe==1.0.2
einops==0.7.0
etils==1.5.2
evdev==1.7.1
exceptiongroup @ file:///croot/exceptiongroup_1706031385326/work
executing @ file:///opt/conda/conda-bld/executing_1646925071911/work
fancycompleter==0.9.1
Farama-Notifications==0.0.4
fastjsonschema @ file:///opt/conda/conda-bld/python-fastjsonschema_1661371079312/work
filelock @ file:///croot/filelock_1700591183607/work
fonttools @ file:///croot/fonttools_1713551344105/work
fsspec @ file:///croot/fsspec_1714461537038/work
future==1.0.0
glfw==2.7.0
gmpy2 @ file:///tmp/build/80754af9/gmpy2_1645438755360/work
grpcio==1.64.1
gym==0.22.0
gym-notices==0.0.8
gymnasium==0.29.1
h5py==3.11.0
huggingface-hub==0.23.3
hydra-core==1.1.0
hydra-submitit-launcher==1.1.5
identify==2.5.36
idna @ file:///croot/idna_1714398848350/work
imagecodecs @ file:///croot/imagecodecs_1695064943445/work
imageio==2.9.0
imageio-ffmpeg==0.4.4
importlib-metadata @ file:///croot/importlib_metadata-suite_1704813515092/work
importlib-resources @ file:///croot/importlib_resources-suite_1704281845041/work
iniconfig==2.0.0
ipdb==0.13.9
ipykernel @ file:///croot/ipykernel_1705933831282/work
ipython @ file:///croot/ipython_1694181358621/work
ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
jedi @ file:///tmp/build/80754af9/jedi_1644297102865/work
Jinja2 @ file:///croot/jinja2_1716993405101/work
joblib==1.4.2
json5 @ file:///tmp/build/80754af9/json5_1624432770122/work
jsonschema @ file:///croot/jsonschema_1699041609003/work
jsonschema-specifications @ file:///croot/jsonschema-specifications_1699032386549/work
jupyter-packaging @ file:///tmp/build/80754af9/jupyter-packaging_1613502826984/work
jupyter-server @ file:///croot/jupyter_server_1671707632269/work
jupyter_client @ file:///croot/jupyter_client_1699455897726/work
jupyter_core @ file:///croot/jupyter_core_1698937308754/work
jupyterlab @ file:///tmp/build/80754af9/jupyterlab_1619133235951/work
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
jupyterlab_server @ file:///croot/jupyterlab_server_1699555425460/work
jupytext==1.16.2
kiwisolver @ file:///croot/kiwisolver_1672387140495/work
labmaze==1.0.6
-e git+https://github.com/Lifelong-Robot-Learning/LIBERO.git@f78abd68ee283de9f9be3c8f7e2a9ad60246e95c#egg=libero
llvmlite==0.43.0
locket @ file:///opt/conda/conda-bld/locket_1652903118915/work
lxml==5.2.2
Markdown==3.6
markdown-it-py==3.0.0
MarkupSafe @ file:///croot/markupsafe_1704205993651/work
matplotlib==3.4.2
matplotlib-inline @ file:///opt/conda/conda-bld/matplotlib-inline_1662014470464/work
mdit-py-plugins==0.4.1
mdurl==0.1.2
-e git+https://github.com/siddhanthaldar/Metaworld.git@65a7c33c8326cadfef7e19590af98b2bf2640cb7#egg=metaworld&subdirectory=../../Metaworld
mistune @ file:///opt/conda/conda-bld/mistune_1661496219659/work
mkl-fft @ file:///croot/mkl_fft_1695058164594/work
mkl-random @ file:///croot/mkl_random_1695059800811/work
mkl-service==2.4.0
mpmath @ file:///croot/mpmath_1690848262763/work
mujoco==3.1.6
mypy-extensions==1.0.0
nbclassic @ file:///croot/nbclassic_1681756170010/work
nbclient @ file:///croot/nbclient_1698934205032/work
nbconvert @ file:///croot/nbconvert_1699022732553/work
nbformat @ file:///croot/nbformat_1694616755618/work
nest-asyncio @ file:///croot/nest-asyncio_1708532673751/work
networkx @ file:///croot/networkx_1717597493534/work
nltk==3.8.1
nodeenv==1.9.1
notebook_shim @ file:///croot/notebook-shim_1699455894279/work
numba==0.60.0
numpy @ file:///croot/numpy_and_numpy_base_1708638617955/work/dist/numpy-1.26.4-cp39-cp39-linux_x86_64.whl#sha256=6094eeedd869502faa0fd0a8c5ad3a70c5779be06ddd1feb7627e5c212fac420
omegaconf==2.1.2
opencv-python==4.5.3.56
packaging @ file:///croot/packaging_1710807400464/work
pandas==1.3.0
pandocfilters @ file:///opt/conda/conda-bld/pandocfilters_1643405455980/work
parso @ file:///opt/conda/conda-bld/parso_1641458642106/work
partd @ file:///croot/partd_1698702562572/work
pathspec==0.12.1
pdbpp==0.10.3
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
pillow @ file:///croot/pillow_1714398848491/work
platformdirs @ file:///croot/platformdirs_1692205439124/work
pluggy==1.5.0
pre-commit==3.7.1
prometheus-client @ file:///tmp/abs_d3zeliano1/croots/recipe/prometheus_client_1659455100375/work
prompt-toolkit @ file:///croot/prompt-toolkit_1704404351921/work
protobuf==3.20.1
psutil @ file:///opt/conda/conda-bld/psutil_1656431268089/work
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure-eval @ file:///opt/conda/conda-bld/pure_eval_1646925070566/work
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
Pygments @ file:///croot/pygments_1684279966437/work
pynput==1.7.7
PyOpenGL==3.1.7
PyOpenGL-accelerate==3.1.7
pyparsing==3.1.2
pyrepl==0.9.0
PySocks @ file:///tmp/build/80754af9/pysocks_1605305812635/work
pytest==8.2.2
python-dateutil @ file:///croot/python-dateutil_1716495738603/work
python-xlib==0.33
pytz @ file:///croot/pytz_1713974312559/work
pywavelets @ file:///croot/pywavelets_1705049820073/work
PyYAML @ file:///croot/pyyaml_1698096049011/work
pyzmq @ file:///croot/pyzmq_1705605076900/work
referencing @ file:///croot/referencing_1699012038513/work
regex==2024.5.15
requests @ file:///croot/requests_1716902831423/work
robomimic==0.3.0
robosuite==1.4.1
rpds-py @ file:///croot/rpds-py_1698945930462/work
safetensors==0.4.3
scikit-image==0.18.1
scikit-learn==1.5.0
scipy @ file:///croot/scipy_1717521478074/work/dist/scipy-1.13.1-cp39-cp39-linux_x86_64.whl#sha256=2cb3d1436f8762a53161486cb9493ccc305fc760e8f830f82888792cbaf99099
Send2Trash @ file:///croot/send2trash_1699371139552/work
sentence-transformers==3.0.1
six @ file:///tmp/build/80754af9/six_1644875935023/work
sklearn==0.0
sniffio @ file:///croot/sniffio_1705431295498/work
soupsieve @ file:///croot/soupsieve_1696347547217/work
stack-data @ file:///opt/conda/conda-bld/stack_data_1646927590127/work
submitit==1.5.1
sympy @ file:///croot/sympy_1701397643339/work
tb-nightly==2.18.0a20240613
tensorboard==2.17.0
tensorboard-data-server==0.7.2
tensorboardX==2.6.2.2
termcolor==1.1.0
terminado @ file:///croot/terminado_1671751832461/work
threadpoolctl==3.5.0
tifffile @ file:///croot/tifffile_1695107451082/work
tinycss2 @ file:///croot/tinycss2_1668168815555/work
tokenizers==0.19.1
toml==0.10.2
tomli==2.0.1
toolz @ file:///croot/toolz_1667464077321/work
torch==2.3.1
torchaudio==2.3.1
torchvision==0.18.1
tornado @ file:///croot/tornado_1696936946304/work
tqdm==4.66.4
traitlets @ file:///croot/traitlets_1718227057033/work
transformers==4.41.2
triton==2.3.1
typing_extensions @ file:///croot/typing_extensions_1715268824938/work
unicodedata2 @ file:///croot/unicodedata2_1713212950228/work
urllib3 @ file:///croot/urllib3_1715635851070/work
virtualenv==20.26.2
wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work
webencodings==0.5.1
websocket-client @ file:///croot/websocket-client_1715878298792/work
Werkzeug==3.0.3
wmctrl==0.5
yapf==0.31.0
zipp @ file:///croot/zipp_1704206909481/work

Hi, I get an error every time I run eval.py. Could you please give me some advice? Thank you very much!

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):