j3soon / omniisaacgymenvs-dofbotreacher Goto Github PK

I'm currently encountering some challenges with the sim2real transition. To provide a clearer picture of the issue, I've attached 3 videos that include demonstrations and a snapshot

I am receiving error messages when I switch Sim2Real to True and then run the Sim2real command. I would greatly appreciate your expertise in guiding me through a debugging process or providing any insights on potential solutions.

https://github.com/j3soon/OmniIsaacGymEnvs-DofbotReacher/assets/1728575/c2ff81a4-07e4-4ad7-af1a-1df884d6ddc1
https://github.com/j3soon/OmniIsaacGymEnvs-DofbotReacher/assets/1728575/8614bc2e-31ed-4ed9-a994-0a29fd45fe06
https://github.com/j3soon/OmniIsaacGymEnvs-DofbotReacher/assets/1728575/e3a85944-c4de-46e3-af59-95c922ba8e1f

Thank you in advance for your time and assistance. Looking forward to your valuable input.

Just wondering what steps I need to take to get it set up on windows, thanks!

Fisrt, I want to say thank you for making this repo, it's really helpfull.

Now I m trying to custom this to a dofbot can pick a thing (Dynamic Cuboid). I get stuck from the first step - pick and place a dynamic cuboid to environment.

I did it bellow:
` def get_object(self):
self.object_start_translation = torch.tensor([0.0, 0.0, 0.0], device=self.device)
self.object_start_orientation = torch.tensor([1.0, 0.0, 0.0, 0.0], device=self.device)

    # self.object_usd_path = "C:\\Users\\tpnei\\Downloads\\random_cube.usd"
    # add_reference_to_stage(self.object_usd_path, self.default_zero_env_path + "/object")

    obj = DynamicCuboid(
                prim_path=self.default_zero_env_path + "/object/object",
                name="object",
                position=np.array([0.221, 0.221, 0.2]),
                scale=np.array([0.03, 0.03, 0.07]),
                color=np.array([0, 0, 1.0]),
                #translation=self.object_start_translation,
                orientation=self.object_start_orientation,
    )
    self._sim_config.apply_articulation_settings(
        "object", get_prim_at_path(obj.prim_path),
        self._sim_config.parse_actor_config("object")
    )`

and goal

` def get_goal(self):
self.goal_displacement_tensor = torch.tensor([0.0, 0.0, 0.0], device=self.device)
self.goal_start_translation = torch.tensor([0.0, 0.0, 0.0], device=self.device) + self.goal_displacement_tensor
self.goal_start_orientation = torch.tensor([1.0, 0.0, 0.0, 0.0], device=self.device)

    # self.goal_usd_path = "C:\\Users\\tpnei\\Downloads\\random_cube.usd"
    # add_reference_to_stage(self.goal_usd_path, self.default_zero_env_path + "/goal")

    goal = DynamicCuboid(
                prim_path=self.default_zero_env_path + "/goal/object",
                name="goal",
                position=np.array([0.221, 0.221, 0.2]),
                scale=np.array([0.03, 0.03, 0.07]),
                color=np.array([0, 0, 1.0]),
                #translation=self.goal_start_translation,
                orientation=self.goal_start_orientation,
    )
    self._sim_config.apply_articulation_settings(
        "goal", get_prim_at_path(goal.prim_path),
        self._sim_config.parse_actor_config("goal_object"))`

and also the set_up_scene

` def set_up_scene(self, scene: Scene) -> None:
self._stage = get_current_stage()
#self._assets_root_path = 'omniverse://localhost/Projects/J3soon/Isaac/2022.1'
self.get_arm()
self.get_object()
self.get_goal()

    super().set_up_scene(scene)
    self._arms = self.get_arm_view(scene)
    scene.add(self._arms)
    self._objects = RigidPrimView(
        prim_paths_expr="/World/envs/env_.*/object/object",
        name="object_view",
        reset_xform_properties=False,
    )
    scene.add(self._objects)
    self._goals = RigidPrimView(
        prim_paths_expr="/World/envs/env_.*/goal/object",
        name="goal_view",
        reset_xform_properties=False,
    )
    scene.add(self._goals)`

I am new to this, so i really dont understand that the log export when i run "python omniisaacgymenvs/scripts/dummy_dofbot_policy.py task=DofbotReacher test=True num_envs=1"

2023-12-19 03:47:41 [35,763ms] [Error] [carb.graphics-direct3d.plugin] NGX EvaluateFeature failed: 0xbad00002
2023-12-19 03:47:41 [35,766ms] [Error] [carb.graphics-direct3d.plugin] Failed to evaluate DLSS feature.
2023-12-19 03:47:41 [35,766ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:41 [35,766ms] [Error] [carb.graphics-direct3d.plugin] CommandList::Close() failed.
2023-12-19 03:47:41 [35,768ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:41 [35,768ms] [Error] [carb.graphics-direct3d.plugin] Closing command list failed.
2023-12-19 03:47:41 [35,768ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to execute RenderGraph on device 0. Error Code: 1
2023-12-19 03:47:41 [35,770ms] [Error] [gpu.foundation.plugin] Failed to submit RenderGraph commands
2023-12-19 03:47:43 [37,368ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:43 [37,368ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2023-12-19 03:47:43 [37,368ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0.
2023-12-19 03:47:43 [37,368ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp 9
2023-12-19 03:47:43 [37,369ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp GBuffer RT. Will not execute this or subsequent RenderGraph operations. Aborting RenderGraph execution
2023-12-19 03:47:44 [37,998ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:44 [37,998ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2023-12-19 03:47:44 [37,998ms] [Error] [gpu.foundation.plugin] subAllocate() failed for AccelStruct size: 74318464
2023-12-19 03:47:44 [37,998ms] [Error] [gpu.foundation.plugin] AccelStruct creation failed for the device: 0.
2023-12-19 03:47:44 [37,999ms] [Error] [rtx.scenedb.plugin] getResourceFromAccelStructDesc() failed for compaction size: 74318464
2023-12-19 03:47:44 [38,039ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:44 [38,039ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2023-12-19 03:47:44 [38,040ms] [Error] [gpu.foundation.plugin] subAllocate() failed for AccelStruct size: 11307264
2023-12-19 03:47:44 [38,040ms] [Error] [gpu.foundation.plugin] AccelStruct creation failed for the device: 0.
2023-12-19 03:47:44 [38,040ms] [Error] [rtx.scenedb.plugin] getResourceFromAccelStructDesc() failed for compaction size: 11307264
2023-12-19 03:47:44 [38,041ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:44 [38,041ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2023-12-19 03:47:44 [38,041ms] [Error] [gpu.foundation.plugin] subAllocate() failed for AccelStruct size: 11239040
2023-12-19 03:47:44 [38,042ms] [Error] [gpu.foundation.plugin] AccelStruct creation failed for the device: 0.
2023-12-19 03:47:44 [38,042ms] [Error] [rtx.scenedb.plugin] getResourceFromAccelStructDesc() failed for compaction size: 11239040
2023-12-19 03:47:44 [38,043ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:44 [38,043ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2023-12-19 03:47:44 [38,043ms] [Error] [gpu.foundation.plugin] subAllocate() failed for AccelStruct size: 13257088
2023-12-19 03:47:44 [38,043ms] [Error] [gpu.foundation.plugin] AccelStruct creation failed for the device: 0.
2023-12-19 03:47:44 [38,044ms] [Error] [rtx.scenedb.plugin] getResourceFromAccelStructDesc() failed for compaction size: 13257088
2023-12-19 03:47:45 [39,420ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:45 [39,420ms] [Error] [carb.graphics-direct3d.plugin] Closing command list failed.
2023-12-19 03:47:45 [39,421ms] [Error] [gpu.foundation.plugin] Failed to submit RenderGraph commands
2023-12-19 03:47:45 [39,469ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:45 [39,469ms] [Error] [carb.graphics-direct3d.plugin] CommandList::Close() failed.
2023-12-19 03:47:45 [39,470ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:45 [39,470ms] [Error] [carb.graphics-direct3d.plugin] Closing command list failed.
2023-12-19 03:47:45 [39,470ms] [Error] [gpu.foundation.plugin] Failed to submit RenderGraph commands
2023-12-19 03:47:45 [39,510ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:45 [39,510ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2023-12-19 03:47:45 [39,510ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2023-12-19 03:47:45 [39,511ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2023-12-19 03:47:45 [39,511ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp 8
2023-12-19 03:47:45 [39,625ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:45 [39,625ms] [Error] [carb.graphics-direct3d.plugin] Closing command list failed.
2023-12-19 03:47:45 [39,625ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0x8007000e
2023-12-19 03:47:45 [39,626ms] [Error] [carb.graphics-direct3d.plugin] Closing command list failed.

I really need your help (or anyone). Thanks for reading it!

I am getting an error in Step 3 saying "AttributeError 'NoneType' object has not attribute 'GetRootLayer'" when running the code from mniisaacgymenvs/utils/usd_utils/create_instanceable_dofbot.py in the script editor of Isaac Gym.

Hi,

I am working through the training example for Dofbot reacher, and wanting to deploy it to a real arm.

I am wondering how I will be able to export the saved .pth to an actual dofbot and run it. What kinds of inputs to the loaded model from the .pth will I need to give it, and what will the output be?

I’m aware that I may be able to use PyTorch to load the model, but I don’t know what kinds of inputs I will need to give it.

In the Config for this task the observation type is set to full. What exactly does the “full” observation look like?

I'm assuming that the full state is like this based on the dofbot_reacher.py code

    self.num_obs_dict = {
        "full": 29,
        # 6: dofbot joints position (action space)
        # 6: dofbot joints velocity
        # 3: goal position
        # 4: goal rotation
        # 4: goal relative rotation
        # 6: previous action
    }

Can you explain to me what each of these values in the state dictionary are?

How exactly are actions being generated from the model for the Dofbot? The exported model I have has a mu, logstd, and value output. How are these being used? Can you show me in the code where predictions are being made for the actions?