How to get the pose information from the generate image？ about blenderproc

[zhangxiaodi]

Thanks for your reply.

[RoyCLR]

Adding a little more info.

Upon running the given code, turns out the payload is a string representation of a python object containing the camera pose

cam_pose = np.array(file["campose"])
print(cam_pose)

out[47]: array(b'[{"cam2world_matrix": [[1.0, 0.0, 0.0, 0.0], [0.0, 0.2674988806247711, -0.9635581970214844, -13.741000175476074], [0.0, 0.9635581970214844, 0.2674988806247711, 4.124199867248535], [0.0, 0.0, 0.0, 1.0]], "cam_K": [[468.60487196180554, 0.0, 255.5], [0.0, 468.60487196180554, 255.5], [0.0, 0.0, 1.0]]}]',
      dtype='|S299')

To extract the data to a usable form:

# eval runs arbitrary code, only run this when you trust content of cam_pose do not contain malicious code
decoded_cam_pose = eval(cam_pose)  

# can now access as a proper python list. eg
decoded_cam_pose[0]["cam_K"]

out[48]: [[468.60487196180554, 0.0, 255.5],
 [0.0, 468.60487196180554, 255.5],
 [0.0, 0.0, 1.0]]

[themasterlink]

Yes, we can render images from randomized camera locations.

If I understand you correctly you should check out our example on camera sampling.

Before you start with that please look at the basic examples to get familiar with our pipeline.

If you have any questions, just ask them.

[zhangxiaodi]

Thanks for your demo. After browsing the camera sampling， we still have a question that how to get the R Matrix and t vector？ Is that can transform from the value max and min？

[themasterlink]

There are several steps involved here:

1. The CameraSampler samples new camera poses, each sampled camera is saved as a keyframe in blender.

"location": { # this here sets the t value of the camera
  "provider":"sampler.Uniform3d", # a provider always returns a value
  "max":[10, 10, 8], # this provider samples uniformly in the given bounding box
  "min":[-10, -10, 12]
},
"rotation": { # here you set the rotation of the camera, we usually use Euler angles
  "format": "look_at",
  "value": {
    "provider": "getter.POI" # this makes sure that the camera looks at the loaded objects
  }
}

2. Rendering of each camera sample this all happens automatically when calling the RGBRenderer module.

3. Hdf5Writer collects all generated images and saves them for each camera frame in one container, if you know also want the camera location stored inside of that .hdf5 container then you can use the CameraStateWriter.

So far we have only written the location (t vector) and the euler angles -> if you need a rotation matrix then you could use the following inside of the CameraStateWriter:

rot_mat = mathutils.Euler(euler_angles, 'XYZ').to_matrix()

One last bit of information, to better understand providers check out: Entity Manipulation

[zhangxiaodi]

Sorry， when I use the CameraStateWriter in config.yaml, but get the null value in campose.
Merging data for frame 0 into /home/znb/research/BlenderProc/examples/camera_sampling/output/0.hdf5
Key: campose - shape: () - dtype: |S158 - path: /dev/shm/blender_proc_13058/campose_0000.npy
Key: normals - shape: (512, 512, 3) - dtype: float64 - path: /dev/shm/blender_proc_13058/normal_0000.exr
Key: colors - shape: (512, 512, 3) - dtype: uint8 - path: /dev/shm/blender_proc_13058/rgb_0000.png
Merging data for frame 1 into /home/znb/research/BlenderProc/examples/camera_sampling/output/1.hdf5
Key: campose - shape: () - dtype: |S157 - path: /dev/shm/blender_proc_13058/campose_0001.npy
Key: normals - shape: (512, 512, 3) - dtype: float64 - path: /dev/shm/blender_proc_13058/normal_0001.exr
Key: colors - shape: (512, 512, 3) - dtype: uint8 - path: /dev/shm/blender_proc_13058/rgb_0001.png
Merging data for frame 2 into /home/znb/research/BlenderProc/examples/camera_sampling/output/2.hdf5
Key: campose - shape: () - dtype: |S158 - path: /dev/shm/blender_proc_13058/campose_0002.npy
Key: normals - shape: (512, 512, 3) - dtype: float64 - path: /dev/shm/blender_proc_13058/normal_0002.exr
Key: colors - shape: (512, 512, 3) - dtype: uint8 - path: /dev/shm/blender_proc_13058/rgb_0002.png
Merging data for frame 3 into /home/znb/research/BlenderProc/examples/camera_sampling/output/3.hdf5
Key: campose - shape: () - dtype: |S152 - path: /dev/shm/blender_proc_13058/campose_0003.npy
Key: normals - shape: (512, 512, 3) - dtype: float64 - path: /dev/shm/blender_proc_13058/normal_0003.exr
Key: colors - shape: (512, 512, 3) - dtype: uint8 - path: /dev/shm/blender_proc_13058/rgb_0003.png
Merging data for frame 4 into /home/znb/research/BlenderProc/examples/camera_sampling/output/4.hdf5
Key: campose - shape: () - dtype: |S158 - path: /dev/shm/blender_proc_13058/campose_0004.npy
Key: normals - shape: (512, 512, 3) - dtype: float64 - path: /dev/shm/blender_proc_13058/normal_0004.exr
Key: colors - shape: (512, 512, 3) - dtype: uint8 - path: /dev/shm/blender_proc_13058/rgb_0004.png

this is my yaml file:

Args: <obj_file> <output_dir>

{
"version": 2,
"setup": {
"blender_install_path": "/home/env:USER/blender/",
"pip": [
"h5py"
]
},
"global": {
"all": {
"output_dir": "args:1"
}
},
"modules": [
{
"module": "main.Initializer",
"config": {}
},
{
"module": "loader.ObjectLoader",
"config": {
"path": "args:0"
}
},
{
"module": "lighting.LightLoader",
"config": {
"lights": [
{
"type": "POINT",
"location": [5, -5, 5],
"energy": 1000
}
]
}
},
{
"module": "camera.CameraSampler",
"config": {
"cam_poses": [
{
"number_of_samples": 5,
"location": {
"provider":"sampler.Uniform3d",
"max":[10, 10, 8],
"min":[-10, -10, 12]
},
"rotation": {
"value": {
"provider":"sampler.Uniform3d",
"max":[1.2217, 0, 6.283185307],
"min":[1.2217, 0, 0]
}
}
}
]
}
},
{
"module": "writer.CameraStateWriter",
"config": {
"attributes_to_write": ["location", "rotation_euler"]
}
},

{
  "module": "renderer.NormalRenderer",
  "config": {
    "output_key": "normals"
  }
},
{
  "module": "renderer.RgbRenderer",
  "config": {
    "output_key": "colors",
    "samples": 350
  }
},

{
  "module": "writer.Hdf5Writer",
  "config": {
  }
}

]
}

[themasterlink]

Okay, it is not empty is just saved as a numpy array.

All data here can be usually stored in a tensor format (color and normal images for example).
The problem is, this is a dictionary not a tensor, the solution now to this is:

import numpy as np
import h5py

with h5py.File(file_path, "r") as file:
    cam_pose = np.array(file["campose"])
    print(cam_pose)

We will try to find a better solution for this.

Does this answer your question?