ollieboyne / blendersynth

Synthetic Blender Dataset Production

Documentation | GitHub | PyPI

BlenderSynth is a Python library for generating large scale synthetic datasets using Blender. Compared to other tools, BlenderSynth provides support for: Custom Shader AOVs to render rich per-pixel information; node control; multithreading support; and multiview rendering support.

Usage

See documentation for installation, debugging/troubleshooting and examples.

BlenderSynth supports Blender 3 and 4.

Contributions and Projects

This project is currently in Beta. Please let me know what new features you would like, or feel free to make a pull request!

If you use BlenderSynth for a project, please contact me about it - I might include it in the documentation as a usage example!

Citing

If you use BlenderSynth in your work, please cite:

@software{blendersynth,
  author       = {Ollie Boyne},
  title        = {BlenderSynth},
  year         = 2023,
  publisher    = {GitHub},
  url          = {https://ollieboyne.github.io/BlenderSynth},
}

Benchmarking

Rendering speed compared to BlenderProc:

blendersynth's Issues

Not freed memory blocks

Hi! Thanks for your great work!

I'm new to blender. Currently I want to generate a few images for one single reflective object with rgb, depth, normal, and camera poses to test the 3D reconstruction algos.

When I'm running the quick start code directly from python, I got this error:

Warning: Falling back to the standard locale ("C")
Traceback (most recent call last):
  File "/workspace/gs_toolkit/data/datasynthetic/object_generation.py", line 3, in <module>
    import blendersynth as bsyn
  File "/home/user/.local/lib/python3.10/site-packages/blendersynth/__init__.py", line 100, in <module>
    has_loaded_scene = ".blend" in sys.argv[1]
IndexError: list index out of range
Error: Not freed memory blocks: 27594, total unfreed memory 9.226070 MB

The environment is

• Ubuntu22.04(docker)
• python3.10

The memory should be sufficient. Do you have any suggestions? Thanks for helping!

Texture for rgb render

Hi!
When I'm rendering the RGB image, I found the texture in the mesh is ignored.
My input mesh is:

and the RGB output is:

the loading and rendering code is:

import blendersynth as bsyn
import numpy as np
import os

bsyn.run_this_script()

# monkey = bsyn.Mesh.from_primitive('monkey')
mesh = bsyn.Mesh.from_obj('data/milk_mesh.obj')

bsyn.render.set_resolution(640, 480)
bsyn.render.set_cycles_samples(10)

comp = bsyn.Compositor()
obj_pass_idx = mesh.assign_pass_index(1)  # To show masking, we assign a pass index to the first object

# we will create 4 cameras, one from each side of the mesh, facing the mesh
# TODO: increase camera numbers
cameras = []
camera_radius = 1
for i in range(4):
    camera = bsyn.Camera.create(name=f'Cam{i}',
                                location = (camera_radius * np.cos(i * np.pi / 2), camera_radius * np.sin(i * np.pi / 2), 0))
    camera.look_at_object(mesh)
    # camera.set_fov(20)  # zoom in
    cameras.append(camera)

	# we'll add a point light at each camera too
    light = bsyn.Light.create('POINT', location=camera.location, intensity=250)

bsyn.render.render_depth()  # Enable standard Blender depth pass
depth_vis = comp.get_depth_visual(max_depth=20)  # Create a visual of the depth pass
rgb_mask = comp.get_mask(obj_pass_idx, 'Image')  # create an RGB mask (i.e. only render mesh)
bounding_box_visual = comp.get_bounding_box_visual()
# keypoints_visual = comp.get_keypoints_visual()  # Create a visual of keypoints


# we'll render RGB, normals, and bounding boxes
normal_aov = bsyn.aov.NormalsAOV(polarity=[-1, 1, -1])
instancing_aov = bsyn.aov.InstanceRGBAOV()
class_aov = bsyn.aov.ClassRGBAOV()
UVAOV = bsyn.aov.UVAOV()  # UV Coordinates
NOCAOV = bsyn.aov.GeneratedAOV()  # Normalized Object Coordinates (NOC)

for aov in [normal_aov, instancing_aov, class_aov, UVAOV, NOCAOV]:
    mesh.assign_aov(aov)

output_folder = 'multiview'
os.makedirs(output_folder, exist_ok=True)

# All of the following will have Blender's Filmic colour correction by default
comp.define_output('Image', file_name='rgb', directory=output_folder+"/rgb")
comp.define_output(rgb_mask, name='rgb_masked', directory=output_folder+"/rgb_masked") # render RGB layer masked by mesh
comp.define_output(bounding_box_visual, output_folder+"/bounding_box", name='bounding_box_visual')
# comp.define_output(keypoints_visual, output_folder, name='keypoints')

# All of the following will not have any colour correction
comp.define_output(normal_aov, file_name='normals', directory=output_folder+"/normals")
comp.define_output(depth_vis, output_folder+"/depth_vis")  # render visual of depth layer
comp.define_output(instancing_aov, output_folder+"/instancing", name='instancing')
comp.define_output(class_aov, output_folder+"/semantic", name='semantic')
comp.define_output(UVAOV, output_folder+"/UV", name='UV')
comp.define_output(NOCAOV, output_folder+"/NOC", name='NOC')
comp.define_output('Depth', output_folder+"/depth", file_format='OPEN_EXR')

bounding_boxes = bsyn.annotations.bounding_boxes([mesh], cameras)
# keypoints = bsyn.annotations.keypoints.project_keypoints(cube_vertices)

comp.render(camera=cameras, annotations=bounding_boxes)

Is it because of the setting problem? If not, do you have any plan to add texture to the RGB rendering?
Thank you!

Extrinsic and Intrinsic extraction

Hi, as we discussed, I'm working on extracting the camera parameters. I think I'm quite close but there is one issue that I might need help. Here is my current intrinsic extraction code:

# Generate the transform.json
# extrinsics = [extrinsic.tolist() for extrinsic in extrinsics]
focal_length = camera.focal_length
sensor_width = camera.data.sensor_width
sensor_height = camera.data.sensor_height
# TODO: pixel_aspect_ratio
s_u = w / sensor_width
s_v = h / sensor_height
cx = w / 2
cy = h / 2

data = {}

data["w"] = w
data["h"] = h
data["fl_x"] = focal_length * s_u
data["fl_y"] = focal_length * s_v
data["cx"] = cx
data["cy"] = cy
data["k1"] = 0
data["k2"] = 0
data["p1"] = 0
data["p2"] = 0
data["camera_model"] = "OPENCV"
data["ply_file_path"] = "colmap/point_cloud.ply"
data["applied_scale"] = 1.0

I need to get the pixel_aspect_ratio for fy. In the blender, I found an example implemented like this:

scene = bpy.context.scene
pixel_aspect_ratio = scene.render.pixel_aspect_x / scene.render.pixel_aspect_y
s_u = width / sensor_width
s_v = height * pixel_aspect_ratio / sensor_height

May I know where I can get the pixel_aspect_x and pixel_aspect_y in this project? Thank you so much for helping!