Coder Social home page Coder Social logo

blockfusion's Introduction

BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation

ACM Transactions on Graphics (SIGGRAPH'24)

This repo contains the official implementation for the paper "BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation".

drawing

drawing

Installation

Pytorch3d is required for postprocess. Come to offical page for installation instructions.

We modified diffusers to adapt triplane structure. Users should build based on the ./src

# clone this repo 
python setup.py install

Pretrained model

Pretrained weights of MLP, VAE, and Diffuser(Condition/Unconditioned). Download the model here [10GB] and extract to ./checkpoints.

Inference

To do uncondtional inference, run

python unconditioned_prediction.py --batch 4 --save output/uncond

To do single block conditional inference, run

python conditioned_prediction.py --layout samplelayouts/exp1_32-56-24/0_0.npy --save output/cond

We provide some sample layouts for demo. If you would like to draw your own layout maps, you can use drawtkinter.py to create your own layout. Here is a tutorial:

visualization

The unit of the graduations are in meters, furnitures need to be in reasonable scale. Walls should surround the floor and furniture should be placed on the floor.

The output layouts directory named after expname_xscale-zscale-stride, xscale-zscale-stride are given in decimeter.

Usage:

python drawtkinter.py --h 56 --w 56 --expname draw

Full pipeline(conditional prediction + extrapolation + resample) is contained in fullpipeline.py. Before running, specifying the layout directory.

python fullpipeline.py --layout samplelayouts/exp2_32-56-24 --resample 15 --save output

Citation

If you find our code or paper helps, please consider citing:

@article{blockfusion,
  title={BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation},
  author={Wu, Zhennan and Li, Yang and Yan, Han and Shang, Taizhang and Sun, Weixuan and Wang, Senbo and Cui, Ruikai and Liu, Weizhe and Sato, Hiroyuki and Li, Hongdong and Ji, Pan},
  journal={ACM Transactions on Graphics},
  volume={43},
  number={4},
  year={2024},
  doi={10.1145/3658188}
 }      

blockfusion's People

Contributors

rabbityl avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

Forkers

bruinxiong q5228

blockfusion's Issues

about the grad of sdf model

hi,
thanks for your great work.
Is it convenient for you to share the code for calculating the SDF gradient? I have implemented the Eq.6 for calculating gradients in your article, but the results are always incorrect

Triplane fitting MLP pretrained weight

Hello,
Thanks for the impressive work!

Iโ€™m trying to test the triplane fitting code located in the fit_triplane directory. However, it seems that the provided pretrained weights do not include the mlp.tar file. Could you provide an alternative link for the pretrained MLP weights(mlp.tar)?

Any practical installation guide?

Would you mind offer a enviroment version list you are using?
I've tried py39, pytorch 2.3.1 + cu118, build pytorch3d from source code, and build the custom diffusers according to your setup.py script.
And during executing the command: python unconditioned_prediction.py --batch 4 --save output/uncond ,got a lot of errors about missing modules like open3d, einops, xformers, etc... after I pip install them each by each. Still got this error:

undefined symbol: _ZN3c1015SmallVectorBaseIjE8grow_podEPvmm

It could be painful to guess the correct runtime enviroment...

FYI, I thinks its a matter of xformers version

link for pretrained weights

Thanks for the awesome work! The link for pretrained weights seem to be wrong, is there an alternative link?

issues with linear attention mode when running the main

Get errors below when running the main function:

constructing SpatialTransformer_ of depth 1 w/ 512 channels and 16 heads
Attention mode 'linear' is not available. Falling back to native attention. This is not a problem in Pytorch >= 2.0. FYI, you are running with PyTorch version 2.4.0

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.