tangjiapeng / diffuscene Goto Github PK

Hi, I find it is a combination, not the concatenation illustrated on paper, right?

if self.seperate_all:
x_class = self.class_embedf(x[:, self.bbox_dim:self.bbox_dim+self.class_dim, :])
if self.objectness_dim >0:
x_object = self.objectness_embedf(x[:, self.bbox_dim+self.class_dim:self.bbox_dim+self.class_dim+self.objectness_dim, :])
else:
x_object = 0

        if self.objfeat_dim > 0:
            x_objfeat = self.objfeat_embedf(x[:, self.bbox_dim+self.class_dim+self.objectness_dim:self.bbox_dim+self.class_dim+self.objectness_dim+self.objfeat_dim, :])
        else:
            x_objfeat = 0
            
        x_bbox = self.bbox_embedf(x[:, 0:self.bbox_dim, :])
        x = x_class + x_bbox + x_object + x_objfeat

I want to run preprocessing and training without using the given pretrained model. However, when I execute generate.sh, it causes a FileNotFoundError for weight files(livingrooms_uncond.pt, bedrooms_uncond.pt) like the below figure. I checked the files related to training process, and it seems there's no code that generates the 'uncond.pt' files. The checkpoints are saved as files named model_00000. Are these files equivalent to the '_uncond.pt' files?

Hi!
I ran into some issues while trying to setup the environment as per the instructions provided on the repo. Initially, the build for the conda environment failed with the following error:

Could not solve for environment specs
The following package could not be installed
└─ trimesh 3.12.7**  does not exist (perhaps a typo or a missing channel).

and after removing trimesh 3.12.7 from the .yaml file,

Pip subprocess error:
ERROR: Could not find a version that satisfies the requirement plyfile==1.21.3 (from versions: 0.4, 0.5, 0.6, 0.7, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.8, 0.8.1, 0.9)
ERROR: No matching distribution found for plyfile==1.21.3

failed

CondaEnvException: Pip failed

I found that trimesh is pip installable, and the latest stable release of plyfile is automatically installed as a dependency for one of the other packages. Hence, the following environment.yaml file fixes the build:

name: diffuscene
channels:
  - conda-forge
  - pytorch
  - defaults
dependencies:
  - cython=0.29.32
  - numpy=1.21.3
  - networkx=2.5.1
  - pyrr=0.10.3
  - pyyaml=5.3.1
  - python=3.8
  - cudatoolkit=11.3
  - pytorch=1.13.0
  - torchvision=0.14.0
  - pillow=7.2.0
  - scipy=1.8.1
  - tqdm=4.64.1
  - matplotlib=3.3.1
  - wxpython=4.0.7
  - pip:
    - simple_3dviz==0.7.0
    - pytorch-fast-transformers==0.4.0
    - wandb==0.13.2
    - open3d==0.15.2
    - seaborn==0.12.2
    - trimesh==3.12.7
    - num2words

But then the installation for ChamferDistancePytorch fails with the following traceback:

Traceback (most recent call last):
  File "setup.py", line 7, in <module>
    CUDAExtension('chamfer_3D', [
  File "/user/dv/miniconda3/envs/diffuscene/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1031, in CUDAExtension
    library_dirs += library_paths(cuda=True)
  File "/user/dv/miniconda3/envs/diffuscene/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1163, in library_paths
    if (not os.path.exists(_join_cuda_home(lib_dir)) and
  File "/user/dv/miniconda3/envs/diffuscene/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 2214, in _join_cuda_home
    raise EnvironmentError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.

Can you please help me figure out what's gone wrong?

hi, thanks for your excellent work! Is there a trained model available ?

Currently it only supports single scene genertion. Is it possible to generate scenes in batch?

I don't know how to write the path in scripts/pickle_threed_fucture_dataset.py:"path_to_3d_front_dataset_directory"、"path_to_3d_future_dataset_directory"、"path_to_model_info"
i was already download the 3d_front dataset

i was already get raw_model_norm_pc_lat32.npz,but i don't know how to get 'raw_model_norm_pc_lat.npz',we don't have 'bed_living_diningrooms_lat.yaml'.
I guess the 'bed_living_diningrooms.yaml' is the 'bed_living_diningrooms_lat.yaml' , just the name was mistake.
please help me

Hi,

I checked the code and could not find the places to define the arrange_scene and complete_scene functions. Will you plan to release them recently?

Hi,

Currently, the generated models cannot be retrieved by object features since the raw_model_norm_pc_lat32 will be read inside each 3DFuture model folder. Can you share your pretrained foldingnet weight so that I can generate these features? Thanks!

Best,
Kaizhi

When I try generate_diffusion.py, it shows

AttributeError: 'ThreedFutureDataset' object has no attribute 'get_closest_furniture_to_objfeats'

How to solve this problem?

The 19th line of train_objautoencoder.py
original:
from utils import yield_forever, load_checkpoints, save_checkpoints
should be:
from training_utils import yield_forever, load_checkpoints, save_checkpoints

I want to train Diffuscene by adding a new scene to 3D-front. However, while performing preprocessing based on the readme you provided, it seems that there is no code to generate threed_front.pkl. Where can I find it?

When I try generate_diffusion.py, it shows No module named 'scene_synthesis' when the objects_dataset is loading

    # Build the dataset of 3D models
    objects_dataset = ThreedFutureDataset.from_pickled_dataset(
        args.path_to_pickled_3d_futute_models
    )
    print("Loaded {} 3D-FUTURE models".format(len(objects_dataset)))

How to solve this problem? Where can I find scene_synthesis module?

Where can i find it,please?
I can only find dataset_directory: "/cluster/balrog/jtang/3d_front_processed/bedrooms_objfeats_32_64",not bedrooms!

Thanks for your work, has the code for the Text-conditioned section been released?

Hi there, great work!
What's the exact settings for evaluation? It's 1000 scenes, no floor, no texture?
And, say that it is <no floor, no texture> in evalution, so I should add <--without_floor --no_texture> when running "preprocess_data.py" to get GT images, right?

(diffuscene5) root@rtx6:/workspace/caitou/DiffuScene1/scripts# python preprocess_data.py /workspace/caitou/DiffuScene1/3d_front_processed/livingrooms_objfeats_32_64 /workspace/caitou/DiffuScene1/3D-FRONT/3D-FRONT /workspace/caitou/DiffuScene1/3D-FUTURE-model/3D-FUTURE-model /workspace/caitou/DiffuScene1/3D-FUTURE-model/3D-FUTURE-model/model_info.json --dataset_filtering threed_front_livingroom --annotation_file /workspace/caitou/DiffuScene1/config/livingroom_threed_front_splits.csv --add_objfeats
libGL error: MESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
libGL error: failed to load driver: swrast
Traceback (most recent call last):
File "preprocess_data.py", line 355, in
main(sys.argv[1:])
File "preprocess_data.py", line 147, in main
scene = scene_from_args(args)
File "/workspace/caitou/DiffuScene1/scripts/utils.py", line 244, in scene_from_args
scene = Scene(size=args.window_size, background=args.background)
File "/workspace/conda/envs/diffuscene5/lib/python3.8/site-packages/simple_3dviz/scenes.py", line 112, in init
super(Scene, self).init(size, background, ctx)
File "/workspace/conda/envs/diffuscene5/lib/python3.8/site-packages/simple_3dviz/scenes.py", line 23, in init
self._ctx = moderngl.create_standalone_context()
File "/workspace/conda/envs/diffuscene5/lib/python3.8/site-packages/moderngl/init.py", line 1960, in create_standalone_context
return create_context(standalone=True, **kwargs)
File "/workspace/conda/envs/diffuscene5/lib/python3.8/site-packages/moderngl/init.py", line 1931, in create_context
ctx.mglo, ctx.version_code = mgl.create_context(glversion=require, mode=mode, **settings)
File "/workspace/conda/envs/diffuscene5/lib/python3.8/site-packages/glcontext/init.py", line 90, in create
return x11.create_context(**kwargs)
Exception: (standalone) cannot create context

Hello , First of all you did great job.
My task was to test your model and see the three cases -
Completition of partial scnes.
Scene Generation.
Text to Scene.
But i have downloaded the pretrained diffuscene and pretrained objautoencoder.
and have also downloaded the pretrained dataset from the link.
But when i tried the two commands generate.sh and the other one it gave me error that such file doesnt exist.
Can you like guide me and give me the proper steps to do so?
I have edited the paths to the model and dataset in the generate.sh and test files.
Thank you

where is the "help menu"

how can i see the scene generated?

When I try to train the text-conditioned model, it show that the data has no 'description' attribute:

File "/ mnt/16T/lhj/files/remote/for_scene_gen/DiffuScene/scene_diffusion/networks/diffusion_scene_layout_ddpm.py", line 226, in get_loss.
Tokenized = self.tokenizer (sample_params ["description"], return_tensors='pt',padding=True) .to (device).
KeyError: 'description'

How to solve this problem?

I use the pretrained model to generate 1000 images (256 x 256) like this:

I also make sure the 4041 GT images in 3D-FRONT are like this either.
When I use the 4041 GT images and 1000 predicted images to evaluate the pretrained model, I get fid=26.23, kid=0.0046, SCA=0.66(improved precision?).
According to the paper, it should be fid=17.21, kid=0.0007 (0.70 x 0.001), SCA=0.5215, which is much better than my results, right?

Please let me know if there is anything wrong.

Plus, is there any chance that the CKL code will be released?

Thanks again for your great work!

When running "generate.sh", it reports the error
"The 'list' object does not have the property 'get_closest_furniture_to_objfeats'."
How to solve?

I want to create a scene that takes only text as input. However, when I looked at the generate_diffusion.py, there seems to be no codes that takes text as input. There are only a code that saves the text as a txt file. Please let me know. Is it impossible to create a scene by taking text as input?

To generate text prompt based scenes i have to run generate_text.sh file?
but it doesnt include any text prompt function.
and it used generate_diffusion.py which also doesnt have any text prompt function ?
so to use it do i have to make changes in the code lines?
like for example -

parser.add_argument(
    "--text_prompt",
    type=str,
    nargs='+',
    help="Custom text prompts for scene generation"
)
??