Training not working. about datasetdm

[weijiawu]

Thank you for your attention. However, I can't recall the specific version of SD. To facilitate your usage, I have uploaded the corresponding SD weights used in my experiments to Google Drive (around 4.5GB) as following:

https://drive.google.com/file/d/12lrOexljsyvFB30-ltbYXnIpQ8oP4lrW/view?usp=sharing

I hope this helps you.

[fabvio]

Hi, you need to publicly share the file otherwise everyone will need to ask you for authorization. Thanks!

[fabvio]

It still does not train. It loads correctly the Stable Diffusion with the files you provided. Now I get this error when I try to train on Cityscapes data.

***********************   begin   **********************************
{'variance_type'} was not found in config. Values will be initialized to default values.
learning_rate: 0.0001
Epoch 0/5000
(512, 512)
  0%|          | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "tools/train_semantic_cityscapes.py", line 472, in <module>
    main()
  File "tools/train_semantic_cityscapes.py", line 396, in main
    images_here, x_t = ptp_utils.text2image(unet,vae,tokenizer,text_encoder,noise_scheduler, prompts, controller, latent=start_code, num_inference_steps=NUM_DIFFUSION_STEPS, guidance_scale=5, generator=g_cpu, low_resource=LOW_RESOURCE, Train=True)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/ptp_utils.py", line 214, in text2image
    image = latent2image(vae, latents)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/ptp_utils.py", line 90, in latent2image
    image = vae.decode(latents)['sample']
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/vae.py", line 552, in decode
    dec = self.decoder(z)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/vae.py", line 202, in forward
    sample = up_block(sample)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/unet_blocks.py", line 1212, in forward
    hidden_states = resnet(hidden_states, temb=None)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/resnet.py", line 336, in forward
    hidden_states = self.norm1(hidden_states.float()).type(hidden_states.dtype)
AttributeError: 'tuple' object has no attribute 'float'

Any idea of the cause? Thanks in advance.

[weimzh]

It still does not train. It loads correctly the Stable Diffusion with the files you provided. Now I get this error when I try to train on Cityscapes data.

***********************   begin   **********************************
{'variance_type'} was not found in config. Values will be initialized to default values.
learning_rate: 0.0001
Epoch 0/5000
(512, 512)
  0%|          | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "tools/train_semantic_cityscapes.py", line 472, in <module>
    main()
  File "tools/train_semantic_cityscapes.py", line 396, in main
    images_here, x_t = ptp_utils.text2image(unet,vae,tokenizer,text_encoder,noise_scheduler, prompts, controller, latent=start_code, num_inference_steps=NUM_DIFFUSION_STEPS, guidance_scale=5, generator=g_cpu, low_resource=LOW_RESOURCE, Train=True)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/ptp_utils.py", line 214, in text2image
    image = latent2image(vae, latents)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/ptp_utils.py", line 90, in latent2image
    image = vae.decode(latents)['sample']
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/vae.py", line 552, in decode
    dec = self.decoder(z)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/vae.py", line 202, in forward
    sample = up_block(sample)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/unet_blocks.py", line 1212, in forward
    hidden_states = resnet(hidden_states, temb=None)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/resnet.py", line 336, in forward
    hidden_states = self.norm1(hidden_states.float()).type(hidden_states.dtype)
AttributeError: 'tuple' object has no attribute 'float'

Any idea of the cause? Thanks in advance.

have you solved this issue? I've got the same problem here.

thanks