salesforce / hive Goto Github PK

Hello, I'm trying to run the code from your repository, and I encountered an error stating
'No such file or directory: 'data/training/instructpix2pix/seeds.json'.' It seems that the required file 'seeds.json' is missing from the specified directory.

Additionally, I tried executing the 'bash scripts/download_hive_data.sh' command to download the train dataset, but it doesn't seem to be working as expected.
Could you please provide some guidance on how to resolve these issues? Thank you!

Hey, great job on this project! Really exciting work.

I want to reproduce the first part of your training process (fine-tuning IP2P) on SD2.1 768x768 but having some trouble.

I've tried to piece together the original IP2P training config and your generate config for SD2.1 but the layer params don't match up between my config and my checkpoint. Do you have any idea how to make the checkpoint and model match?

My model

https://huggingface.co/stabilityai/stable-diffusion-2-1/blob/main/v2-1_768-ema-pruned.ckpt

My config

# train.yaml
model:
  base_learning_rate: 1.0e-04
  target: ldm.models.diffusion.ddpm_edit_v21.LatentDiffusion
  params:
    ckpt_path: checkpoints/v2-1_768-ema-pruned.ckpt
    # ...

The errors

I get a lot of errors like these when starting training:

size mismatch for model.diffusion_model.input_blocks.2.1.x.y.z: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current 
model is torch.Size([320, 768]).

The mismatches are between a shape element being 1024 in checkpoint vs. 768 in the model definition, or something like torch.Size([640, 640]) vs. torch.Size([640, 640, 1, 1])

Exact errors

Hi, thanks for you great work. But it seems the code and training scripts for the reward model have not been released yet. May I ask when will you release these parts of code?

Is it possible to upload the checkpoints & datasets to huggingface ?

hi, thanks for your great work.
Is there any planning to release the reward model checkpoint (blip model)?

Hi Authors, recently I am doing a comparison between HIVE and InstructPix2Pix. To my surprise HIVE is not working that good as compared to InstructPix2Pix. It is even quite off compared to it. I am really wondering if this is really the case or am I missing something. I have tested both condition and weighted based models with SD 2.1 as backbone. Both are not good and weighted based is quite even worse than condition based. And, in the paper figure 20 it shows that they are not supposed to be too different. I am testing with simple instructions even. Can you please help me here what is going on? Thanks a lot!

Hi, thanks for your great work and releasing your code and data publicly! I was wondering if you are planning to release the human annotations you collected for training your reward model? Thanks!

Dear HIVE team,

I really enjoy your work. Thanks for sharing this great work.

Is there another way to download your dataset? It is not easy for us to use google cloud.

Thank you for your help.

Best Wishes,
Zongze

Hey, for me the conda environment in this repo doesn't resolve upon adding xformers via conda.
Adding xformers via pip seems to break something about PyTorch/CUDA compat which leads to some tensors being on the wrong device at runtime (does not happen without xformers installation).

I used the installation procedures from the xformers Readme.

I assume you used xformers for training so let me know if you can share a working setup.