carperai / magicarp-v2 Goto Github PK

Could be a bug with timestep embeddings for the MidU reward model

• Currently it is taking only denoised images, in the future we want to try noisy images so that it can give reward for in-progress generations (steering in different stages could help in different ways)
• Have to walk before we can run so for now just working with purely denoised images
• Time steps need to reflect that the images are entirely denoised
• Currently not sure if timestep input is signifying pure signal or pure noise (using a value of 0)
• Need to get a better dataset to test on
• To validate the other code is working, will need to get good results on said dataset with CLIP RM model first

Need a diffusion model RM that takes variance schedule as input. Taking pretrained cross-attention DM and finetuning with a reward head at the UNET bottleneck (before upsampling blocks) should suffice.

BCE with logits not ideal in this instance

• Really messy to load and iterate through a validation dataloader
• Need to incorporate accuracy into validation in both trainers as well

• Accelerate support for Image/Text encoder
• Accelerate support for Diffusion encoder

• freezing bottom causal layers
• clipping gradients by norms

Diffusers reportedly has subpar performance compared to benchmarks. Preferable to use different DM model

• Should work with accelerate but currently causes bugs, need to fix