thudm / imagereward

[NeurIPS 2023] ImageReward: Learning and Evaluating Human Preferences for Text-to-image Generation

ImageReward

📃 Paper • 🖼 Dataset • 🌐 中文博客 • 🤗 HF Repo • 🐦 Twitter

ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation

🔥News! The paper of ImageReward is accepted by NeurIPS 2023!

ImageReward is the first general-purpose text-to-image human preference RM, which is trained on in total 137k pairs of expert comparisons, outperforming existing text-image scoring methods, such as CLIP (by 38.6%), Aesthetic (by 39.6%), and BLIP (by 31.6%), in terms of understanding human preference in text-to-image synthesis.

Additionally, we introduce Reward Feedback Learning (ReFL) for direct optimizing a text-to-image diffusion model using ImageReward. ReFL-tuned Stable Diffusion wins against untuned version by 58.4% in human evaluation.

Both ImageReward and ReFL are all packed up to Python image-reward package now!

Try image-reward package in only 3 lines of code for ImageReward scoring!

# pip install image-reward
import ImageReward as RM
model = RM.load("ImageReward-v1.0")

rewards = model.score("<prompt>", ["<img1_obj_or_path>", "<img2_obj_or_path>", ...])

Try image-reward package in only 4 lines of code for ReFL fine-tuning!

# pip install image-reward
# pip install diffusers==0.16.0 accelerate==0.16.0 datasets==2.11.0
from ImageReward import ReFL
args = ReFL.parse_args()
trainer = ReFL.Trainer("CompVis/stable-diffusion-v1-4", "data/refl_data.json", args=args)
trainer.train(args=args)

If you find ImageReward's open-source effort useful, please 🌟 us to encourage our following developement!

• ImageReward
  • Quick Start
    • Install Dependency
    • Example Use
  • ReFL
    • Install Dependency
    • Example Use
  • Demos of ImageReward and ReFL
  • Training code for ImageReward
  • Integration into Stable Diffusion Web UI
    • Features
      • Score generated images and append to image information
        • Usage
        • Demo video
      • Automatically filter out images with low scores
        • Usage
        • Demo video
      • View the scores of images that have been scored
        • Usage
        • Example
      • Other Features
        • Memory Management
    • FAQ
  • Reproduce Experiments in Table 1
  • Reproduce Experiments in Table 3
  • Citation

Quick Start

Install Dependency

We have integrated the whole repository to a single python package image-reward. Following the commands below to prepare the environment:

# Clone the ImageReward repository (containing data for testing)
git clone https://github.com/THUDM/ImageReward.git
cd ImageReward

# Install the integrated package `image-reward`
pip install image-reward

Example Use

We provide example images in the assets/images directory of this repo. The example prompt is:

a painting of an ocean with clouds and birds, day time, low depth field effect

Use the following code to get the human preference scores from ImageReward:

import os
import torch
import ImageReward as RM

if __name__ == "__main__":
    prompt = "a painting of an ocean with clouds and birds, day time, low depth field effect"
    img_prefix = "assets/images"
    generations = [f"{pic_id}.webp" for pic_id in range(1, 5)]
    img_list = [os.path.join(img_prefix, img) for img in generations]
    model = RM.load("ImageReward-v1.0")
    with torch.no_grad():
        ranking, rewards = model.inference_rank(prompt, img_list)
        # Print the result
        print("\nPreference predictions:\n")
        print(f"ranking = {ranking}")
        print(f"rewards = {rewards}")
        for index in range(len(img_list)):
            score = model.score(prompt, img_list[index])
            print(f"{generations[index]:>16s}: {score:.2f}")

The output should be like as follow (the exact numbers may be slightly different depending on the compute device):

Preference predictions:

ranking = [1, 2, 3, 4]
rewards = [[0.5811622738838196], [0.2745276093482971], [-1.4131819009780884], [-2.029569625854492]]
          1.webp: 0.58
          2.webp: 0.27
          3.webp: -1.41
          4.webp: -2.03

ReFL

Install Dependency

pip install diffusers==0.16.0 accelerate==0.16.0 datasets==2.11.0

Example Use

We provide example dataset for ReFL in the data/refl_data.json of this repo. Run ReFL as following:

bash scripts/train_refl.sh

Demos of ImageReward and ReFL

Training code for ImageReward

1. Download data: 🖼 Dataset.

2. Make dataset.

cd train
python src/make_dataset.py

3. Set training config: train/src/config/config.yaml

4. One command to train.

bash scripts/train_one_node.sh

Integration into Stable Diffusion Web UI

We have developed a custom script to integrate ImageReward into SD Web UI for a convenient experience.

The script is located at sdwebui/image_reward.py in this repository.

The usage of the script is described as follows:

1. Install: put the custom script into the stable-diffusion-webui/scripts/ directory
2. Reload: restart the service, or click the "Reload custom script" button at the bottom of the settings tab of SD Web UI. (If the button can't be found, try clicking the "Show all pages" button at the bottom of the left sidebar.)
3. Select: go back to the "txt2img"/"img2img" tab, and select "ImageReward - generate human preference scores" from the "Script" dropdown menu in the lower left corner.
4. Run: the specific usage varies depending on the functional requirements, as described in the "Features" section below.

Features

Score generated images and append to image information

Usage

1. Do not check the "Filter out images with low scores" checkbox.
2. Click the "Generate" button to generate images.
3. Check the ImageReward at the bottom of the image information below the gallery.

Demo video

score-and-append-to-info.mp4

Automatically filter out images with low scores

Usage

1. Check the "Filter out images with low scores" checkbox.
2. Enter the score lower limit in "Lower score limit". (ImageReward roughly follows the standard normal distribution, with a mean of 0 and a variance of 1.)
3. Click the "Generate" button to generate images.
4. Images with scores below the lower limit will be automatically filtered out and will not appear in the gallery.
5. Check the ImageReward at the bottom of the image information below the gallery.

Demo video

filter-out-images-with-low-scores.mp4

View the scores of images that have been scored

Usage

1. Upload the scored image file in the "PNG Info" tab
2. Check the image information on the right with the score of the image at the bottom.

Example

Other Features

Memory Management

• ImageReward model will not be loaded until first script run.
• "Reload UI" will not reload the model nor unload it, but reuses the currently loaded model (if it exists).
• A "Unload Model" button is provided to manually unload the currently loaded model.

FAQ

How to adjust the Python environment used by the SD Web UI (e.g. reinstall a package)?

Note that SD Web UI has two ways to set up its Python environment:

• If you launch with python launch.py, Web UI will use the Python environment found in your PATH (in Linux, you can check its exact path with which python).
• If you launch with a script like webui-user.bat, Web UI creates a new venv environment in the directory stable-diffusion-webui\venv.
  • Generally, you need some other operations to activate this environment. For example, in Windows, you need to enter the stable-diffusion-webui\venv\Scripts directory, run activate or activate.bat (if you are using cmd) or activate.ps1 (if you are using PowerShell) from .
  • If you see the prompt (venv) appear at the far left of the command line, you have successfully activated venv created by the SD Web UI.

After activating the right Python environment, just do what you want to do true to form.

Reproduce Experiments in Table 1

Note: The experimental results are produced in an environment that satisfies:

• (NVIDIA) Driver Version: 515.86.01
• CUDA Version: 11.7
• torch Version: 1.12.1+cu113 According to our own reproduction experience, reproducing this experiment in other environments may cause the last decimal place to fluctuate, typically within a range of ±0.1.

Run the following script to automatically download data, baseline models, and run experiments:

bash ./scripts/test-benchmark.sh

Then you can check the results in benchmark/results/ or the terminal.

If you want to check the raw data files individually:

• Test prompts and corresponding human rankings for images are located in benchmark/benchmark-prompts.json.
• Generated outputs for each prompt (originally from DiffusionDB) can be downloaded from Hugging Face or Tsinghua Cloud.
  • Each <model_name>.zip contains a directory of the same name, in which there are in total 1000 images generated from 100 prompts of 10 images each.
  • Every <model_name>.zip should be decompressed into benchmark/generations/ as directory <model_name> that contains images.

Reproduce Experiments in Table 3

Run the following script to automatically download data, baseline models, and run experiments:

bash ./scripts/test.sh

If you want to check the raw data files individually:

• Test prompts and corresponding human rankings for images are located in data/test.json.
• Generated outputs for each prompt (originally from DiffusionDB) can be downloaded from Hugging Face or Tsinghua Cloud. It should be decompressed to data/test_images.

Citation

@misc{xu2023imagereward,
      title={ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation},
      author={Jiazheng Xu and Xiao Liu and Yuchen Wu and Yuxuan Tong and Qinkai Li and Ming Ding and Jie Tang and Yuxiao Dong},
      year={2023},
      eprint={2304.05977},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

imagereward's Issues

Why the mean and std in Reward Model is set as 0.16717362830052426 and 1.0333394966054072 ?

Hi, @tongyx361

Thanks for your contribution, I want to figure out why the mean and std in the Reward Model are set as the following values:

ImageReward/ImageReward/ImageReward.py

Line 80 in c0b9080

self.mean = 0.16717362830052426

In addition, there are negative reward values during inference, which confuses me that what's the range of rewards during training?

Could you comment on what the "researcher" in Table 1 refers to?

Hi,

Thanks for the excellent job! After reading the paper, I am not sure what the "researcher" in Table 1 refers to. Is that the label annotated by professional subjects?

Best!

关于纯图像的得分评估

您好，感谢您开源如此重要的AIGC图文得分偏好模型和方法。
我看您readme文件中的如下命令：
rewards = model.score("", ["<img1_obj_or_path>", "<img2_obj_or_path>", ...])
是在评估同一个prompt生成的不同图像的得分偏好。如果我只有一堆图文对，即 [[prompt1, image1], [prompt2, image2], ...[promptN, imageN]]，他们之间是一一对应，而不是一对多的关系，请问，这种情况下我应该如何使用您的模型对这些图文对的美学、人类偏好进行得分排序？
期待您的回复

Only supports square images?

may be only supports square images? because image size only one dim

about mean and std

Thanks for your wonderful work, I have a question that in ImageReward.py l80-81 set the self.mean=0.167 and self.std=1.033,
How did you get these two numbers？

EMA model

Dear authors, thanks for your project again.
I am new to stable diffusion. I don't really get the context and idea of EMA model (or use_ema). Can you please tell me what is the purpose of this practice? Thanks in advance.

text tokenizer max_length=35?

I found that in inference_rank(), you set the max_length for text tokenizer to 35, which means that all the text input longer than 35 will be truncated to 35 characters only? I was wondering if it's the same setting as during training, and what's the intuition behind that. Since 35 characters are actually pretty short and lots of user prompts are longer than that and the truncation may cause information loss so that the truncated text may not contain all the important elements

downsampling input images to 224x?

wondering if there's any motivation for downsampling the input images to 224x (other than memory efficiency?). Since most of the generated images are at least 512x, resizing them to 224 might lead to information loss e.g. some disturbing face details might not be noticeable after downsampling? Not sure if you think that's a valid concern here

score without prompt

Hi, thanks for the great work!

Just wondering is ImageReward able to score a single image without prompt? For example, something like the aesthetic predictor.

Thanks!

about train

Hi, I ran your code: train/src/train.py and found that reading every piece of data from train.json, found no "generations","ranking" fields

Error loading script in SD Webui 5ab7f213

Running the most recent version of SD Webui. pip install image-reward ran successfully. Dropped image_reward.py into C:\Users\user\stable-diffusion-webui\scripts. SD Webui is printing the following error:

Error loading script: image_reward.py
Traceback (most recent call last):
  File "C:\Users\user\stable-diffusion-webui\modules\scripts.py", line 256, in load_scripts
    script_module = script_loading.load_module(scriptfile.path)
  File "C:\Users\user\stable-diffusion-webui\modules\script_loading.py", line 11, in load_module
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\user\stable-diffusion-webui\scripts\image_reward.py", line 18, in <module>
    import ImageReward as reward
ModuleNotFoundError: No module named 'ImageReward'

commandline args: --xformers --autolaunch --deepdanbooru
python: 3.10.6
commit: 5ab7f213

How long to fine-tune diffusion model?

How to use HuggingFace Data?

Hi, @xujz18 @Xiao9905

Thanks for this nice contribution. I noticed that we can load ImageReward data with:
datasets.load_dataset("THUDM/ImageRewardDB", "8k")

However, the loaded data seem to not match with existing code, I have no idea how to move on with these code (I downloaded these HuggingFace data and save to disk, so I use load_from_disk to load them):

train_dataset = load_from_disk("data/RLHF/ImageRewardDB_8k/train")
valid_dataset = load_from_disk("data/RLHF/ImageRewardDB_8k/validation")
test_dataset  = load_from_disk("data/RLHF/ImageRewardDB_8k/test")

When I print train_dataset[0].keys(), it shows the same results in HuggingFace Dataset introduction:

dict_keys(['image', 'prompt_id', 'prompt', 'classification', 'image_amount_in_total', 'rank', 'overall_rating', 'image_text_alignment_rating', 'fidelity_rating'])

When I run python src/make_dataset.py, following the instruction in the README, this error happens:

making dataset:   0%|                                                         | 0/10000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/tiger/code/ImageReward/train/src/make_dataset.py", line 12, in <module>
    train_dataset = rank_pair_dataset("train")
  File "/home/tiger/code/ImageReward/train/src/rank_pair_dataset.py", line 59, in __init__
    self.data = self.make_data()
  File "/home/tiger/code/ImageReward/train/src/rank_pair_dataset.py", line 80, in make_data
    for generations in item["generations"]:
KeyError: 'generations'

Unfortunately, it is not compatible with the existing datasets code:

ImageReward/train/src/rank_pair_dataset.py

Line 47 in 1beb4e4

class rank_pair_dataset(Dataset):

Does this mean we have to re-write the code if we want to use the downloaded dataset from HuggingFace?

ReFL implement details

As mentioned in #24 and #34, the current ReFL code only the ReFL loss is implemented and the pre-training loss is not included. In addition, the two losses are optimized alternately.

I want to add pre-training data myself. If we don't use the gradient accumulation, the pseudo code would be like this:

# Given optimizer and lr_scheduler with unet.
# Compute Pre-training Loss `train_loss` with unet and update unet.
train_loss.backward()
optimizer.step()
lr_scheduler.step()  # is it necessary?
optimizer.zero_grad()

# Compute ReFL Loss `refl_loss` with unet and update unet.
refl_loss.backward()
optimizer.step()
lr_scheduler.step()
optimizer.zero_grad()

However, I'm confused about how to add accelerator.accumulate(unet) for gradient accumulation after reading this post. And I also raised the issue huggingface/accelerate#1870 and discussion in the huggingface accelerate github repo and forum. But I don't seem to get a clear answer. Can you give me some pseudo codes or hints? Thank you very much! @xujz18 @tongyx361

About embedding selection of reward model

Thank you for the great work! I just have a small question regarding the training and inference stage of the reward model. In the encode_pair function in train/src/ImageReward.py you are using last_hidden_state[:, -1, :] of the embedding, but in encode_data function and in the inference stage in ImageReward/ImageReward.py you are using last_hidden_state[:,0,:]. Why are they different? What is the intention of using the first or the last dimension? Thanks a lot!

how is the score related with alignment?

From the paper, this metric seems a general purpose score measuring human preference. If I am only interested in image text alignment, it this metric suitable?

where use the relu fuction?

The paper algorithm says use relu for reward model to predict score. But there is no relu in your code. Anything I miss?
If do not use non-linear activation fuction, the multiple nn.Linear are equal to a single nn.Linear. For example, equal to nn.Linear(1024,1) in your settings.

issue installing and running on sdwebui

apologies if not the place but am needing help have been trying to get this to work but im struggling to figure out what step im missing, have done everything as instructed but have come at this road block, when running the install is recognized as done improperly and attempts a new one everything is checked off and it says succesfully installed but then when it trys to load the image_reward.py script it gets an error with the traceback it shows lines followed by a ModuleNotFoundError: No module named 'datasets' will paste below as the trace back, if someone can recoginse the issue and has a solution wherther is something ive missed or something please let me know asap it all looks very intruiging

"image-reward" package is successfully installed!
Error loading script: image_reward.py
Traceback (most recent call last):
File "E:\Aishit\stable-diffusion\stable-diffusion-webui\scripts\image_reward.py", line 21, in
import ImageReward as reward
File "E:\Aishit\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\ImageReward_init_.py", line 3, in
from .ReFL import *
File "E:\Aishit\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\ImageReward\ReFL.py", line 27, in
from datasets import load_dataset
ModuleNotFoundError: No module named 'datasets'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "E:\Aishit\stable-diffusion\stable-diffusion-webui\modules\scripts.py", line 263, in load_scripts
script_module = script_loading.load_module(scriptfile.path)
File "E:\Aishit\stable-diffusion\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module
module_spec.loader.exec_module(module)
File "", line 883, in exec_module
File "", line 241, in call_with_frames_removed
File "E:\Aishit\stable-diffusion\stable-diffusion-webui\scripts\image_reward.py", line 26, in
import ImageReward as reward
File "E:\Aishit\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\ImageReward_init.py", line 3, in
from .ReFL import *
File "E:\Aishit\stable-diffusion\stable-diffusion-webui\venv\lib\site-packages\ImageReward\ReFL.py", line 27, in
from datasets import load_dataset
ModuleNotFoundError: No module named 'datasets'

Image Browser: ImageReward is not installed, cannot be used.

Any plans to release dataset?

Just following up on a previous inquiry about full data release plan. The annotation dataset would be a great contribution to the research community; thanks for your work on this!

Update requirements.txt

I use Automatic1111 and the requirements are constantly updating. Image Browser uses outdated requirements of following packages:

1. Accelerate
2. diffusers
3. huggingface-hub
4. transformers

Another issue opened in A1111 repo

PIP Error:
Installing collected packages: transformers ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. image-reward 1.3 requires accelerate==0.16.0, but you have accelerate 0.19.0 which is incompatible. image-reward 1.3 requires diffusers==0.16.0, but you have diffusers 0.16.1 which is incompatible. image-reward 1.3 requires huggingface-hub==0.13.4, but you have huggingface-hub 0.15.1 which is incompatible. image-reward 1.3 requires transformers==4.27.4, but you have transformers 4.30.2 which is incompatible. Successfully installed transformers-4.30.2

I suggest to bump the requirements.txt to current and future releases of these packages.

ReFL tuned SD realease

Thank you for your amazing work.

Do you plan to release ReFL tuned SD model?

Thanks

It seems that current code only realize the reward loss, how about the pretrain loss mentioned in the paper? Do the reward loss and pretrain loss are trained sequentially?

关于score结果的疑问

请问最终得分有什么具体的参考意义吗？还是说只是用来排序的？

Finetune ImageReward?

Hi, I want to use my personal preference ranking data to fine-tune ImageReward. But I have encountered several problems:

1. Why use BertTokenizer rather than blip.text_encoder when training reward models?
2. Given preference data (including train, validation and test) in test.json format, can I use encode_data in forward to compute the ranking loss rather than make_dataset first?

Thank you in advance!

Code to train with custom data or version data you give . Any code examples

I can't understand the meaning of the code, especially why the red variable is set like this?

In code: ImageReward/ReFL.py lines 730~737

not connection

Run the code below:
import ImageReward as RM
model = RM.load('ImageReward-v1.0')

but it will emerge some mistakes below:
LocalEntryNotFoundError: Connection error, and we cannot find the requested files in the disk cache. Please try again or make sure your Internet connection is on.

And then I download the " ImageReward.pt" on my own device, but it will emerge teh same mistake as before, what happend.?

Dataset download issue

Hi,

I am trying to reproduce the 1k and 4k numbers for the ImageReward function accuracy, as mentioned in the paper. To do so, I downloaded the data, and modified it slightly so that it could be loaded using the script make_dataset.py. However, there are some file IDs in the training set, that have null images, that is 0K file size.

Following is the list of the IDs.

005050-0024
005389-0008
005795-0038
006272-0041
006756-0071
005165-0028
005332-0172
005356-0019
006011-0030
006167-0087
006758-0099
005179-0097
005444-0063
005434-0068
005459-0003
005344-0055
006174-0048
006190-0114
006214-0021
006787-0015
006857-0073
006830-0003

May I ask what is the scope of the reward, and is there any plan to update imagereward 2.0

Training ImageReward model on different budgets

Hi!
The paper mentions that the training for the ImageReward model is not easy and is sensitive to hyperparameters. In the section about hyperparameters, it says --" We find that fixing 70% of transformer layers with a learning rate of 1e-5 and batch size of 64 can reach up to the best preference accuracy."

Is this for the 8k budget? Can we get the suitable hyperparams for the other budgets?

Secondly, which part of the code freezes the transformer layers? Thanks!

Image Reward Fails to Load

ImageReward does not work. I am using stable-diffusion-webui-directml. I followed instructions from "Quick Start" as well as "Integration into Stable Diffusion Web UI". Attached screenshots to show that ImageReward is installed, and script is in then place instructions say.

doesnt work in SDwebui

Error completing request
Arguments: (0, '{nynaneki _ 1.5},The Petite gril with Ocean Camo Coat on the street,masterpiece_1.1, coloerful cloud,bloom,fantasy,sunlight,sun', 'sea', 'None', 'None', <PIL.Image.Image image mode=RGB size=1920x1440 at 0x1F3F4613100>, None, None, None, 0, 20, 0, 4, 1, False, False, 1, 1, 7, 0.75, -1.0, -1.0, 0, 0, 0, False, 512, 512, 0, False, 32, 0, '', '', 1, '', '', 1, 50, 0, False, 4, 1, '

Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8

', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, None, '', '

Will upscale the image to twice the dimensions; use width and height sliders to set tile size

', 64, 0, 1, '', 0, '', True, False) {}
Traceback (most recent call last):
File "D:\Ts\modules\ui.py", line 184, in f
res = list(func(*args, **kwargs))
File "D:\Ts\webui.py", line 64, in f
res = func(*args, **kwargs)
File "D:\Ts\modules\img2img.py", line 124, in img2img
processed = modules.scripts.scripts_img2img.run(p, *args)
File "D:\Ts\modules\scripts.py", line 162, in run
processed = script.run(p, *script_args)
TypeError: run() missing 2 required positional arguments: 'filter_out_low_scores' and 'lower_score_limit'

error in download data.

If I run load_dataset("THUDM/ImageRewardDB", "8k_pair"). I will get a error.

Traceback (most recent call last):
  File "/miniconda3/envs/torch1.13.0/lib/python3.8/site-packages/datasets/builder.py", line 1637, in _prepare_split_single
    num_examples, num_bytes = writer.finalize()
  File "/miniconda3/envs/torch1.13.0/lib/python3.8/site-packages/datasets/arrow_writer.py", line 579, in finalize
    self.check_duplicate_keys()
  File "/miniconda3/envs/torch1.13.0/lib/python3.8/site-packages/datasets/arrow_writer.py", line 501, in check_duplicate_keys
    raise DuplicatedKeysError(key, duplicate_key_indices)
datasets.keyhash.DuplicatedKeysError: Found multiple examples generated with the same key
The examples at index 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17 have the key 000904-0035

ValueError

Attached is the error I got, I suppose it has to do with the version of SD that I'm using which is automatic by vlad
Could this be fix here or should this be done on that SD's end?

Thanks

Change ImageReward model location on WebUI

I have Windows so when I run the script the first time, it downloaded the model in C:\Users\User\.cache\huggingface\hub, while WebUI is on another drive. Any way to change the location?

BLIP Tokenizer `max_length`

Many thanks for sharing the code.

I was wondering why you're calling the tokenizer with max_length=35? Is there anything preventing me from relaxing this limit?
(Also noticed that for CLIPScore you don't impose the same limitation)

Released date of next version

Hello authors,
It is really a wonderful work. When do you plan to release the next version (imagereward-2.0) of this project. Thanks a lot.

"ranking" in `test.json` does not align with the predicted reward produced by the model

Hello! This is a very interesting work, and potentially a very impactful one. I really appreciate you guys for sharing this model.

I just have a question related to the "ranking" data provided in the test.json. It seems that many rankings did not break tie. Were these labeled by human annotators? Because I notice that the rankings produced by the reward model will almost always breaks tie. It rarely is the case that two images will get the same scalar reward value.

For example, the image set 001135-0100 has the ranking [ 2, 3, 3, 2, 4, 4, 1, 1 ] in test.json, but on the same images the reward model produced: [ 2, 6, 3, 5, 4, 7, 8, 1 ]. Just wondering why is this the case.

Thanks!

Full data release

Will the full dataset of human annotations (137k pairs of expert comparisons as mentioned in paper) be released?

ask for help

Due to network issues, I was unable to download the model in the code, so I place the ImageReward.pt and med_config.json file locally and execute:
Model=RM. load (name="/home/lw/ImageRward/ckpt/ImageRward. pt", med_config="/home/lw/ImageRward/ckpt/med_config. json")

Then I received the following error message:
OSError: Can't load tokenizer for 'bert-base-uncased'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'bert-base-uncased' is the correct path to a directory containing all relevant files for a BertTokenizer tokenizer.

May I ask how to solve this problem？

Memory leak with model loading

I noticed that image_reward.py loads model durring sdwebui start-up. Thats is waste of memory when script is not beeing used. Model should only be loaded when script is activated durring Generation not on sdwebui start-up.

Also if you use Reload UI in settings it is loads model again without unloading data from previous start.
After 5 UI reloads no free vram left:

By using Settings > Actions > Unload SD checkpoint to free VRAM only last loaded model is unloaded. Older loads sits in vram like garbage data.

Getting attribute error trying to use with Stable Diffusion

I followed the instructions to use ImageReward with Stable Diffusion's webui, but I get the following error:
AttributeError: module 'ImageReward' has no attribute 'load'

Error when running bash scripts/test-benchmark.sh

Dear authors,

When I run the test--benchmark.sh, there is an error:
line 167, in inference_rank
txt_features = torch.cat(txt_set, 0).float() # [image_num, feature_dim]
RuntimeError: torch.cat(): expected a non-empty list of Tensors

Thanks.

How to re-implement the result by using CLIP-score as the reward model?

Hello authors,
Thanks a lot for your wonderful work.
I want to know how to change the code and re-run it by using the CLIP-score as the reward model to have the baseline result?

Auto1111 Script not showing up in scripts

Getting this upon loading auto1111:

{"payload":{"allShortcutsEnabled":true,"fileTree":{"sdwebui":{"items":[{"name":"image_reward.py","path":"sdwebui/image_reward.py","contentType":"file"}],"totalCount":1},"":{"items":[{"name":"ImageReward","path":"ImageReward","contentType":"directory"},{"name":"assets","path":"assets","contentType":"directory"},{"name":"benchmark","path":"benchmark","contentType":"directory"},{"name":"data","path":"data","contentType":"directory"},{"name":"figures","path":"figures","contentType":"directory"},{"name":"scripts","path":"scripts","contentType":"directory"},{"name":"sdwebui","path":"sdwebui","contentType":"directory"},{"name":"train","path":"train","contentType":"directory"},{"name":".gitignore","path":".gitignore","contentType":"file"},{"name":"LICENSE","path":"LICENSE","contentType":"file"},{"name":"README.md","path":"README.md","contentType":"file"},{"name":"example.py","path":"example.py","contentType":"file"},{"name":"refl.py","path":"refl.py","contentType":"file"},{"name":"requirements.txt","path":"requirements.txt","contentType":"file"},{"name":"requirements_refl.txt","path":"requirements_refl.txt","contentType":"file"},{"name":"setup.py","path":"setup.py","contentType":"file"},{"name":"test-benchmark.py","path":"test-benchmark.py","contentType":"file"},{"name":"test.py","path":"test.py","contentType":"file"}],"totalCount":18}},"fileTreeProcessingTime":4.689069,"foldersToFetch":[],"reducedMotionEnabled":"system","repo":{"id":622147274,"defaultBranch":"main","name":"ImageReward","ownerLogin":"THUDM","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-04-01T04:04:17.000-05:00","ownerAvatar":"https://avatars.githubusercontent.com/u/48590610?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"main","listCacheKey":"v0:1681308785.0","canEdit":true,"refType":"branch","currentOid":"a4a2336f4109509ad2cc9e8af4274f41a130004d"},"path":"sdwebui/image_reward.py","currentUser":{"id":27867421,"login":"harspeck","userEmail":"[email protected]"},"blob":{"rawLines":["import os","import subprocess","import sys","from pathlib import Path","","import gradio as gr","import modules.images as images","import modules.scripts as scripts","import torch","from huggingface_hub.constants import HUGGINGFACE_HUB_CACHE","from modules import sd_samplers, shared","from modules.processing import ("," Processed,"," StableDiffusionProcessing,"," create_infotext,"," process_images,",")","from modules.shared import cmd_opts, opts, state","","try:"," import ImageReward as reward","except ModuleNotFoundError as error:"," print('"image-reward" package has not been properly installed. Installing...')"," if subprocess.check_call(["pip", "install", "image-reward"]) == 0:"," print(f'"image-reward" package is successfully installed!')"," import ImageReward as reward"," else:"," print('"image-reward" package installation failed!')"," print("," "Please open an issue with full error message at https://github.com/THUDM/ImageReward/issues\""," )","","","def unload_image_reward_model():"," del shared.image_reward_model","","","class Script(scripts.Script):"," def title(self):"," return "ImageReward - generate human preference scores"",""," def show(self, is_txt2img):"," return True",""," def ui(self, is_txt2img):"," with gr.Blocks():"," with gr.Row():"," gr.Markdown("," value="Tip: It will take a little time to load the ImageReward model before the first generation.""," )"," with gr.Row():"," with gr.Column():"," filter_out_low_scores = gr.Checkbox("," value=False, label="Filter out images with low scores""," )"," with gr.Column():"," lower_score_limit = gr.Textbox(value=0, label="Lower score limit")"," with gr.Row():"," gr.Markdown("," value="ImageReward model takes about 1,600 MB of memory.""," )"," with gr.Row():"," unload_button = gr.Button(value="Unload Model From Memory")"," unload_button.click(unload_image_reward_model)"," return [filter_out_low_scores, lower_score_limit]",""," def run(self, p, filter_out_low_scores, lower_score_limit):"," try:"," shared.image_reward_model # if loaded, do nothing"," except AttributeError:"," # load the model"," if sys.platform == "win32":"," download_root = HUGGINGFACE_HUB_CACHE"," else:"," download_root = None"," print(f"Loading ImageReward model from {download_root}...")"," shared.image_reward_model = reward.load("," "ImageReward-v1.0", download_root=download_root"," ) # using shared to make the model object global among modules",""," # preprocess parameters"," if lower_score_limit != "":"," lower_score_limit = float(lower_score_limit)",""," # generate images"," proc = process_images(p)",""," # score"," gens = proc.images"," with torch.no_grad():"," for img in gens:"," score = shared.image_reward_model.score(p.prompt, img)"," img.info["score"] = score"," if img.info.get("parameters") is None:"," img.info["parameters"] = f"ImageReward Score: {score:.4f}""," else:"," img.info["parameters"] += f"\n ImageReward Score: {score:.4f}"",""," # filter out images with scores lower than the lower limit"," if filter_out_low_scores:"," imgs = list(filter(lambda x: x.info["score"] > lower_score_limit, gens))"," else:"," imgs = gens",""," # append score to info"," infotexts = [img.info["parameters"] for img in imgs]",""," # sort to score"," img_info_list = list(zip(imgs, infotexts))"," img_info_list.sort(key=lambda x: x[0].info["score"], reverse=True)"," imgs, infotexts = list(zip(*img_info_list))",""," # return Processed object"," return Processed("," p=p,"," images_list=imgs,"," info=proc.info,"," seed=proc.seed,"," infotexts=infotexts,"," index_of_first_image=proc.index_of_first_image,"," )"],"stylingDirectives":[[{"start":0,"end":6,"cssClass":"pl-k"},{"start":7,"end":9,"cssClass":"pl-s1"}],[{"start":0,"end":6,"cssClass":"pl-k"},{"start":7,"end":17,"cssClass":"pl-s1"}],[{"start":0,"end":6,"cssClass":"pl-k"},{"start":7,"end":10,"cssClass":"pl-s1"}],[{"start":0,"end":4,"cssClass":"pl-k"},{"start":5,"end":12,"cssClass":"pl-s1"},{"start":13,"end":19,"cssClass":"pl-k"},{"start":20,"end":24,"cssClass":"pl-v"}],[],[{"start":0,"end":6,"cssClass":"pl-k"},{"start":7,"end":13,"cssClass":"pl-s1"},{"start":14,"end":16,"cssClass":"pl-k"},{"start":17,"end":19,"cssClass":"pl-s1"}],[{"start":0,"end":6,"cssClass":"pl-k"},{"start":7,"end":14,"cssClass":"pl-s1"},{"start":15,"end":21,"cssClass":"pl-s1"},{"start":22,"end":24,"cssClass":"pl-k"},{"start":25,"end":31,"cssClass":"pl-s1"}],[{"start":0,"end":6,"cssClass":"pl-k"},{"start":7,"end":14,"cssClass":"pl-s1"},{"start":15,"end":22,"cssClass":"pl-s1"},{"start":23,"end":25,"cssClass":"pl-k"},{"start":26,"end":33,"cssClass":"pl-s1"}],[{"start":0,"end":6,"cssClass":"pl-k"},{"start":7,"end":12,"cssClass":"pl-s1"}],[{"start":0,"end":4,"cssClass":"pl-k"},{"start":5,"end":20,"cssClass":"pl-s1"},{"start":21,"end":30,"cssClass":"pl-s1"},{"start":31,"end":37,"cssClass":"pl-k"},{"start":38,"end":59,"cssClass":"pl-v"}],[{"start":0,"end":4,"cssClass":"pl-k"},{"start":5,"end":12,"cssClass":"pl-s1"},{"start":13,"end":19,"cssClass":"pl-k"},{"start":20,"end":31,"cssClass":"pl-s1"},{"start":33,"end":39,"cssClass":"pl-s1"}],[{"start":0,"end":4,"cssClass":"pl-k"},{"start":5,"end":12,"cssClass":"pl-s1"},{"start":13,"end":23,"cssClass":"pl-s1"},{"start":24,"end":30,"cssClass":"pl-k"}],[{"start":4,"end":13,"cssClass":"pl-v"}],[{"start":4,"end":29,"cssClass":"pl-v"}],[{"start":4,"end":19,"cssClass":"pl-s1"}],[{"start":4,"end":18,"cssClass":"pl-s1"}],[],[{"start":0,"end":4,"cssClass":"pl-k"},{"start":5,"end":12,"cssClass":"pl-s1"},{"start":13,"end":19,"cssClass":"pl-s1"},{"start":20,"end":26,"cssClass":"pl-k"},{"start":27,"end":35,"cssClass":"pl-s1"},{"start":37,"end":41,"cssClass":"pl-s1"},{"start":43,"end":48,"cssClass":"pl-s1"}],[],[{"start":0,"end":3,"cssClass":"pl-k"}],[{"start":4,"end":10,"cssClass":"pl-k"},{"start":11,"end":22,"cssClass":"pl-v"},{"start":23,"end":25,"cssClass":"pl-k"},{"start":26,"end":32,"cssClass":"pl-s1"}],[{"start":0,"end":6,"cssClass":"pl-k"},{"start":7,"end":26,"cssClass":"pl-v"},{"start":27,"end":29,"cssClass":"pl-k"},{"start":30,"end":35,"cssClass":"pl-s1"}],[{"start":4,"end":9,"cssClass":"pl-en"},{"start":10,"end":81,"cssClass":"pl-s"}],[{"start":4,"end":6,"cssClass":"pl-k"},{"start":7,"end":17,"cssClass":"pl-s1"},{"start":18,"end":28,"cssClass":"pl-en"},{"start":30,"end":35,"cssClass":"pl-s"},{"start":37,"end":46,"cssClass":"pl-s"},{"start":48,"end":62,"cssClass":"pl-s"},{"start":65,"end":67,"cssClass":"pl-c1"},{"start":68,"end":69,"cssClass":"pl-c1"}],[{"start":8,"end":13,"cssClass":"pl-en"},{"start":14,"end":66,"cssClass":"pl-s"}],[{"start":8,"end":14,"cssClass":"pl-k"},{"start":15,"end":26,"cssClass":"pl-v"},{"start":27,"end":29,"cssClass":"pl-k"},{"start":30,"end":36,"cssClass":"pl-s1"}],[{"start":4,"end":8,"cssClass":"pl-k"}],[{"start":8,"end":13,"cssClass":"pl-en"},{"start":14,"end":59,"cssClass":"pl-s"}],[{"start":8,"end":13,"cssClass":"pl-en"}],[{"start":12,"end":105,"cssClass":"pl-s"}],[],[],[],[{"start":0,"end":3,"cssClass":"pl-k"},{"start":4,"end":29,"cssClass":"pl-en"}],[{"start":4,"end":7,"cssClass":"pl-k"},{"start":8,"end":14,"cssClass":"pl-s1"},{"start":15,"end":33,"cssClass":"pl-s1"}],[],[],[{"start":0,"end":5,"cssClass":"pl-k"},{"start":6,"end":12,"cssClass":"pl-v"},{"start":13,"end":20,"cssClass":"pl-s1"},{"start":21,"end":27,"cssClass":"pl-v"}],[{"start":4,"end":7,"cssClass":"pl-k"},{"start":8,"end":13,"cssClass":"pl-en"},{"start":14,"end":18,"cssClass":"pl-s1"}],[{"start":8,"end":14,"cssClass":"pl-k"},{"start":15,"end":63,"cssClass":"pl-s"}],[],[{"start":4,"end":7,"cssClass":"pl-k"},{"start":8,"end":12,"cssClass":"pl-en"},{"start":13,"end":17,"cssClass":"pl-s1"},{"start":19,"end":29,"cssClass":"pl-s1"}],[{"start":8,"end":14,"cssClass":"pl-k"},{"start":15,"end":19,"cssClass":"pl-c1"}],[],[{"start":4,"end":7,"cssClass":"pl-k"},{"start":8,"end":10,"cssClass":"pl-en"},{"start":11,"end":15,"cssClass":"pl-s1"},{"start":17,"end":27,"cssClass":"pl-s1"}],[{"start":8,"end":12,"cssClass":"pl-k"},{"start":13,"end":15,"cssClass":"pl-s1"},{"start":16,"end":22,"cssClass":"pl-v"}],[{"start":12,"end":16,"cssClass":"pl-k"},{"start":17,"end":19,"cssClass":"pl-s1"},{"start":20,"end":23,"cssClass":"pl-v"}],[{"start":16,"end":18,"cssClass":"pl-s1"},{"start":19,"end":27,"cssClass":"pl-v"}],[{"start":20,"end":25,"cssClass":"pl-s1"},{"start":25,"end":26,"cssClass":"pl-c1"},{"start":26,"end":130,"cssClass":"pl-s"}],[],[{"start":12,"end":16,"cssClass":"pl-k"},{"start":17,"end":19,"cssClass":"pl-s1"},{"start":20,"end":23,"cssClass":"pl-v"}],[{"start":16,"end":20,"cssClass":"pl-k"},{"start":21,"end":23,"cssClass":"pl-s1"},{"start":24,"end":30,"cssClass":"pl-v"}],[{"start":20,"end":41,"cssClass":"pl-s1"},{"start":42,"end":43,"cssClass":"pl-c1"},{"start":44,"end":46,"cssClass":"pl-s1"},{"start":47,"end":55,"cssClass":"pl-v"}],[{"start":24,"end":29,"cssClass":"pl-s1"},{"start":29,"end":30,"cssClass":"pl-c1"},{"start":30,"end":35,"cssClass":"pl-c1"},{"start":37,"end":42,"cssClass":"pl-s1"},{"start":42,"end":43,"cssClass":"pl-c1"},{"start":43,"end":78,"cssClass":"pl-s"}],[],[{"start":16,"end":20,"cssClass":"pl-k"},{"start":21,"end":23,"cssClass":"pl-s1"},{"start":24,"end":30,"cssClass":"pl-v"}],[{"start":20,"end":37,"cssClass":"pl-s1"},{"start":38,"end":39,"cssClass":"pl-c1"},{"start":40,"end":42,"cssClass":"pl-s1"},{"start":43,"end":50,"cssClass":"pl-v"},{"start":51,"end":56,"cssClass":"pl-s1"},{"start":56,"end":57,"cssClass":"pl-c1"},{"start":57,"end":58,"cssClass":"pl-c1"},{"start":60,"end":65,"cssClass":"pl-s1"},{"start":65,"end":66,"cssClass":"pl-c1"},{"start":66,"end":85,"cssClass":"pl-s"}],[{"start":12,"end":16,"cssClass":"pl-k"},{"start":17,"end":19,"cssClass":"pl-s1"},{"start":20,"end":23,"cssClass":"pl-v"}],[{"start":16,"end":18,"cssClass":"pl-s1"},{"start":19,"end":27,"cssClass":"pl-v"}],[{"start":20,"end":25,"cssClass":"pl-s1"},{"start":25,"end":26,"cssClass":"pl-c1"},{"start":26,"end":81,"cssClass":"pl-s"}],[],[{"start":12,"end":16,"cssClass":"pl-k"},{"start":17,"end":19,"cssClass":"pl-s1"},{"start":20,"end":23,"cssClass":"pl-v"}],[{"start":16,"end":29,"cssClass":"pl-s1"},{"start":30,"end":31,"cssClass":"pl-c1"},{"start":32,"end":34,"cssClass":"pl-s1"},{"start":35,"end":41,"cssClass":"pl-v"},{"start":42,"end":47,"cssClass":"pl-s1"},{"start":47,"end":48,"cssClass":"pl-c1"},{"start":48,"end":74,"cssClass":"pl-s"}],[{"start":16,"end":29,"cssClass":"pl-s1"},{"start":30,"end":35,"cssClass":"pl-en"},{"start":36,"end":61,"cssClass":"pl-s1"}],[{"start":8,"end":14,"cssClass":"pl-k"},{"start":16,"end":37,"cssClass":"pl-s1"},{"start":39,"end":56,"cssClass":"pl-s1"}],[],[{"start":4,"end":7,"cssClass":"pl-k"},{"start":8,"end":11,"cssClass":"pl-en"},{"start":12,"end":16,"cssClass":"pl-s1"},{"start":18,"end":19,"cssClass":"pl-s1"},{"start":21,"end":42,"cssClass":"pl-s1"},{"start":44,"end":61,"cssClass":"pl-s1"}],[{"start":8,"end":11,"cssClass":"pl-k"}],[{"start":12,"end":18,"cssClass":"pl-s1"},{"start":19,"end":37,"cssClass":"pl-s1"},{"start":39,"end":62,"cssClass":"pl-c"}],[{"start":8,"end":14,"cssClass":"pl-k"},{"start":15,"end":29,"cssClass":"pl-v"}],[{"start":12,"end":28,"cssClass":"pl-c"}],[{"start":12,"end":14,"cssClass":"pl-k"},{"start":15,"end":18,"cssClass":"pl-s1"},{"start":19,"end":27,"cssClass":"pl-s1"},{"start":28,"end":30,"cssClass":"pl-c1"},{"start":31,"end":38,"cssClass":"pl-s"}],[{"start":16,"end":29,"cssClass":"pl-s1"},{"start":30,"end":31,"cssClass":"pl-c1"},{"start":32,"end":53,"cssClass":"pl-v"}],[{"start":12,"end":16,"cssClass":"pl-k"}],[{"start":16,"end":29,"cssClass":"pl-s1"},{"start":30,"end":31,"cssClass":"pl-c1"},{"start":32,"end":36,"cssClass":"pl-c1"}],[{"start":12,"end":17,"cssClass":"pl-en"},{"start":18,"end":70,"cssClass":"pl-s"},{"start":51,"end":66,"cssClass":"pl-s1"},{"start":51,"end":52,"cssClass":"pl-kos"},{"start":52,"end":65,"cssClass":"pl-s1"},{"start":65,"end":66,"cssClass":"pl-kos"}],[{"start":12,"end":18,"cssClass":"pl-s1"},{"start":19,"end":37,"cssClass":"pl-s1"},{"start":38,"end":39,"cssClass":"pl-c1"},{"start":40,"end":46,"cssClass":"pl-s1"},{"start":47,"end":51,"cssClass":"pl-en"}],[{"start":16,"end":34,"cssClass":"pl-s"},{"start":36,"end":49,"cssClass":"pl-s1"},{"start":49,"end":50,"cssClass":"pl-c1"},{"start":50,"end":63,"cssClass":"pl-s1"}],[{"start":15,"end":75,"cssClass":"pl-c"}],[],[{"start":8,"end":31,"cssClass":"pl-c"}],[{"start":8,"end":10,"cssClass":"pl-k"},{"start":11,"end":28,"cssClass":"pl-s1"},{"start":29,"end":31,"cssClass":"pl-c1"},{"start":32,"end":34,"cssClass":"pl-s"}],[{"start":12,"end":29,"cssClass":"pl-s1"},{"start":30,"end":31,"cssClass":"pl-c1"},{"start":32,"end":37,"cssClass":"pl-en"},{"start":38,"end":55,"cssClass":"pl-s1"}],[],[{"start":8,"end":25,"cssClass":"pl-c"}],[{"start":8,"end":12,"cssClass":"pl-s1"},{"start":13,"end":14,"cssClass":"pl-c1"},{"start":15,"end":29,"cssClass":"pl-en"},{"start":30,"end":31,"cssClass":"pl-s1"}],[],[{"start":8,"end":15,"cssClass":"pl-c"}],[{"start":8,"end":12,"cssClass":"pl-s1"},{"start":13,"end":14,"cssClass":"pl-c1"},{"start":15,"end":19,"cssClass":"pl-s1"},{"start":20,"end":26,"cssClass":"pl-s1"}],[{"start":8,"end":12,"cssClass":"pl-k"},{"start":13,"end":18,"cssClass":"pl-s1"},{"start":19,"end":26,"cssClass":"pl-en"}],[{"start":12,"end":15,"cssClass":"pl-k"},{"start":16,"end":19,"cssClass":"pl-s1"},{"start":20,"end":22,"cssClass":"pl-c1"},{"start":23,"end":27,"cssClass":"pl-s1"}],[{"start":16,"end":21,"cssClass":"pl-s1"},{"start":22,"end":23,"cssClass":"pl-c1"},{"start":24,"end":30,"cssClass":"pl-s1"},{"start":31,"end":49,"cssClass":"pl-s1"},{"start":50,"end":55,"cssClass":"pl-en"},{"start":56,"end":57,"cssClass":"pl-s1"},{"start":58,"end":64,"cssClass":"pl-s1"},{"start":66,"end":69,"cssClass":"pl-s1"}],[{"start":16,"end":19,"cssClass":"pl-s1"},{"start":20,"end":24,"cssClass":"pl-s1"},{"start":25,"end":32,"cssClass":"pl-s"},{"start":34,"end":35,"cssClass":"pl-c1"},{"start":36,"end":41,"cssClass":"pl-s1"}],[{"start":16,"end":18,"cssClass":"pl-k"},{"start":19,"end":22,"cssClass":"pl-s1"},{"start":23,"end":27,"cssClass":"pl-s1"},{"start":28,"end":31,"cssClass":"pl-en"},{"start":32,"end":44,"cssClass":"pl-s"},{"start":46,"end":48,"cssClass":"pl-c1"},{"start":49,"end":53,"cssClass":"pl-c1"}],[{"start":20,"end":23,"cssClass":"pl-s1"},{"start":24,"end":28,"cssClass":"pl-s1"},{"start":29,"end":41,"cssClass":"pl-s"},{"start":43,"end":44,"cssClass":"pl-c1"},{"start":45,"end":78,"cssClass":"pl-s"},{"start":66,"end":77,"cssClass":"pl-s1"},{"start":66,"end":67,"cssClass":"pl-kos"},{"start":67,"end":72,"cssClass":"pl-s1"},{"start":76,"end":77,"cssClass":"pl-kos"}],[{"start":16,"end":20,"cssClass":"pl-k"}],[{"start":20,"end":23,"cssClass":"pl-s1"},{"start":24,"end":28,"cssClass":"pl-s1"},{"start":29,"end":41,"cssClass":"pl-s"},{"start":43,"end":45,"cssClass":"pl-c1"},{"start":46,"end":82,"cssClass":"pl-s"},{"start":48,"end":50,"cssClass":"pl-cce"},{"start":70,"end":81,"cssClass":"pl-s1"},{"start":70,"end":71,"cssClass":"pl-kos"},{"start":71,"end":76,"cssClass":"pl-s1"},{"start":80,"end":81,"cssClass":"pl-kos"}],[],[{"start":8,"end":66,"cssClass":"pl-c"}],[{"start":8,"end":10,"cssClass":"pl-k"},{"start":11,"end":32,"cssClass":"pl-s1"}],[{"start":12,"end":16,"cssClass":"pl-s1"},{"start":17,"end":18,"cssClass":"pl-c1"},{"start":19,"end":23,"cssClass":"pl-en"},{"start":24,"end":30,"cssClass":"pl-en"},{"start":31,"end":37,"cssClass":"pl-k"},{"start":38,"end":39,"cssClass":"pl-s1"},{"start":41,"end":42,"cssClass":"pl-s1"},{"start":43,"end":47,"cssClass":"pl-s1"},{"start":48,"end":55,"cssClass":"pl-s"},{"start":57,"end":58,"cssClass":"pl-c1"},{"start":59,"end":76,"cssClass":"pl-s1"},{"start":78,"end":82,"cssClass":"pl-s1"}],[{"start":8,"end":12,"cssClass":"pl-k"}],[{"start":12,"end":16,"cssClass":"pl-s1"},{"start":17,"end":18,"cssClass":"pl-c1"},{"start":19,"end":23,"cssClass":"pl-s1"}],[],[{"start":8,"end":30,"cssClass":"pl-c"}],[{"start":8,"end":17,"cssClass":"pl-s1"},{"start":18,"end":19,"cssClass":"pl-c1"},{"start":21,"end":24,"cssClass":"pl-s1"},{"start":25,"end":29,"cssClass":"pl-s1"},{"start":30,"end":42,"cssClass":"pl-s"},{"start":44,"end":47,"cssClass":"pl-k"},{"start":48,"end":51,"cssClass":"pl-s1"},{"start":52,"end":54,"cssClass":"pl-c1"},{"start":55,"end":59,"cssClass":"pl-s1"}],[],[{"start":8,"end":23,"cssClass":"pl-c"}],[{"start":8,"end":21,"cssClass":"pl-s1"},{"start":22,"end":23,"cssClass":"pl-c1"},{"start":24,"end":28,"cssClass":"pl-en"},{"start":29,"end":32,"cssClass":"pl-en"},{"start":33,"end":37,"cssClass":"pl-s1"},{"start":39,"end":48,"cssClass":"pl-s1"}],[{"start":8,"end":21,"cssClass":"pl-s1"},{"start":22,"end":26,"cssClass":"pl-en"},{"start":27,"end":30,"cssClass":"pl-s1"},{"start":30,"end":31,"cssClass":"pl-c1"},{"start":31,"end":37,"cssClass":"pl-k"},{"start":38,"end":39,"cssClass":"pl-s1"},{"start":41,"end":42,"cssClass":"pl-s1"},{"start":43,"end":44,"cssClass":"pl-c1"},{"start":46,"end":50,"cssClass":"pl-s1"},{"start":51,"end":58,"cssClass":"pl-s"},{"start":61,"end":68,"cssClass":"pl-s1"},{"start":68,"end":69,"cssClass":"pl-c1"},{"start":69,"end":73,"cssClass":"pl-c1"}],[{"start":8,"end":12,"cssClass":"pl-s1"},{"start":14,"end":23,"cssClass":"pl-s1"},{"start":24,"end":25,"cssClass":"pl-c1"},{"start":26,"end":30,"cssClass":"pl-en"},{"start":31,"end":34,"cssClass":"pl-en"},{"start":35,"end":36,"cssClass":"pl-c1"},{"start":36,"end":49,"cssClass":"pl-s1"}],[],[{"start":8,"end":33,"cssClass":"pl-c"}],[{"start":8,"end":14,"cssClass":"pl-k"},{"start":15,"end":24,"cssClass":"pl-v"}],[{"start":12,"end":13,"cssClass":"pl-s1"},{"start":13,"end":14,"cssClass":"pl-c1"},{"start":14,"end":15,"cssClass":"pl-s1"}],[{"start":12,"end":23,"cssClass":"pl-s1"},{"start":23,"end":24,"cssClass":"pl-c1"},{"start":24,"end":28,"cssClass":"pl-s1"}],[{"start":12,"end":16,"cssClass":"pl-s1"},{"start":16,"end":17,"cssClass":"pl-c1"},{"start":17,"end":21,"cssClass":"pl-s1"},{"start":22,"end":26,"cssClass":"pl-s1"}],[{"start":12,"end":16,"cssClass":"pl-s1"},{"start":16,"end":17,"cssClass":"pl-c1"},{"start":17,"end":21,"cssClass":"pl-s1"},{"start":22,"end":26,"cssClass":"pl-s1"}],[{"start":12,"end":21,"cssClass":"pl-s1"},{"start":21,"end":22,"cssClass":"pl-c1"},{"start":22,"end":31,"cssClass":"pl-s1"}],[{"start":12,"end":32,"cssClass":"pl-s1"},{"start":32,"end":33,"cssClass":"pl-c1"},{"start":33,"end":37,"cssClass":"pl-s1"},{"start":38,"end":58,"cssClass":"pl-s1"}],[]],"csv":null,"csvError":null,"dependabotInfo":{"showConfigurationBanner":false,"configFilePath":null,"networkDependabotPath":"/THUDM/ImageReward/network/updates","dismissConfigurationNoticePath":"/settings/dismiss-notice/dependabot_configuration_notice","configurationNoticeDismissed":false,"repoAlertsPath":"/THUDM/ImageReward/security/dependabot","repoSecurityAndAnalysisPath":"/THUDM/ImageReward/settings/security_analysis","repoOwnerIsOrg":true,"currentUserCanAdminRepo":false},"displayName":"image_reward.py","displayUrl":"https://github.com/THUDM/ImageReward/blob/main/sdwebui/image_reward.py?raw=true","headerInfo":{"blobSize":"4.14 KB","deleteInfo":{"deleteTooltip":"Fork this repository and delete the file"},"editInfo":{"editTooltip":"Fork this repository and edit the file"},"ghDesktopPath":"https://desktop.github.com","gitLfsPath":null,"onBranch":true,"shortPath":"3d445c1","siteNavLoginPath":"/login?return_to=https%3A%2F%2Fgithub.com%2FTHUDM%2FImageReward%2Fblob%2Fmain%2Fsdwebui%2Fimage_reward.py","isCSV":false,"isRichtext":false,"toc":null,"lineInfo":{"truncatedLoc":"121","truncatedSloc":"105"},"mode":"file"},"image":false,"isCodeownersFile":null,"isValidLegacyIssueTemplate":false,"issueTemplateHelpUrl":"https://docs.github.com/articles/about-issue-and-pull-request-templates","issueTemplate":null,"discussionTemplate":null,"language":"Python","languageID":303,"large":false,"loggedIn":true,"newDiscussionPath":"/THUDM/ImageReward/discussions/new","newIssuePath":"/THUDM/ImageReward/issues/new","planSupportInfo":{"repoIsFork":null,"repoOwnedByCurrentUser":null,"requestFullPath":"/THUDM/ImageReward/blob/main/sdwebui/image_reward.py","showFreeOrgGatedFeatureMessage":null,"showPlanSupportBanner":null,"upgradeDataAttributes":null,"upgradePath":null},"publishBannersInfo":{"dismissActionNoticePath":"/settings/dismiss-notice/publish_action_from_dockerfile","dismissStackNoticePath":"/settings/dismiss-notice/publish_stack_from_file","releasePath":"/THUDM/ImageReward/releases/new?marketplace=true","showPublishActionBanner":false,"showPublishStackBanner":false},"renderImageOrRaw":false,"richText":null,"renderedFileInfo":null,"tabSize":8,"topBannersInfo":{"overridingGlobalFundingFile":false,"globalPreferredFundingPath":null,"repoOwner":"THUDM","repoName":"ImageReward","showInvalidCitationWarning":false,"citationHelpUrl":"https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-on-github/about-citation-files","showDependabotConfigurationBanner":false,"actionsOnboardingTip":null},"truncated":false,"viewable":true,"workflowRedirectUrl":null,"symbols":{"timedOut":false,"notAnalyzed":false,"symbols":[{"name":"unload_image_reward_model","kind":"function","identStart":978,"identEnd":1003,"extentStart":974,"extentEnd":1040,"fullyQualifiedName":"unload_image_reward_model","identUtf16":{"start":{"lineNumber":33,"utf16Col":4},"end":{"lineNumber":33,"utf16Col":29}},"extentUtf16":{"start":{"lineNumber":33,"utf16Col":0},"end":{"lineNumber":34,"utf16Col":33}}},{"name":"Script","kind":"class","identStart":1049,"identEnd":1055,"extentStart":1043,"extentEnd":4236,"fullyQualifiedName":"Script","identUtf16":{"start":{"lineNumber":37,"utf16Col":6},"end":{"lineNumber":37,"utf16Col":12}},"extentUtf16":{"start":{"lineNumber":37,"utf16Col":0},"end":{"lineNumber":120,"utf16Col":9}}},{"name":"title","kind":"function","identStart":1081,"identEnd":1086,"extentStart":1077,"extentEnd":1157,"fullyQualifiedName":"Script.title","identUtf16":{"start":{"lineNumber":38,"utf16Col":8},"end":{"lineNumber":38,"utf16Col":13}},"extentUtf16":{"start":{"lineNumber":38,"utf16Col":4},"end":{"lineNumber":39,"utf16Col":63}}},{"name":"show","kind":"function","identStart":1167,"identEnd":1171,"extentStart":1163,"extentEnd":1210,"fullyQualifiedName":"Script.show","identUtf16":{"start":{"lineNumber":41,"utf16Col":8},"end":{"lineNumber":41,"utf16Col":12}},"extentUtf16":{"start":{"lineNumber":41,"utf16Col":4},"end":{"lineNumber":42,"utf16Col":19}}},{"name":"ui","kind":"function","identStart":1220,"identEnd":1222,"extentStart":1216,"extentEnd":2192,"fullyQualifiedName":"Script.ui","identUtf16":{"start":{"lineNumber":44,"utf16Col":8},"end":{"lineNumber":44,"utf16Col":10}},"extentUtf16":{"start":{"lineNumber":44,"utf16Col":4},"end":{"lineNumber":64,"utf16Col":57}}},{"name":"run","kind":"function","identStart":2202,"identEnd":2205,"extentStart":2198,"extentEnd":4236,"fullyQualifiedName":"Script.run","identUtf16":{"start":{"lineNumber":66,"utf16Col":8},"end":{"lineNumber":66,"utf16Col":11}},"extentUtf16":{"start":{"lineNumber":66,"utf16Col":4},"end":{"lineNumber":120,"utf16Col":9}}}]}},"copilotInfo":{"notices":{"codeViewPopover":{}},"userAccess":{"accessAllowed":false,"hasSubscriptionEnded":false,"orgHasCFBAccess":false,"userHasCFIAccess":false,"userHasOrgs":false,"userIsOrgAdmin":false,"userIsOrgMember":false,"business":null,"featureRequestInfo":{"showFeatureRequest":true,"alreadyRequested":false,"featureName":"copilot_for_business","requestPath":"/orgs/THUDM/member_feature_request"}}},"csrf_tokens":{"/THUDM/ImageReward/branches":{"post":"DqePMAYrzjnRmqcZvz0je0BcHrcAGWJbgY6ClINRa3oRo1ns3yQM88dDgxZohbk9j_8s-7kVBP6P8tBim7Bjww"}}},"title":"ImageReward/sdwebui/image_reward.py at main · THUDM/ImageReward"}
NameError: name 'true' is not defined

Any methods to reduce the gpu resources?

hello authors,
thanks for your wonderful project.
Since I am in the academic environment, i do not have sufficient gpu resources to run the experiment. May I ask, from your experiences, what kinds of changes of the code can reduce the gpu requirement?

Errors loading image_reward.py with SD-WEBUI

I followed the instructions for imagereward installation, and integration into sd-webui. Using automatic1111, getting the following when trying to startup. Imagereward of course, does not work.

ImageRewarderror.txt

about score

Hello author, I run the code example.py you provided, and some scores appear to be less than 0. May I ask, what is the meaning of less than 0? Does it mean that there is an error in this picture？

Implementation Details of Reward Weighted

Hi, Thanks for the nice work!

I noticed that Reward Weighted is implemented in the paper, but how is the probability $p(x|z)$ (Equation 4 in the Appendix) calculated (implemented in diffusers)? I also read the original paper carefully, but I didn't find the detailed description of this part.

how to reproduce fine-tune result？

Can I use following code to reproduce your fine-tuning?

from ImageReward import ReFL
args = ReFL.parse_args()
trainer = ReFL.Trainer("CompVis/stable-diffusion-v1-4", "data/refl_data.json", args=args)
trainer.train(args=args)