gaparmar / clean-fid Goto Github PK

Hi, thanks for sharing your work.
In my case, I have a reference folder (fdir1), and about 50 target folders (fdir2), each containing hundreds of images. It takes a long time to calculate the score by default fid.compute_fid(fdir1, fdir2). It seems it is chocked by the CPU. Is there any way to speed up it?

Dear all,

I wonder whether this module can be applied to evaluate video quality.

Thanks,
Junyeong Ahn

Hi,

Thanks for maintaining the library for GAN research!
I noticed that the KID implementation in this repository is quite different from the official KID implementation
Could the author explain a bit about the differences?

Best,
Hubert

From 3 mins on a100 it went to 40 mins.

Also, Pytorch added antialiasing to interpolation, should it be used for speding up calculations?

Hi!
Uppercase JPEG extension ignored by get_folder_features
I've noticed this issue while making custom statistics from a folder with .JPEG files.
It seems like it has been thought of here for processing .zip

Probably the easiest fix is to expand the EXTENSIONS with the upper-case versions

Hello, I think the following implementation of the Fréchet distance is faster than the current one and would allow to drop the scipy dependency.

def frechet_distance(mu_x: Tensor, sigma_x: Tensor, mu_y: Tensor, sigma_y: Tensor) -> Tensor:
    a = (mu_x - mu_y).square().sum(dim=-1)
    b = sigma_x.trace() + sigma_y.trace()
    c = torch.linalg.eigvals(sigma_x @ sigma_y).sqrt().real.sum(dim=-1)

    return a + b - 2 * c

The implementation is based on two facts:

1. The trace of $A$ equals the sum of its eigenvalues.
2. The eigenvalues of $\sqrt{A}$ are the square-roots of the eigenvalues of $A$.

However, it is required when using fid.make_custom_stats(). I used pip.

Also, the README instructs one to pip install -r requirements.txt while such a file is not present.

Is the teaser figure depicting problems of downsampling still the same, given the new Pytorch 2.0 update? (i.e., is this issue the same for the current implementation of PyTorch bilinear...).

I know it is not a direct issue on the repo, but I could not find the answer or any updates about this.

Thanks for the great work!

I was trying to load a Generator from Progan implementation in pytorch. This generator receives a tensor of size
(batch_size, Z_DIM, 1, 1) as input, where Z_DIM is the latent vector dimension (e.g. 512). The code in
cleanfid/fid.py, line 214 accept an input of size z_batch = torch.randn((batch_size, z_dim)).to(device),
which is a 2D tensor, suitable only for non-RGB images (like in Vanilla GAN or other examples). I think
the code should also support 4D tensors for RGB images, as most GAN implementations accept
4D tensors as inputs to their generators.

[Error]:

compute FID of a model with <name-of-precomputed-statistics-file>
FID model:   0%|                                                                       | 0/1563 [00:00<?, ?it/s]
Traceback (most recent call last):
....
....
  File "/home/user/Documents/GANS/metrics/cleanfid/fid.py", line 251, in fid_model
    np_feats = get_model_features(G, model, mode=mode,
  File "/home/user/Documents/GANS/metrics/cleanfid/fid.py", line 214, in get_model_features
    img_batch = G(z_batch)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 204, in forward
    input = module(input)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 956, in forward
    return F.conv_transpose2d(
RuntimeError: Expected 3D (unbatched) or 4D (batched) input to conv_transpose2d, but got input of size: [32, 512]

code snippet [fid.py]:

"""
Compute the FID stats from a generator model
"""
def get_model_features(G, model, mode="clean", z_dim=512,
        num_gen=50_000, batch_size=128, device=torch.device("cuda"),
        desc="FID model: ", verbose=True, return_z=False,
        custom_image_tranform=None, custom_fn_resize=None):
    if custom_fn_resize is None:
        fn_resize = build_resizer(mode)
    else:
        fn_resize = custom_fn_resize
    
    # Generate test features
    num_iters = int(np.ceil(num_gen / batch_size))
    l_feats = []
    latents = []
    if verbose:
        pbar = tqdm(range(num_iters), desc=desc)
    else:
        pbar = range(num_iters)
    for idx in pbar:
        with torch.no_grad():
            z_batch = torch.randn((batch_size, z_dim)).to(device)
            if return_z:
                latents.append(z_batch)
            # generated image is in range [0,255]
            img_batch = G(z_batch)
            # split into individual batches for resizing if needed
            if mode != "legacy_tensorflow":
                l_resized_batch = []
                for idx in range(batch_size):

I'm very new to all this so this question might not make sense, but am I right in the assumption that FID relies on some pretrained model to evaluate the difference between the "fake" and "real" folders? If so, I assume such a model was trained only on generic photo content, which will limit its efficiency for drawn pictures and anime in particular. Or just having a large enough "real" folder will be enough?

If not, do I have to fine-tune said model for this specific content or there are pretrained models for 2D images that I could use here?

Default saved path
No such file or directory: '/tmp\\inception-2015-12-05.pt'
which is not so suitable for Windows ~

Hi Gaurav, thanks for sharing this amazing tool! I spot a block of suspicious lines that might worth your attention. Specifically, this resizing function of the default "clean" resizer:

It seems to me that the output_size in L47 (s1, s2) supposes to be a (w, h) tuple whilie L48 expects it as (h, w). What do you think? This might mean that the default resizer only works for square output resolution.

Shouldn't be a big problem since it does not affect default behavior.

May I ask what will be the difference if I directly call the function instead of clarifying its mode?

Hi there, thanks for this package, it's really helpful!

On a cluster with multiple GPUs, I have my model on device cuda:1.

When calculating FID with a passed gen function, new samples are generated during FID calculation. To that end, a model_fn(x) function is defined here:

and if use_dataparallel=True, the model will be wrapped with model = torch.nn.DataParallel(model).

Problem: DataParallel has a kwarg device_ids=None which defaults to all the available devices and then selects the first device as the "source" device, i.e., cuda:0. Later it asserts that all parameters and buffers of the model are on that device.
Now, if device_ids is not passed, this will result in an error because my model device is different from cuda:0.
I am wondering why DataParallel just hard codes everything to the first of all available devices, but there is a solution on the cleanfid side for this problem.

Solution: pass device_ids with the device of the model:

        if use_dataparallel:
            device_ids = [torch.cuda.current_device()]  # or use next(model.parameters()).device
            model = torch.nn.DataParallel(model, device_ids=device_ids)
        def model_fn(x): return model(x)

I would be happy to make a PR fixing this. Unless I am missing something?

Cheers,
Jan

This version is pretty old...I just want to know why is it being pinned, as it conflicts with many of the other libraries I have.

Trying to use CLIP FID after pip install clean-fid results in ModuleNotFoundError: No module named 'clip'. pip install git+https://github.com/openai/CLIP.git is missing from the requirements, though it is mentioned in

Thanks for sharing this great work. I'm just wondering about the reasons behind resize the image per channel and not resizing it as 3 channels array. I'm referring to this piece of code in the file resize.py.

def resize_single_channel(x_np, output_size):
    s1, s2 = output_size
    img = Image.fromarray(x_np.astype(np.float32), mode='F')
    img = img.resize(output_size, resample=Image.Resampling.BICUBIC)
    return np.asarray(img).clip(0, 255).reshape(s2, s1, 1)
def resize_channels(x, new_size):
    x = [resize_single_channel(x[:, :, idx], new_size) for idx in range(3)]
    x = np.concatenate(x, axis=2).astype(np.uint8)
    return x

I hope this is the way to ask questions here since It's my first time to ask a question on Issues instead of submitting an issue.

Thanks for your understanding.

Looks like the actual number of generations is rounded up to the nearest multiple of batch size:

for the Line 391 and 396 in compute_kid(), I think the 'None' should actually be 'feat_model'

Passing device="cpu" and model_name="clip_vit_b_32" crashes with the following error:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument weight in method wrapper___slow_conv2d_forward)

Hi,

Thanks for your contribution on GAN research!

We tried to get clean-fid for the images generated by styleganv2 using your code. However, the score was about 24, which was very different from the results reported in the paper. 50k images were generated using official checkpoints and calculated using pre-computed datasets statistics.

How to solve it?

Best,
Jungeun Kim

Has anyone encountered this issue?

Can I create custom dataset statistics to compute KID?

In my trial, after make_custom_stats, the invoking of compute_fid works fine but compute_kid tries to download the statistics:

>>> fid.compute_kid('sunglasses/0', dataset_name='sunglasses', mode='clean', dataset_res=256, dataset_split='custom')
compute KID of a folder with sunglasses statistics
downloading statistics to /home/brando/.local/lib/python3.7/site-packages/cleanfid/stats/sunglasses_clean_custom_na_kid.npz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/brando/.local/lib/python3.7/site-packages/cleanfid/fid.py", line 335, in compute_kid
    mode=mode, seed=0, split=dataset_split, metric="KID")
  File "/home/brando/.local/lib/python3.7/site-packages/cleanfid/features.py", line 95, in get_reference_statistics
    fpath = check_download_url(local_folder=stats_folder, url=url)
  File "/home/brando/.local/lib/python3.7/site-packages/cleanfid/downloads_helper.py", line 27, in check_download_url
    with urllib.request.urlopen(url) as response, open(local_path, 'wb') as f:
  File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

Detailed context can be found in #9.

We follow the StyleGAN-ADA to extract FFHQ256 from tfrecord file. However, when I computed the statistic, it results in different statistics from your pre-computed trainval70K. I want to know whether the calculation steps have changed?

Recently, I've come across a post on LinkedIn that describes how we should carefully choose the right resize function while stressing the fact that using different libraries/frameworks leads to different results. So, I decided to test it myself.
Click here to find the post that I took the inspiration from.

The following is the code snippet that I've edited(using this colab notebook) to give the correct way of using resize methods in different frameworks.

import numpy as np
import torch
import torchvision.transforms.functional as F
from torchvision import transforms
from torchvision.transforms.functional import InterpolationMode
from PIL import Image
import tensorflow as tf
import cv2
import matplotlib.pyplot as plt
from skimage import draw

image = np.ones((128, 128), dtype=np.float64)
rr, cc = draw.circle_perimeter(64, 64, radius=45, shape=image.shape)
image[rr, cc] = 0
plt.imshow(image, cmap='gray')
print(f"Unique values of image: {np.unique(arr)}")
print(image.dtype)
output_size = 17
def inspect_img(*, img):
    plt.imshow(img, cmap='gray')
    print(f"Value of pixel with coordinates (14,9): {img[14, 9]}")

def resize_PIL(*, img, output_size):
    img = Image.fromarray(image)
    img = img.resize((output_size, output_size), resample=Image.BICUBIC)
    img = np.asarray(img,dtype=np.float64)
    inspect_img(img=img)
    return img
def resize_pytorch(*, img, output_size):
    img = F.resize(Image.fromarray(np.float64(img)), # Provide a PIL image rather than a Tensor.
                   size=output_size, 
                   interpolation=InterpolationMode.BICUBIC)
    img = np.asarray(img, dtype=np.float64) 
    inspect_img(img=img)
    return img
def resize_tensorflow(*, img, output_size):
    img = img[tf.newaxis, ..., tf.newaxis]
    img = tf.image.resize(img, size = [output_size] * 2, method="bicubic", antialias=True)
    img = img[0, ..., 0].numpy()
    inspect_img(img=img)
    return img
image_PIL = resize_PIL(img=image, output_size=output_size)
image_pytorch = resize_pytorch(img=image, output_size=output_size)
image_tensorflow = resize_tensorflow(img=image, output_size=output_size)
assert np.array_equal(image_PIL, image_pytorch) == True, 'Not Identical!'
# assert np.array_equal(image_PIL, image_tensorflow) == True, 'Not Identical!'  --> fails
assert np.allclose(image_PIL, image_tensorflow) == True, 'Not Close!'
# assert np.array_equal(image_tensorflow, image_pytorch) == True, 'Not Identical!'  --> fails 
assert np.allclose(image_tensorflow, image_pytorch) == True, 'Not Close!'
# tensorflow gives a slightly different values than pytorch and PIL.

which gives us the following results:

Therefore, TensorFlow, PyTorch, and PIL give similar results if the resize method is used properly like in the above snippet code.

You can read my comments on linkedin to find out how I came to this solution.

The only remaining library is OpenCV which I'll test in the future.

Have a great day/night!

I have this problem, and I guess it may relate to the version of pytorch or other environments, I don't know how to deal with.

/home/m11113013/.local/lib/python3.8/site-packages/scipy/init.py:138: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3)
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion} is required for this version of "
compute FID between two folders
Found 8091 images in the folder /dataset/flickr/images/
FID : 0%| | 0/506 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/m11113013/ProjectCode/MasterProject4/model/metric.py", line 12, in
score = calculate_fid(p1, p1)
File "/home/m11113013/ProjectCode/MasterProject4/model/metric.py", line 5, in calculate_fid
return fid.compute_fid(x_dir, y_dir, mode='clean', num_workers=0, batch_size=16, device=torch.device("cpu"))
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/fid.py", line 478, in compute_fid
score = compare_folders(fdir1, fdir2, feat_model,
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/fid.py", line 269, in compare_folders
np_feats1 = get_folder_features(fdir1, feat_model, num_workers=num_workers,
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/fid.py", line 147, in get_folder_features
np_feats = get_files_features(files, model, num_workers=num_workers,
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/fid.py", line 119, in get_files_features
l_feats.append(get_batch_features(batch, model, device))
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/fid.py", line 88, in get_batch_features
feat = model(batch.to(device))
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/features.py", line 25, in model_fn
def model_fn(x): return model(x)
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 154, in forward
raise RuntimeError("module must have its parameters and buffers "
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu

import torch
from cleanfid import fid

def calculate_fid(x_dir, y_dir):
return fid.compute_fid(x_dir, y_dir, mode='clean', num_workers=0, batch_size=16, device=torch.device("cpu"))

if name == "main":
p1 = '/dataset/flickr/images/'
score = calculate_fid(p1, p1)
print(score)

python version: 3.8.16
pytorch version: 1.12.1
cuda version: 11.3

RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

How to solve it?

Dear clean-fid group,

Thank you for sharing this great work, I really like it.

You mention in the paper that the PIL implementation is anti alias. But the PIL library could not back backpropagate gradient. Do you have plan to implement a way resize function that support anti alias and backpropagation? Or could you inform me the right way to do this?

I believe this will be very usefull for the community. For example, we invert a real image to the latent space of GAN, usually we dont use the full resolution image. The generated image must downsampling then compare to real image (LPIPS), and then backpropagate gradient. This is the implementation in Stylegan-ada (https://github.com/NVlabs/stylegan2-ada-pytorch/blob/main/projector.py#L97), it uses the torch resize function which could not anti alias as you mentioned in the paper.

Thank you for your help.

Best Wishes,

Alex

I am working on vid2vid GAN model from Nvidia-Imaginaire library. It uses clean-fid library. While running the model on google colab. I encounter this error related to the clean-fid.

from imaginaire.evaluation import compute_fid
File "/content/imaginaire/imaginaire/evaluation/init.py", line 5, in
from .fid import compute_fid, compute_fid_data
File "/content/imaginaire/imaginaire/evaluation/fid.py", line 10, in
from imaginaire.evaluation.common import load_or_compute_activations
File "/content/imaginaire/imaginaire/evaluation/common.py", line 14, in
from cleanfid.resize import build_resizer
File "/usr/local/lib/python3.7/dist-packages/cleanfid/resize.py", line 10, in
from cleanfid.utils import *
File "/usr/local/lib/python3.7/dist-packages/cleanfid/utils.py", line 5, in
from cleanfid.resize import build_resizer
ImportError: cannot import name 'build_resizer' from 'cleanfid.resize' (/usr/local/lib/python3.7/dist-packages/cleanfid/resize.py)

The Issue is it is not able import the build_resizer from cleanfid.resize.
I don't have this issue like two days back now I am having this issue. Is it due to the new version.

Thanks in advance

Thanks for this excellent repository. Comparing with https://github.com/mseitzer/pytorch-fid, I would like to extract features from different pooling layers like the first max pooling features (64), second max pooling features (192), pre-aux classifier features (768), and final average pooling features (2048) and compare FID scores. I believe the default option in your case is extracting the features from the final average pooling layer. Correct me if I am wrong.

from cleanfid import fid
fdir1 = '/content/gdrive/MyDrive/syn'
fdir2 = '/content/gdrive/MyDrive/orig'
score = fid.compute_fid(fdir1, fdir2)
print(score)

Is there an option to modify the function call to extract features from different layers and compare the scores? Thanks in advance.

I found that when calculating custom dataset, using UPPERCASE named custom_name such as "dragan_Rs", it will be ERROR. And i change it to "dragan_rs", its OK.
I don't know why.

The complete error as follows:
Traceback (most recent call last):
  File "/home/user/duzongwei/Projects/FSGAN/metrics/fid.py", line 46, in <module>
    compute_fid_kid(fake_fdir, custom_name)
  File "/home/user/duzongwei/Projects/FSGAN/metrics/fid.py", line 15, in compute_fid_kid
    score_fid = fid.compute_fid(fake_fdir, dataset_name=custom_name, mode=mode, dataset_split=dataset_split)
  File "/home/user/anaconda3/envs/dzw_gan/lib/python3.7/site-packages/cleanfid/fid.py", line 456, in compute_fid
    batch_size=batch_size, device=device, verbose=verbose)
  File "/home/user/anaconda3/envs/dzw_gan/lib/python3.7/site-packages/cleanfid/fid.py", line 179, in fid_folder
    mode=mode, seed=0, split=dataset_split)
  File "/home/user/anaconda3/envs/dzw_gan/lib/python3.7/site-packages/cleanfid/features.py", line 58, in get_reference_statistics
    fpath = check_download_url(local_folder=stats_folder, url=url)
  File "/home/user/anaconda3/envs/dzw_gan/lib/python3.7/site-packages/cleanfid/downloads_helper.py", line 36, in check_download_url
    with urllib.request.urlopen(url) as response, open(local_path, 'wb') as f:
  File "/home/user/anaconda3/envs/dzw_gan/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/home/user/anaconda3/envs/dzw_gan/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/home/user/anaconda3/envs/dzw_gan/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)

Thank you for your great work!! It's really helpful.

I wonder if we can calculate the fid between a numpy file (.npy) that contains an array in the shape (B, C, H, W) and pre-computed datasets statistics?

Massive thanks in advance.

Env: python 3.9.6, clean-fid: 0.1.15
My code:

from cleanfid import fid
score = fid.compute_fid('./fake_images', '../real_images')

Error:

  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\site-packages\cleanfid\fid.py", line 389, in compute_fid
    score = compare_folders(fdir1, fdir2, feat_model,
  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\site-packages\cleanfid\fid.py", line 238, in compare_folders
    np_feats1 = get_folder_features(fdir1, feat_model, num_workers=num_workers,
  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\site-packages\cleanfid\fid.py", line 131, in get_folder_features
    np_feats = get_files_features(files, model, num_workers=num_workers,
  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\site-packages\cleanfid\fid.py", line 109, in get_files_features
    for batch in tqdm(dataloader, desc=description):
  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\site-packages\tqdm\std.py", line 1180, in __iter__
    for obj in iterable:
  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\utils\data\dataloader.py", line 359, in __iter__
    return self._get_iterator()
  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\utils\data\dataloader.py", line 305, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\utils\data\dataloader.py", line 918, in __init__
    w.start()
  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\10034\AppData\Local\Programs\Python\Python39\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'make_resizer.<locals>.func'

Hi,

I am a little confused that should the resolution of images in 2 folders be the same or different (folder1: 256x256, folder2: 1024x1024)? If not, can we use PIL to resize it or use torch transform resize?

Many thanks.

When I download one from internet, how can I use it directly.

I want to cleanly calculate the FID score because I had some differences due to resizing. I appreciate the work you have done to make the process more consistent.

I am running the module on PyTorch 1.9.0, CUDA 11.2, and get the following error when calculating the FID score in the "clean" mode using the "torchscript_inception" model from https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/inception-2015-12-05.pt:

RuntimeError: MALFORMED INPUT: lanes dont match
On the following line:

It is a very peculiar error coming from this file:
https://github.com/pytorch/pytorch/blob/fce85480b97d8d79e92e11fbcd3c03a25ae446e0/torch/csrc/jit/tensorexpr/types.h#L35

I'm not familiar with TorchScript but maybe it has something to do with the model being composed/uploaded in another Pytorch version. It is probably better to ask in the Pytorch github what causes this.

However, I want to ask if just using the regular InceptionV3 model provided here would give the same FID score / would also be good practice. In short: calculate FID in "clean" mode but use the "pytorch_inception" model

Hello, I tried using the package, but its throwing this runtime error. I checked image size mismatch, whether they are corrupted or not but no leads as to what is causing this.

This code is from fid.py line 459-470.

elif gen is not None:
    if not verbose:
        print(f"compute FID of a model with {dataset_name}-{dataset_res} statistics")
    score = fid_model(gen, dataset_name, dataset_res, dataset_split,
            model=feat_model, z_dim=z_dim, num_gen=num_gen,
            mode=mode, num_workers=num_workers, batch_size=batch_size,
            device=device, verbose=verbose)
    return score

# compute fid for a generator, using images in fdir2
elif gen is not None and fdir2 is not None:

There is no way we enter the last elif, so I can't compare my generator with images in fdir2. Is this intentional?

I tried fid.compute_fid function with cifar10 dataset.
It went perfectly until last week.
Seem like the dataset's URL is no longer supported.
Does anyone have the same error as me?

compute FID of a folder with cifar10 statistics
downloading statistics to /usr/local/lib/python3.10/dist-packages/cleanfid/stats/cifar10_clean_train_1024.npz

HTTPError Traceback (most recent call last)
in <cell line: 1>()
----> 1 score_clean = fid.compute_fid("folder_real", dataset_name="cifar10")
2 print(f"clean-fid score is {score_clean:.3f}")

9 frames
/usr/local/lib/python3.10/dist-packages/cleanfid/fid.py in compute_fid(fdir1, fdir2, gen, mode, model_name, num_workers, batch_size, device, dataset_name, dataset_res, dataset_split, num_gen, z_dim, custom_feat_extractor, verbose, custom_image_tranform, custom_fn_resize, use_dataparallel)
488 if verbose:
489 print(f"compute FID of a folder with {dataset_name} statistics")
--> 490 score = fid_folder(fdir1, dataset_name, dataset_res, dataset_split,
491 model=feat_model, mode=mode, model_name=model_name,
492 custom_fn_resize=custom_fn_resize, custom_image_tranform=custom_image_tranform,

/usr/local/lib/python3.10/dist-packages/cleanfid/fid.py in fid_folder(fdir, dataset_name, dataset_res, dataset_split, model, mode, model_name, num_workers, batch_size, device, verbose, custom_image_tranform, custom_fn_resize)
171 custom_image_tranform=None, custom_fn_resize=None):
172 # Load reference FID statistics (download if needed)
--> 173 ref_mu, ref_sigma = get_reference_statistics(dataset_name, dataset_res,
174 mode=mode, model_name=model_name, seed=0, split=dataset_split)
175 fbname = os.path.basename(fdir)

/usr/local/lib/python3.10/dist-packages/cleanfid/features.py in get_reference_statistics(name, res, mode, model_name, seed, split, metric)
64 mod_path = os.path.dirname(cleanfid.file)
65 stats_folder = os.path.join(mod_path, "stats")
---> 66 fpath = check_download_url(local_folder=stats_folder, url=url)
67 stats = np.load(fpath)
68 mu, sigma = stats["mu"], stats["sigma"]

/usr/local/lib/python3.10/dist-packages/cleanfid/downloads_helper.py in check_download_url(local_folder, url)
34 os.makedirs(local_folder, exist_ok=True)
35 print(f"downloading statistics to {local_path}")
---> 36 with urllib.request.urlopen(url) as response, open(local_path, 'wb') as f:
37 shutil.copyfileobj(response, f)
38 return local_path

/usr/lib/python3.10/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
214 else:
215 opener = _opener
--> 216 return opener.open(url, data, timeout)
217
218 def install_opener(opener):

/usr/lib/python3.10/urllib/request.py in open(self, fullurl, data, timeout)
523 for processor in self.process_response.get(protocol, []):
524 meth = getattr(processor, meth_name)
--> 525 response = meth(req, response)
526
527 return response

/usr/lib/python3.10/urllib/request.py in http_response(self, request, response)
632 # request was successfully received, understood, and accepted.
633 if not (200 <= code < 300):
--> 634 response = self.parent.error(
635 'http', request, response, code, msg, hdrs)
636

/usr/lib/python3.10/urllib/request.py in error(self, proto, *args)
561 if http_err:
562 args = (dict, 'default', 'http_error_default') + orig_args
--> 563 return self._call_chain(*args)
564
565 # XXX probably also want an abstract factory that knows when it makes

/usr/lib/python3.10/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
494 for handler in handlers:
495 func = getattr(handler, meth_name)
--> 496 result = func(*args)
497 if result is not None:
498 return result

/usr/lib/python3.10/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
641 class HTTPDefaultErrorHandler(BaseHandler):
642 def http_error_default(self, req, fp, code, msg, hdrs):
--> 643 raise HTTPError(req.full_url, code, msg, hdrs, fp)
644
645 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 404: Not Found

Would be possible to add pre-computed statistics for MSCoco dataset?

Hi,

I am facing the Imaginary component issue unless I have more than 2048 images in each folder.

I use this line of code to compute it.
fid_score = fid.compute_fid(source, target, mode="clean", verbose=True, dataset_split="custom", use_dataparallel=False)

The weird thing is that happens only on validation set, on a subset of training set it works well. Unless I perform exact same operations on both, as saving model predictions and targets using torchvision.

I am using clean-fid==0.1.35
CUDA 11.8
Python 3.10.13
numpy==1.26
torch==2.0.1
Debian 11

Thanks for tyour help in advance ;)

My dataset has nonsquare samples, so on training, I'm doing random crop and for validation, I'm calculating fid with center cropped copy of dataset.
Would be cool to have some option for passing my own transform, for example.

I am trying to compute KID, but it is generating negative values. Can KID be a negative number?

Here is the code that I used:
`
from cleanfid import fid
fdir1 = my_folder1_path
fdir2 = my_folder2_path

kid_score = fid.compute_kid(fdir1, fdir2)
`

Each folder has only 6 images. My kid_score is -0.0406.

Could someone please help me understand why the KID is less that zero?

Thank you,
Chandrakanth

Hi, thanks for your work,
I wanted to calculate the FID score between multiple generations of the same input image. I setup my directories such that directory 1 contains n generated image and directory 2 contains n copies of the input image. Is this ok way to go about this ? I ran into the following error while doing this :
RuntimeWarning: invalid value encountered in scalar divide arg2 = norm(X.dot(X) - A, 'fro')**2 / norm(A, 'fro')

Thank you

UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.3)
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion} is required for this version of "
compute FID between two folders
Found 8091 images in the folder /dataset/flickr/images/
FID : 0%| | 0/506 [00:02<?, ?it/s]
Traceback (most recent call last):
File "/home/m11113013/ProjectCode/MasterProject4/model/metric.py", line 11, in
score = calculate_fid(p1, p1)
File "/home/m11113013/ProjectCode/MasterProject4/model/metric.py", line 4, in calculate_fid
return fid.compute_fid(x_dir, y_dir, mode='clean', num_workers=0, batch_size=16)
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/fid.py", line 478, in compute_fid
score = compare_folders(fdir1, fdir2, feat_model,
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/fid.py", line 269, in compare_folders
np_feats1 = get_folder_features(fdir1, feat_model, num_workers=num_workers,
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/fid.py", line 147, in get_folder_features
np_feats = get_files_features(files, model, num_workers=num_workers,
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/fid.py", line 119, in get_files_features
l_feats.append(get_batch_features(batch, model, device))
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/fid.py", line 88, in get_batch_features
feat = model(batch.to(device))
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/features.py", line 25, in model_fn
def model_fn(x): return model(x)
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 168, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 178, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
output.reraise()
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/torch/_utils.py", line 461, in reraise
raise exception
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
output = module(*input, **kwargs)
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/m11113013/miniconda3/envs/pytorch/lib/python3.8/site-packages/cleanfid/inception_torchscript.py", line 54, in forward
features = self.layers.forward(x2, ).view((bs, 2048))
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
File "code/torch/torch/nn/modules/container/___torch_mangle_9.py", line 45, in forward
_17 = self.mixed_10
_18 = self.pool2
input0 = (_0).forward(input, )
~~~~~~~~~~~ <--- HERE
input1 = (_1).forward(input0, )
input2 = (_2).forward(input1, )
File "code/torch.py", line 74, in forward
_22 = self.stride
_23 = self.padding
x3 = torch.conv2d(x, _21, None, [_22, _22], [_23, _23], [1, 1], 1)
~~~~~~~~~~~~ <--- HERE
x4 = _20(x3, self.mean, self.var, None, self.beta, False, 0.10000000000000001, 0.001, )
x5 = torch.torch.nn.functional.relu(x4, False, )

Traceback of TorchScript, original code (most recent call last):
File "C:\Users\tkarras\Anaconda3\lib\site-packages\torch\nn\modules\container.py", line 117, in forward
def forward(self, input):
for module in self:
input = module(input)
~~~~~~ <--- HERE
return input
File "c:\p4research\research\tkarras\dnn\gan3support\feature_detectors\inception.py", line 28, in forward
def forward(self, x):
x = torch.nn.functional.conv2d(x, self.weight.to(x.dtype), stride=self.stride, padding=self.padding)
~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
x = torch.nn.functional.batch_norm(x, running_mean=self.mean, running_var=self.var, bias=self.beta, eps=1e-3)
x = torch.nn.functional.relu(x)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

from cleanfid import fid

def calculate_fid(x_dir, y_dir):
return fid.compute_fid(x_dir, y_dir, mode='clean', num_workers=0, batch_size=16)

if name == "main":
p1 = '/dataset/flickr/images/'
score = calculate_fid(p1, p1)
print(score)

python version: 3.8.16
pytorch version: 1.12.1
cuda version:11.3

Hello, thanks for the great work and the package.
Are there any plans to release ImageNet-1k statistics?
if not, I can try to do it, and provide the steps to reproduce.