Can't get the right results about clip-based-nsfw-detector

[BIGJUN777]

shall we need to normalize the CLIP embedding before inputting it to the model?

[BIGJUN777]

I downloaded several images from laion2B-en-aesthetic and tried to use the CLIP model (ViT-L/14) to get the clip embedding and take it as input to the NSFW detector. However, the results were different from those shown on laion2B-en-aesthetic.

[rom1504]

Yes you need to normalize the embeddings in input

…

On Thu, Aug 25, 2022, 09:48 BIGJUN777 ***@***.***> wrote: I downloaded several images from laion2B-en-aesthetic <https://huggingface.co/datasets/laion/laion2B-en-aesthetic> and tried to use the CLIP model (ViT-L/14) to get the clip embedding and take it as input to the NSFW detector. However, the results were different from those shown on laion2B-en-aesthetic <https://huggingface.co/datasets/laion/laion2B-en-aesthetic>. — Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR437UBH2TZIBVBUJKO5TLV24QMLANCNFSM57RNSXEA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[BIGJUN777]

Yes, I tried to leverage the normalization from improved-aesthetic-predictor.

def normalized(self, a, axis=-1, order=2):
    l2 = np.atleast_1d(np.linalg.norm(a, order, axis))
    l2[l2 == 0] = 1
    return a / np.expand_dims(l2, axis)
    
def __call__(self, clip_embs):
    if isinstance(clip_embs, torch.Tensor):
        clip_embs = self.normalized(clip_embs.cpu().numpy())
    return self.model.predict_on_batch(clip_embs)

But the results still could not match your results shown on laion2B-en-aesthetic. I checked the img_embs in your provided dataset, and their data type is float16. I tried to use fp16 inference to get the float16 embeddings, but the results were still not correct. Did I miss something? Thanks.

with autocast(enabled=fp16_model):
    clip_embs = extractor.encode_image(data['image'].cuda())

I constructed the CLIP model as following:

class ClipExtractor(nn.Module):
    def __init__(self, model_name="ViT-L/14", jit=False):
        super().__init__()
        self.model, self.transform = clip.load(model_name, device='cpu', jit=self.jit)

    @torch.no_grad()
    def encode_image(self, images):
        with torch.no_grad():
            images_embeddings = self.model.encode_image(self.transform(images))
        return images_embeddings

[BIGJUN777]

@rom1504 @christophschuhmann Any ideas on my problems? Thanks.