how to load data in inference.py about sagemaker-huggingface-inference-toolkit

[philschmid]

Hey @moshonkel,

what you described on what you want to sounds like a perfect use case for SageMaker Batch Transform, where you can provide files from s3 which are then used to run predictions.

I created a video and sample notebook on how to do batch transform: https://www.youtube.com/watch?v=lnTixz0tUBg&ab_channel=HuggingFace
To be able to use it you might need to create a custom inference.py with your custom logic.
Notebook: https://github.com/huggingface/notebooks/blob/master/sagemaker/12_batch_transform_inference/sagemaker-notebook.ipynb

[moshonkel]

This is the solution i came up with. You can just load data from s3 in the inference.py . For my case this is the most practical solution.

import boto3
import io

s3 = boto3.client('s3')
obj = s3.get_object(Bucket='custommodels', Key='sentences.csv')
df = pd.read_csv(io.BytesIO(obj['Body'].read()))
sentences = pd.read_csv('s3://custommodels/sentences.csv', index_col=0)

Still, thanks for the help @philschmid

[moshonkel]

Hey,

thanks for your answer. I do not understand how batch transform is helpful in this case.

I have an endpoint with a custom inference.py script, this is the inference.py:

import torch
import torch.nn.functional as F
import pandas as pd
import numpy as np

corpus_embeddings = np.load('s3://custommodels/sentence_embeddings.npy')
data = pd.read_csv('s3://custommodels/sentences.csv', index_col=0)

def similarity(embeddings_1, embeddings_2):
    normalized_embeddings_1 = F.normalize(embeddings_1, p=2)
    normalized_embeddings_2 = F.normalize(embeddings_2, p=2)
    return torch.matmul(
        normalized_embeddings_1, normalized_embeddings_2.transpose(0, 1)
    )

def output_fn(prediction, accept):
    prediction = torch.tensor(prediction)
    attention_mask = torch.ones(prediction.shape[1], dtype=torch.uint8)
    mask = attention_mask.unsqueeze(-1).expand(prediction.size()).float()
    masked_embeddings = prediction * mask
    summed = torch.sum(masked_embeddings, 1)
    summed_mask = torch.clamp(mask.sum(1), min=1e-9)
    query_embedding = (summed / summed_mask)
    scores = similarity(torch.tensor(corpus_embeddings, dtype=torch.float32), torch.tensor(query_embedding, dtype=torch.float32))
    maximum_score = torch.max(scores, 0)
    result = data.iloc[int(maximum_score[1])].to_dict()
    result['Ähnlichkeit'] = float(maximum_score[0][0])
    return result

The endpoint with the huggingfacemodel and the l2 regularization work fine the way i wrote it. Meaning i can get a correct sentence embedding from the endpoint.

What i am trying to do now is to compare the embedding from the inference of the endpoint, with embeddings, that are stored or deployed with the model

corpus_embeddings = np.load('s3://custommodels/sentence_embeddings.npy')

This does not work, and i dont understand how my inference.py can access data from anywhere (no matter if its on s3, or also deployed?, or anywhere else).
When i am in a sagemaker notebook, i can load files from s3 into ram. How is this done with an endpoint?

Ps the files i need are less than 10mb.

[b5y]

Hey @moshonkel !

The solution doesn't seem to be efficient. Here is why:

Every time when you call inference.py, you read the csv file from S3 bucket. It might be okay when the file is very small, but what will you do if the file is too large?

I am also facing similar problem right now and looking for the most efficient solution. Batch transform is also not something I would like to have as a proper solution.

Are there other ways to store the embeddings in the memory? Like we do with reading the model.

UPDATE: I do know there is a possibility to use some database, but currently want to avoid it.