Online Inference Latency for XR-TRANSFORMER about pecos

[jiong-zhang]

Hi @xiaokening , the XRTransformer.predict is an API for batch inference (e.g. including dataset construction, batching, logging and etc). You can just XRTransformer.predict on the 100 data with batch_size=1 for an estimation of the realtime inference time. We are planning to enable a realtime inference API for XR-Transformer in later releases. Thanks.

[jiong-zhang]

What model (encoder, output space) are you running inference on? Could you provide sample code how you measure the latency?

[xiaokening]

the encoder is chinese-roberta-wwm-ext,the outspace is 1073 labels。the XR-TRANSFORMER model is embedded in Flask service。the measure method of the online Inference lattency is to send http requeset method-post。the average of the online Inference lattency of 100 instances is about 420ms。the running env that the Flask service is deployed is a docker container which has 8GPU, 16GB memory, 1 V100 GPU.

there is sample code:

import time
start = time.time()
for i, text in enumerate(X_text):
    if i >= 100:
        break
    payload={'text': text}
    r = requests.post(url_2, json=payload)
    res.append(r.json()["result"])
(time.time() - start) * 1000 / 100

there is another puzzling question。the average of the online Inference lattency of 100 instances is about 306ms using only CPU ! That's why ?

[jiong-zhang]

Could you also look at the inference time on your local GPU machine (without http request)?

[xiaokening]

Sorry, I don't have a a local GPU machine, I can test the online Inference lattency of XR-TRANSFORMER in a docker conotainer(I think the effect is the same as in a local GPU machine) based on k8s。
there is sample code:

import time

def predict(text):
    """
    Use the XR-Transformer model to predict on single text,and the predict time of single instance.
    
    args:
        text(str): the input text to predict on.
    
    return:
        pred_csr：instance to label prediction (csr_matrix, 1 * nr_labels).
        pred_time:the predict time of instance (unit: ms).
        
    """
    tokenized_text = ' '.join(HanLP(text))
    x = vectorizer.transform([tokenized_text])
    tfidf_x = tfidf_transformer.transform(x)
    tfidf_x.sort_indices()
    start_time = time.time()
    P_matrix = xtf.predict([text],X_feat=tfidf_x,use_gpu=True)
    pred_time = (time.time() - start_time) * 1000
    
    return P_matrix, pred_time
    
def test_online_inference_latency(corpus, n):
    """
    test the average online inference latency of XR-TRANSFORMER.
    
    args：
        corpus(List(str)):instance text list to predict on.
        n:num of ins for predict.
    
    return:
        average_time:the average online inference latency of sing instance.
    
    """
    total_time = 0
    for i, text in enumerate(corpus):
        if i == 0:
            _, _ = predict(text) # the first instance used for initializing XR-TRANSFORMER（copy to GPU）
            continue
        if i <= n:
            _, tmp = predict(text)
            total_time += tmp
        else:
            break
    return total_time / n

test_online_inference_latency(X_text, 100)

the average of the online Inference lattency of 100 instances is about 402ms.

[xiaokening]

Thanks! I have solved this @problem。@jiong-zhang