justtuananh / gpt-4v_ocr

Evaluation of the Optical Character Recognition (OCR) capabilities of GPT-4V(ision)

GPT-4V_OCR

[arXiv 2310.16809]Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation

This paper presents a comprehensive evaluation of the Optical Character Recognition (OCR) capabilities of the recently released GPT-4V(ision), a Large Multimodal Model (LMM). We assess the model's performance across a range of OCR tasks, including scene text recognition, handwritten text recognition, handwritten mathematical expression recognition, table structure recognition, and information extraction from visually-rich document. The evaluation reveals that GPT-4V performs well in recognizing and understanding Latin contents, but struggles with multilingual scenarios and complex tasks. Based on these observations, we delve deeper into the necessity of specialized OCR models and deliberate on the strategies to fully harness the pretrained general LMMs like GPT-4V for OCR downstream tasks. The study offers a critical reference for future research in OCR with LMMs.

Scene Text Recognition

Scene Text Recognition (STR) aims to recognize textual information in natural scene pictures.

Handwritten Text Recognition

Handwritten Text Recognition (HTR) aims to recognize handwritten text.

Handwritten Mathematical Expression Recognition

Handwritten Mathematical Expression Recognition (HMER) aims to recognize handwritten mathematical formulas.

Visual Information Extraction

To learn more about Visual Information Extraction, please refer to Document-AI-Recommendations.

Visual Information Extraction(VIE) aims at mining, analyzing, and extracting key fields entities contained in visually rich documents. For example, given an image of a receipt, the VIE algorithms will tell information such as store name, product details, price, etc. For documents like forms, VIE algorithms will tell the key-value pairs contained.

Table Structure Recognition

To learn more about Table Structure Recognition, please refer to Document-AI-Recommendations.

Table Structure Recognition(TSR) aims to recognize the cellular structures of tables from table images by extracting the coordinates of cell boxes and row/column spanning information. This task is very challenging since tables may have complex structures, diverse styles and contents, and become geometrically distorted or even curved during an image capturing process.

Data Download

• Google Drive
• Baidu Drive(6wyf)

Citation

@misc{shi2023exploring,
      title={Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation}, 
      author={Yongxin Shi and Dezhi Peng and Wenhui Liao and Zening Lin and Xinhong Chen and Chongyu Liu and Yuyi Zhang and Lianwen Jin},
      year={2023},
      eprint={2310.16809},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}