microsoft / udop

Unifying Vision, Text, and Layout for Universal Document Processing

Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, Mohit Bansal

Code Release Here

Code is rehosted at part of the i-code project

Open Source Checklist:

• Release Model (Encoder + Text decoder)
• Release Most Scripts
• Vision Decoder / Weights (Due to fake document generation ethical consideration, we plan to release this functionality as an Azure API)
• Demos

Introduction

UDOP unifies vision, text, and layout through vision-text-layout Transformer and unified generative pretraining tasks including vision task, text task, layout task, and mixed task. We show the task prompts (left) and task targets (right) for all self-supervised objectives (joint text-layout reconstruction, visual text recognition, layout modeling, and masked autoencoding) and two example supervised objectives (question answering and layout analysis).

udop's Issues

Happy to support this model in 🤗 Transformers

Hi there,

Impressive work! We've got multiple Document AI models in 🤗 Transformers, including:

• LayoutLM
• LayoutLMv2
• LayoutLMv3
• LiLT
• Table Transformer
• MarkupLM
• Donut

So happy to add support for UDOP as well! Let me know whether you are interested in integrating the model in the library.

Number of Epochs for Fine-tuning Tasks

Hi Authors @zinengtang,
Appreciate your interesting paper UDOP. Wanted to know the specifics on fine-tuning. Appendix C.6 in the paper does provide some details but the crucial detail of number of epochs is missing. Specifically:

1. Number of epochs fine-tuned for each downstream task?
2. Are all downstream tasks fine-tuned for the same number of epochs?

p.s. I posted this question in i-code repo too but that repo seems to be inactive.

Question regarding Training of the

Thank you for publishing UDOP, a real step towards unified document processing!

Can you give Information about the training time (on the used equipment) for the self-supervised learning and the curriculum learning parts?

thank you in advance!