scav6411 / personalised_handwriting_generator

IITISoC Project: Type In Your Handwriting

Contents

• Environment Requirements
• Overview
• Dataset
• Preview
• Training
• Results and Accuracy Measures
• Documentation
• References
• Contributors

Environment Requirements

• Python 3
• Pytorch 2.0.1
• cv2 4.0.1

Overview

This machine learning model learns and mimics personal handwriting style. The model is trained on a dataset of handwritten samples of a person and is capable of converting standard fonts into any given person’s personalized handwriting. It generates handwritten text that closely resembles unique writing style for various applications.

Dataset

We have used IAM Offline Handwriting Dataset consisting of over 20,000 samples of handwriting of over 500 different authors. We have used the handwriting samples of 339 authors for training purpose and 159 for validation purpose. This dataset was used because of its easy availability and diverse handwriting styles that were required for the Handwriting Imitation model.

• IAM Handwriting Database

Preview

Handwriting Imitation Model Algorithm

Handwriting Transformers model: Handwriting Transformers model works on generating the handwritten text of the query text based on the input handwriting. Our model has been trained on the vast IAM Dataset to generate handwriting using transformers and GAN. Later we finetune the model weights according to the input handwriting to generate images that match the input

Model architecture: The basic architecture of our GAN model is such that we have used transformer layers in our generator to do the encoding and decoding procedure. First of all we have cnn encoded images as input to the model which are converted to grayscale format and also padded. Subsequently the images are converted to tensor. These images go to the transformer encoder which processes the image descriptors helps in refining and enhancing the features. Then this goes to the transformer decoder layer where the query text is embedded with the input text.The query text is also encoded using the OCR Network encoder. Now this goes to the CNN decoder where the fake image generated by the generator is passed on to the discriminator and based on its feedback generator modifies its weights during the training.

Training

800 epochs	2000 epochs

4000 epochs	6000 epochs

8000 epochs	10000 epochs

Demo

See the random handwriting generation model here.

See the personalized handwriting generation model here.

Note: Please change the runtime the runtime to GPU before running all the cells. For personalised handwriting upload your image and change its image path in the mentioned location.

Website

You can check the video for our website. You can try our website for the demo file and try out our features here. Please download the zip file and change some paths.

Results and Accuracy Measures

We have used FID Score as our metrics for the following model. Below are some of our results and their fid scores.

FID SCORE IN VOCAB:113.253975	FID SCORE IN VOCAB:105.99905
FID SCORE OUT VOCAB:105.119934	FID SCORE OUT VOCAB:112.280205

Documentation

See the Handwriting Imitation document for full documentation on implementation of models and the website.

References

• https://github.com/M4rt1nM4yr/HandwritingImitationDemo/blob/main/HandwritingTransformers
• https://github.com/ganji15/HiGANplus
• https://openaccess.thecvf.com/content/ICCV2021/papers/Bhunia_Handwriting_Transformers_ICCV_2021_paper.pdf
• https://github.com/JaidedAI/EasyOCRtr
• https://huggingface.co/docs/transformers/model_doc/trocr

Contributers

• @sohampandit
• @lovelybhardwaj
• @rishitmehrotra
• @rainatathed