Towards Self-Supervised Handwritten Text Recognition Using Generative Adversarial Networks

Koopmans, Lisa (2024) Towards Self-Supervised Handwritten Text Recognition Using Generative Adversarial Networks. Master's Thesis / Essay, Artificial Intelligence.

Preview	Text mAI2024KoopmansLD.pdf Download (30MB) | Preview
	Text toestemming_ Scriptie inzending van_ Lisa Koopmans _ opleiding_ Artificial Intelligence.pdf Restricted to Registered users only Download (183kB)

Abstract

Handwritten text recognition (HTR) heavily relies on supervised deep learning, which requires vast amounts of data. However, annotating handwritten documents takes much time and effort. This thesis proposes a novel approach that leverages handwritten text generation for the self-supervised training of HTR models. It is based on the idea that images with the same style and text are more similar than those with different texts. In the proposed framework, synthetic images are produced using the predicted text of an HTR network. A loss function computes the similarity between the input and synthetic images. We first explored the framework with the MNIST dataset, for which the findings indicate a high-level loss function is needed for decent performance. These results were used to apply the proposed framework to HTR. GANwriting was trained on half the IAM dataset to produce images with arbitrary style and text and integrated with a CNN-BLSTM HTR network architecture. Experiments with image-based and style-invariant losses were conducted, where self-supervised HTR models were trained on data independent from GANwriting. When no useful information was learned by the models, stylistic differences between input and synthetic images were minimized by conducting experiments with the image-based losses on data recreated with GANwriting. Our findings showed a high-quality generative model and background knowledge are necessary for adequate performance of image-based self-supervised HTR.

Item Type:	Thesis (Master's Thesis / Essay)
Supervisor name:	Dhali, M.A. and Schomaker, L.R.B.
Degree programme:	Artificial Intelligence
Thesis type:	Master's Thesis / Essay
Language:	English
Date Deposited:	10 Jul 2024 12:01
Last Modified:	10 Jul 2024 12:01
URI:	https://fse.studenttheses.ub.rug.nl/id/eprint/33045

Actions (login required)

View Item