Mini project 5 · CMU 16-720
Neural networks for character recognition.
Reading handwritten text out of an image, which turns out to be as much a classical image processing problem as a learning one: the network only works once the characters have been found, isolated and ordered.
Finding the characters first
A fully connected neural network recognizes handwritten letters, trained on the NIST36 dataset to about 76% test accuracy. But given an image of text rather than a pre-cropped character, the first step is extraction: drawing a bounding box around each character.
The pipeline blurs, thresholds, and applies opening morphology to separate character pixels from the background, finds connected groups of character pixels and boxes each one, groups the boxes by the line of text they belong to and sorts them, then resizes each box to the network input and classifies it. Every one of those steps has to work before the network sees anything useful.
The trained network output worked well aside from a few misclassifications.
Image compression with autoencoders
An autoencoder is a network trained to approximately copy its input to its output, which is a useful way to learn compressed representations: forcing the signal through a narrow bottleneck means the network has to keep only what matters. By changing the activation function and adding momentum for speed, I got the output below, with an average PSNR (peak signal-to-noise ratio) of 15.68 across the images.
Going convolutional
To improve on the fully connected network, I also trained a convolutional neural network in PyTorch on the same NIST36 dataset. The CNN did significantly better, reaching 90.2% validation accuracy versus the 76% of the fully connected network, which is the expected payoff for a architecture that exploits spatial locality instead of flattening the image into an unstructured vector.
