Google's InkSight: AI-Powered Handwriting to Digital Text Conversion

Google Inksight lets you convert your scrappy handwriting to digital text using AI

Google has just announced a new model technology called Insight, which it's calling an 'offline to online handwriting conversion tool'. Basically that means scanning a photo of handwritten text and extracting the letters using AI rather than optical scanning as with OCR. The team has trained the model so it can actually 'read' and recognize words, and then employ digital handwriting methods to output accurate letters. The developers claim that doing this handwriting conversion digitally is more resilient across various scenarios, such as dealing with badly lit photo sources and on-page artifacts. The fun, and maybe useful, feature is its ability to output digital content in the shape of normal handwriting. For example, the team is suggesting that users could take their handwritten notes, convert them to a digital clone of their handwriting, edit the digital form, and then re-export the result in their original handwriting. The result of all this could be a document archive containing handwritten notes which could be freely edited, and yet retain the character of the original notes in digital form. It's like having an indestructible version of your personal notes, available online, or on your computer, for permanent access. Unsurprisingly a lot of interest in the new technology has come from academia and other groups, who have struggled for years to make OCR and scanners fit the need for fast, accurate notation and storage. This becomes even more acute for field workers who don't have access to specialized equipment. Think archaeological dig notes, insurance assessments and other similar situations. At the moment their options are to store scrappy-paper notes or use pen and tablet-based computer transcription, which comes with its own set of accuracy issues. The technology is still in the research stage at the moment, but the results are looking promising for future development. The team at Google has conducted tests showing that the output from the new models -- they've released three different-sized versions -- delivered results that in 87% of cases were considered good or with only minor defects. It remains to be seen how long it will take before this form of AI reaches the hands of the public. But anyone who's ever struggled with reading a GP's prescription will cheer from the sidelines when it eventually does arrive.

Google shows off AI tool for reading handwritten text by rewriting it digitally

Google Research is showing off a new way to use AI to read handwriting that might radically change how machines convert what you put on paper into digital letters. The InkSight system transforms photos of handwritten words into digital text by leveraging AI without the need for any devices as intermediaries. The idea is to replace the sometimes fallible optical character recognition (OCR) with AI that can emulate how humans actually learn to read, specifically by rewriting existing text to learn what whole words look like and mean. Doing so required the researchers to tutor the AI in both recognizing and mimicking handwriting by humans. "Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes in the vectorized form, known as digital ink. However, a substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice still favored by a vast majority," the researchers explain in their paper. "Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds." InkSight is more than just an alternative technique. It makes for more accurate results in circumstances that aren't ideal. For instance, if the photo is taken in dim light, has partially obscured text, or is on a confusing background when examined with OCR. The researchers found that humans could read 87% of the InkSight-made tracings of text. Two-thirds were good enough that people couldn't tell them from actual handwriting; you can see below how it looks when InkSight works. If you like writing things by hand, InkSight has some potential benefits. Imagine writing by hand in a paper notebook, then showing the notes to your camera to instantly make them searchable and organize them in context with previous notes on physical pages. If you're like me and have particularly messy handwriting, InkSight could help turn your chicken scratch into typewritten text that is still accurate to what you scribble. On a bigger scale, this could be a crucial tool for deciphering and converting handwritten text from across the centuries into digital form. Even when the text is in a language without much of a digital presence, InkSight could help preserve handwriting to help build up training sources for those languages. Google isn't the only place where AI tools to decipher handwriting are underway. For example, Amazon's new Kindle Scribe upgrades the e-reader's ability to transform handwritten notes into legible text. There's also Goodnotes, a digital notetaking app that can read handwriting, and recently debuted handwriting editing tools using its Goodnotes Smart Ink technology to turn handwriting into typed text. The added tools let you edit handwritten notes as if they were typed, including aligning notes, copying and pasting, and reflowing the text to make it more logical.