Why handwriting recognition on the iPad isn’t a joke anymore

Egg freckles. Eat up Martha. Cafe Fwiblob.

There was a time when these words would have elicited instant giggles. Memes from the time before Twitter or Facebook, Apple’s Newton and its often-errant handwriting recognition offered a wellspring of humor for the minds behind Doonesbury and The Simpsons, a spot-on sendup of a technology that was so close yet so far.

The signature feature of the original MessagePad was indeed worthy of ridicule. While Newton OS’s handwriting recognition engine was impressive from an I’ve-never-seen-anything-like-this perspective, its real-world application was loaded with flaws and miscues, far more than iPhone OS’s autocorrect or Siri’s listening abilities. As the jokes piled up, the Newton became synonymous with silly errors, and it was never able to fully shake its reputation, no matter how much better it got.

By the time Steve Jobs killed the whole project in 1998, Newton OS—by then on version 2.1—had a very solid handwriting recognition system that read both print and script with remarkable accuracy. It wasn’t just useable, it was widely regarded as the best in the industry. But despite its advances (not to mention his fondness for skeuomorphism), Jobs never really explored the possibilities for handwriting recognition, save the underpublicized Ink feature that was announced with OS X Jaguar, and Apple has done even less to promote the technology on its multitouch devices.

But it might finally be ready for its closeup.

Writing process

If the termination of the Newton project didn’t kill the dream of handwriting recognition, the introduction of the iPhone certainly did. Not only did it herald a whole new way of typing that combined smart autocorrect with fast, responsive keys, its capacitive screen was very clearly built for finger operation.

Jobs may have publicly dissed the stylus back in 2007, but the landscape has changed quite a bit since then. Styli are no longer dumb sticks, and the Apple Pencil represents once of the biggest advancements for input devices since the trackpad. Artists, sketchers and doodlers alike are effusive in their praise for its pressure-sensing, lag-free responsiveness, but there’s a lot for writers to love, too. Writing in Notes on the iPad Pro feels just as comfortable as using a notebook, and the end result looks nearly identical to actual handwriting, with Apple Pencil capturing the rhythm, spacing and thickness of the letters with remarkable accuracy.

There’s no denying that it does a stellar job at handwriting emulation, but real recognition is not part of the Apple Pencil experience. iPad Pro users who write with their Pencils will more or less get a digital rendition of the pen-and-paper experience, but notes aren’t any more editable or searchable than they would be inside a Field Notes book or on a scrap of paper. With the Newton, the handwriting recognition engine was meant to be a replacement for the keyboard: a quicker, more natural way to input notes than tapping away at tiny buttons, but the iPad Pro is as tied to its keyboard as the MacBook is.

Pencil sharpener

While handwriting recognition isn’t baked into iOS like it was with Newton, the underlying technology certainly hasn’t languished over the past two decades. With the Newton, it was completely software driven, but modern digital ink relies as much on the instrument as it does on the implementation.

“Devices for decades have never lived up to the expectations of the end users,” said Gary Baum, vice president of marketing at MyScript, which offers an array of handwriting recognition apps for iOS and announced a partnership with Wacom at CES to power the ink-to-text technology in Bamboo Spark. “The stylus itself needs to be as good as a pen or pencil. If it doesn’t work well it’s probably the most frustrating thing.”

Nearly every tablet platform uses a proprietary writing instrument to implement some version of handwriting recognition—Microsoft’s Surface Pen, Samsung’s S-Pen, even HP has the Executive Tablet Pen for its HP EliteBook Revolve 810 G2—but until the iPad Pro, Apple frowned upon styli, smart or otherwise. But it made a great one with Apple Pencil, and if handwriting recognition is something that’s eventually planned for iOS, Apple hit all the right marks.

“We’re seeing a tremendous focus on the stylus, because it’s the last real man-machine interface that is natural and expressive and creative,” Baum said. “No one wants to carry paper anymore; you can’t search it, you can’t store it easily. It’s kind of like Siri—once you use it you never want to go back.”

Ink block

But while today’s styli are a million times better than the one that shipped with the MessagePad, the pen is not mightier than the software. What makes handwriting recognition so difficult to get right is the same thing that harpooned Newton and makes Siri so frustrating at times: the lack of understanding. It’s a problem MyScript is still working on, but it’s one that Baum believes will be solved by changing the way our devices handle our handwriting.

“Today most people who write notes either leave them in ink or retype them,” he said. “The next generation of ink management is interactive. With a couple of simple gestures I’ll be able to do anything I can do with my keyboard—insert, edit, change, delete characters. With a tap I can convert it all into something that’s fully digital and easily shared. The benefit is that you go back and forth between digital and ink.”

It all sounds like something straight out of Cupertino. To hear Baum describe the still-in-development technology makes the iPad Pro sound like the perfect device to lead the next breakthrough in handwriting recognition. In fact, over the past few months (coinciding with the release of iPad Pro), MyScript has seen twice as much interest in its proprietary developer tools, and Baum expects that trend to continue as Apple Pencil pushes the stylus back into the mainstream.

But just because Apple has the ability to take another crack at handwriting recognition doesn’t mean it will. While Baum suspects iOS will soon have “some level of handwriting capability,” it remains to be seen whether Apple will ever make it a key component of the Apple Pencil experience and risk opening itself up to a new crop of Simpsons and Doonesbury jokes.

Talk to the hand

Thomas Brand, one of the world’s last Newton users, thinks Apple’s history with handwriting recognition is precisely why we won’t be seeing a resurgence anytime soon.

“Apple probably knows better than almost any other company that handwriting recognition is a bet with long odds,” said Brand, who published his full handwriting recognition thoughts on his blog, Egg Freckles. “For over 10 years Apple worked on improving the Newton’s handwriting…Rosetta, the Newton’s print recognizer, has been considered by many reviewers, testers, and users to be the best in the industry, even 10 years after it was introduced. But still the Newton’s handwriting recognition was ridiculed with bad press every step of the way. Handwriting recognition just hasn’t been reliable enough to sell a premium product.”

But this is Apple we’re talking about, and it’s hard to believe Pencil’s full capabilities are only meant for artists. Opening up the iPad Pro to writers with a modern take on handwriting recognition would position the iPad Pro as a tool for all professionals, not just creative types. But still, Brand isn’t convinced.

“Maybe Apple will come out with an amazing new handwriting recognition system, but I doubt it,” he said. “When you are the most successful company in the world with priorities from streaming music to self-driving cars, there is very little room for the long bet. Especially when you gambled big and lost the same bet over two decades ago.”

If Apple ever does take another crack at handwriting recognition, it’s likely to be a baby step rather than a leap, perhaps as a beta feature of Notes, not unlike how Siri got its start. There’s also the education route. The iOS 9.3 beta introduced a renewed push into schools, with a dedicated app for teachers, powerful classroom management and smart sharing tools designed to position the iPad as an essential tool for students. Handwriting recognition would fit nicely into that endeavor, helping kindergarteners learn how to write and high schoolers take notes naturally and, perhaps more importantly, without talking to Siri.

“Voice is good, but handwriting is the way we’re going to move,” Baum said. “The ultimate goal is to capture anything you can write with a pen and understand it and translate and allow it to transfer into your digital workflow…We have to make it easy, and this is the technology that will do that.”