Apple's guiding voice(over) helps the blind create software
Kids and adults with visual impairments can use Apple's Swift Playgrounds to learn programming on their iPads.
Dean Hudson, a quality and assurance engineer on Apple's accessibility design and quality team, makes sure the tech giant's products, including Swift Playgrounds, can be used by everyone.
This story is part of CNET's "Tech Enabled" series about the role technology plays in helping the disability community.
When Dean Hudson was 12 years old, he'd bribe his brothers and sisters to read computer manuals to him.
Hudson's vision was muddled by glaucoma, and he wanted to learn how to code on the computer his dad brought home.
"At that age, I'd do whatever it took," said Hudson, whose vision at that time meant he had to hold his face as close to the computer monitor as possible to see anything. "I knew I was going to be a hardware engineer or software or something. I liked creating things."
Nearly 40 years later, Hudson -- who went blind at the age of 20 -- now works as a quality and assurance engineer on the accessibility design and quality team at Apple. The group makes sure all of the company's hardware, software and services are accessible to everyone.
A key part of making everything accessible is VoiceOver, a screen-reading technology that speaks descriptions aloud to users. For example, it plays a key role in Swift Playgrounds, Apple's iPad app that teaches kids (and adults) how to code. Playgrounds gives basic programming instructions for making animated characters do things like collect gems, and it's based on the Swift programming language designed to make it easier for anyone to build apps for the iPhone or iPad.
Apple's Swift Playgrounds app features VoiceOver technology that lets vision impaired people use it.
That's particularly relevant this week, when Apple holds workshops as part of the Hour of Code, a global movement to promote computer science to students.
Daniel Montejano, an 8-year-old fourth grader, has always liked puzzles. He started coding on his own a couple of years ago, but he hadn't used Apple's coding software before this year's Hour of Code. He took two different Swift Playgrounds sessions at the Apple Store in San Francisco's Union Square to learn more about how the software works and hopefully score a job in tech when he's older.
"I want to make an app kind of like Xoom [the money transfer app] to help people send money faster from anywhere in the world," said Montejano, who has no visual impairments. "And let people change from American to any other kind of money."
Making Swift Playgrounds more accessible is key to giving all children the tools to succeed in technology.
"One of the roadblocks that often comes up for kids who are blind is STEM [science, technology, engineering and mathematics] education, and students may reach a point where people say, 'As someone who's blind, engineering may not be the route for you,'" said Sarah Herrlinger, Apple's senior manager for global accessibility policy and initiatives.
Apple sees things differently.
Daniel Montejano concentrates on the Swift Playgrounds coding app during an Hour of Code class at the San Francisco Union Square Apple Store.
"If we're talking about coding, how do we make sure students who are blind have opportunities to learn?" she said. "Swift Playgrounds is an example of how accessibility permeates all of the many facets of what we do."
More than 130,000 apps in the App Store have been created using Swift, including apps from Disney, NBA, Strava and Uber. (There were about 2 million apps in the App Store as of June.)
Giving voice to the blind
Accessibility, which has been a big focus for Apple for several years, has gotten a more public push of late. CEO Tim Cook kicked off Apple's MacBook Pro launch in late October with a video about accessibility and unveiled the company's new accessibility website a day after Microsoft kicked off an event with its own accessibility-focused video.
Along with narrating Swift Playgrounds, VoiceOver also lets blind people take photos or navigate a city's public transportation system on their own. Apple built VoiceOver into its iOS mobile software in 2009 and since then has incorporated other free features that people previously had to purchase.
People with visual impairments once had to shell out thousands of dollars for technology that magnified their computer screens, spoke navigation directions, identified their money or recognized the color of their clothes. Hudson recalled having to work with human readers when he was coding in college and how they'd sometimes fall asleep on the job.
Dean Hudson is one of the Apple engineers who makes sure Apple's software makes sense for seeing impaired users.
Now most of those capabilities are packed into the iPhone.
"When I first came [to Apple in 2006], I would go to a store, look at some things on the table and say, 'Nope, I can't use that,'" Hudson said. "Now I can use all Apple devices in the store. It's really powerful."
VoiceOver works by turning off the iPad's single-tap function on the display. After that, users can move their fingers across the screen to hear what's on the display. In the case of Swift Playgrounds, Hudson and other engineers made sure the instructions were direct enough to explain the highly visual software to someone who can't see it.
Custom swiping gestures in VoiceOver mode tell users an animated character's location (such as, "You're in a three-dimensional grid, you're on the third row, second column and are facing left") and the location of gems on the screen. If there's an error in the code, users can swipe down and review the different actions they can take.
Apple's team also took pains to make the animations fun for people who are hearing the descriptions read out loud. For instance, if the user hasn't completed the puzzle, VoiceOver will say, "Sad face."
"I never thought I couldn't do it, even when I lost my vision," Hudson said. "This tool, Playgrounds, provides the means for someone to learn skills and become gainfully employed. That's always, for me, the bottom line -- how do we improve the situation for people with disabilities?"
