This Technology Could Change the Way Deaf People Live

Ryan Hait-Campbell says his San Francisco company’s invention is really about jobs. Deaf people like himself, explains the MotionSavvy CEO, are too often shunted into positions that don’t require talking to anyone—washing dishes, fishing or other solitary vocations that often have low wages, little opportunity for advancement and no need for an employer to hire an interpreter. One study found that only 58% of working-age Americans with a severe hearing impairment have a job at all.

MotionSavvy’s first product, though still in prototype stage, could revolutionize the prospects of millions who are deaf or hard of hearing. Called Uni, the device clasps around a PC tablet and uses MotionSavvy software to act as an interpreter between a signer (who can’t speak) and speaker (who can’t understand sign language) in very-close-to-real-time.

Two cameras read and project images of a deaf person’s gestures into a 3D virtual space. Uni’s software interprets those movements into English words that are spoken for them in a Siri-like voice. Then, when a speaker responds in words, the program uses voice recognition to display those sounds as text.

Here’s what the screen looks like:

You can also watch a short video showing how it works on the company website.

The current options a deaf person has to communicate with people who don’t understand sign language are often expensive, cumbersome and leaving the signer at the mercy of an intermediary’s interpretation. They can hire an interpreter, either in person or through video relay services like FaceTime, paying rates that could be $50 an hour. Or they can use some equivalent of writing their words on a piece of paper and handing it to someone, who then writes their response on the paper and hands it back—whether that’s on actual paper or an app.

MotionSavvy’s chief design officer Jordan Stemper—one of eight hearing impaired MotionSavvy employees besides Hait-Campbell—says that nuance is often lost through interpreters, and points out that deaf people have been in situations where none of the available options suffice. Banks, for instance, have refused to allow deaf customers to call them using relay services because of privacy concerns (and have been sued for it), meaning any banking they want to do has to be done in person.

The key piece of technology in Uni is what MotionSavvy calls its “sign builder,” a system that can record gestures (made over and over and over again to account for variation among signers) and assign them English words. Right now, Uni can understand just 300 words and the alphabet. But Hait-Campbell says that the company plans to recruit about 200 beta testers this coming spring who will both try out the device and add needed signs, putting their lexicon at over 15,000 by fall 2015, when pre-orders are set to ship. The devices will also adjust to a user’s particular movements over time through machine learning, Hait-Campbell says. And if someone wants to add a non-standard sign for slang like “ridonculous,” they can.

The beta testers will be drawn from people who pre-order Uni through MotionSavvy’s Indiegogo campaign, a crowdfunding effort started this week that will determine how many devices can be shipped in fall 2015 and whether the products remain at their $499 price point, which Hait-Campbell says has caused sticker-shock among some in the deaf community. The MotionSavvy team wants to put the device—one they hope to eventually shrink to a mobile phone case and perhaps even an app—in as many hands as possible, and may consider cheaper subscription models to do so.

“I do not consider being deaf a handicap, but in reality it is,” Hait-Campbell writes to TIME. “There’s not been any real innovation for those deaf who cannot speak . . . Most deaf people, if they have jobs, have jobs that require little communication, like grunt work jobs. And it sucks, because the potential of these people, including my friends, can take them so far.” Most deaf people he knows are living on Social Security, he says, getting by month-to-month on what might be $500 checks.

The National Association of the Deaf does not endorse products, but spokesperson Lizzie Sorkin says the group is aware of Uni and sees it as “promising technology.” She also hints at some current limitations, like the fact that sign language is often conveyed through entire body movements, not just the fingers and forearms that show up on Uni’s screen. Hait-Campbell says later versions of the product will account for a wider range of motion, including facial expressions.

The app’s development will likely be of interest to far more people than the hearing impaired. Hait-Campbell says his company has already been approached by players in other industries who are interested in the technology, like defense contractors who want their software for controlling drones through gestures, as well as home automation companies. For now, he says, MotionSavvy has tunnel vision. “We want to focus on making this the best we can for the deaf world,” he says. “There is nothing like this out there at all. The need for this is so great.”