TIME Innovation

Microsoft Is Getting Close to Perfecting a Universal Communicator

Some 40,000 people are using software program Skype Translator in hopes of achieving real-time translation

Gurdeep Pall was confident Skype’s automatic translation program would work. But as Microsoft’s corporate vice president in charge of Skype prepared to hold the first public demonstration of the program last May, Pall found himself worrying about the room itself. “Any sound that goes into the microphone, you basically have logic running trying to figure out what the sound said,” he says. “You can have feedback or you can have somebody coughing faraway that the mic picked up, somebody shifting far away, the squeak from their foot.”

Pall’s anxiety was for naught. An audience of several hundred reporters and industry insiders watched on as Pall and a native German speaker held a nearly flawless conversation through the company’s prototype of Skype Translator. Roughly a second after Pall Spoke, subtitles in German and English appeared at the bottom of the screen, and a synthetic Siri-like voice read the words aloud to the German caller. The audience murmured in astonishment, but the program didn’t falter as it shot back a translation from German to English. Pall, on the other hand, was flustered as his jitters about the room metastasized to two presenters who were whispering to one another nearby throughout the demonstration. “I’m thinking, ‘Get out of here!’” Pall recalls, laughing.

Researchers working on automatic translation technology like this are familiar with this blend of hope and anxiety. The concept of a universal translator has long been a fixture of science fiction, not to mention a dream of inventors and linguists since long before computers existed. The granodiorite slab announcing the kingly reign of Ptolemy V in Egypt circa 196 BC, better known as the Rosetta Stone, might be considered an early stab at the idea. In the 1930s, two inventors filed patents for “mechanical dictionaries” promising to translate words in real time. And in more recent decades, firms ranging from NEC to Jibbigo have periodically tried to crack the problem. But as practical reality, the idea has been perennially delayed.

Now, advances in so-called machine learning—computer programs that can essentially self-teach with enough exposure to spoken language—hope for a universal translator is increasingly replacing anxiety. What has changed from previous generations is that the underlying technology thrives through use, trial and error, recorded and reviewed, ad nasueam. The current crop of translation software gets smarter, researchers and programmers say, the more it absorbs. “The more data you have, the better you’re going to do,” explains Lane Schwartz, a linguistics professor at the University of Illinois.

Which is why Microsoft released a preview version of Skype Translator to a limited number of users last December. (The Redmond, Washington-based tech giant bought Skype for $8.5 billion in 2011.) The program is expected to reach a major milestone near the end of March. Late last year, Google announced its translation app for text would include a “conversation mode” for the spoken word. Baidu, the so-called Google of China, has had a similar feature available in its home market for several years. And the forthcoming release of the Apple Watch, a powerful computer with echoes of Dick Tracey’s famous wrist wear, has some speculating that near-instant translation might be the nascent wearables market’s killer app.

That leaves a handful of search giants—Microsoft, Google and Baidu—racing to fine-tune the technology. Andrew Ng, Baidu’s chief scientist likens what’s coming next to the space race. “It doesn’t work if you have a giant engine and only a little fuel,” he says. “It doesn’t work if you have a lot of fuel and a small engine.” The few companies that can combine the two, however, may blast ahead.

So Many Fails

There’s no shortage of false summits in the history of translation. Cold War footage from 1954 captured one of the earliest machine translators in action. One of the lead researchers predicted that legions of these machines might be used to monitor the entirety of Soviet communications “within perhaps 5 years.” The demonstration helped generate a surge of government funding, totalling $3 million in 1958, or $24 million in present-day dollars.

But by the 1960s, the bubble had burst. The government convened a panel of scientific experts to survey the quality of machine translations. They returned with an unsparing critique. Early translations were “deceptively encouraging,” the Automatic Language Processing Advisory Committee wrote in a 1966 report. Automatic translation, the panel concluded, “serves no useful purpose without postediting, and that with postediting the overall process is slow and probably uneconomical.”

Funding for machine translation was drastically curtailed in the wake of the report. It would be the first of several boom and bust cycles to buffet the research community. To this day, researchers are loath to predict how far they can advance the field. “There is no magic,” says Chris Wendt, who has been working on machine translation at Microsoft Research for nearly a decade. But he admits that the latest improvements resulting from artificial intelligence can, at times, be mystifying. “There are things that you don’t have an explanation for why it works,” he says.

Wendt works out of Building 99, Microsoft’s research hub on the western edge of its Redmond campus. The building’s central atrium is wrapped by four floors of glass-walled conference rooms, where Microsoft engineers and researchers can be seen working on pretty much any project they please. The open-ended aspect of their work is a point of pride enshrined in the lab’s mission statement. “It states, first and foremost, that our goal as an institution is to move the state of the art forward,” said Rick Rashid in 2011, twenty years after he launched the lab, according to a Microsoft blog post celebrating the milestone. “It doesn’t matter what part of the state of the art we’re moving forward, and it doesn’t say anything in that first part of the mission statement about Microsoft.”

In other words, if Microsoft’s researchers want to tinker with strange and unproven technologies, say motion-sensing cameras or holographic projectors, nobody is likely to stop them. In the mid-2000’s, there were few technologies quite as strange and unproven as “deep neural networks,” algorithms that can parse through millions of spoken words and spot the underlying sound patterns. Say, “pig,” for instance, and the algorithm will identify the unique sound curve of the letter “p.” Expose it to more “p” words and the shape of that curve becomes more refined. Before long, the algorithm can detect a “p” sound across multiple languages, and exposure to those languages further attunes its senses. “P” words in German (prozent) improves its detection of “p” words in English (percent).

Those same lessons, it turns out, apply to volume, pitch or accents. A lilt at the end of the sentence may indicate that the speaker has asked a question. It may also indicate that the speaker talks like a Valley girl. Expose the deep learning algorithm to a range of voices, however, and it may begin to notice the difference. This profusion of voices, which used to overwhelm supercomputers, now improves their performance. “Add training data that is not perfect, like people speaking in a French accent, and it does not degrade overall quality for people speaking without a French accent,” says Wendt.

The results of deep neural network research in language applications stunned Microsoft’s research team in 2011. Error rates in transcription, for instance, plummeted by 50%—from one out of every four words to one out of eight. Until then, the misunderstood word was one of the most persistent and insurmountable obstacles to machine translation. “The system cannot recover from that because it takes that word at face value and translates it,” explains Wendt. “Employing deep learning on the speech recognition part brought the error rate low enough to attempt translation.”

Speaking into Skype Translator, the commercial face of all of Microsoft’s linguistic research, shows how far things have come. The sound of your voice zips into Microsoft’s cloud of servers, where it is parsed by a panoply of software developed by the company. The team that developed those green squiggly lines under grammatical errors in Word documents laid the groundwork for automatic punctuation, for example. The team that created Microsoft’s translation app, which is currently used to translate posts on Facebook and Yelp, provided the engine for text translation. The team that developed the voice for Cortana, Microsoft’s voice-activated personal assistant similar to Apple’s Siri, helped develop the voice for Skype.

When Microsoft’s researchers debuted a prototype of Skype Translator at the company’s version of an annual science fair, they enclosed it in a cardboard telephone booth, modeled after the time-traveling machine from the Dr. Who television series. Co-founder Bill Gates stepped inside and phoned a Spanish speaker in Argentina. The speaker had been warned that when the caller said, “Hi, it’s Bill Gates,” it wasn’t a joke. It really would be Bill Gates. What did Gates say? Pretty much what everyone says at first, according to the team: “Hi. How are you? Where are you?”

My Turn

I posed the same questions to Karin, a professional translator hired for a hands-on demonstration at Microsoft’s Building 99. She answered in Spanish, and paused as Skype’s digital interpreter read a translated reply: “Hello, nice to meet you. Now I’m in Slovakia.”

The program has the basic niceties of conversation down cold, and for a moment, the Star Trek fantasy of a “universal translator” seemed tantalizingly within reach. But then a few hiccups emerged as the conversation progressed. Her reason for visiting New York was intelligible, but awkwardly phrased: “I want to meet all of New York City and I want to attach it with a concert of a group I like,” from which I gathered that she wanted to see a concert during her visit. I asked her if the program often faltered in her experience. “In the beginning,” came the translated reply, “but each time it gets better. It’s like one child first. There were things not translated, but now he’s a teenager and knows a lot of words.”

With some 40,000 people signed up to use Skype Translator, it has been getting a crash course in the art of conversation, and those words could work wonders on its error rates. An odd quirk of machine translation systems is that they tend to excel at translating European Union parliamentary proceedings. For a long time the EU produced some of the best training data out there: a raft of speeches professionally translated into dozens of languages.

But Microsoft is rapidly accumulating its own record of casual conversations. Users of the preview version are informed that their utterances may be recorded and stored in an anonymous, shuffled pile that makes it impossible to trace the words back to their source, Microsoft stresses. The team expects the error rate to drop continuously as Skype Translator absorbs slang, proper names and idioms into its system. Few companies can tap such a massive corpus of spoken words. “Microsoft is in a good position,” says Wendt. “Google is also in a good position. Then there’s a big gap between us and everyone else.”

For now, the Skype team is focused on adding users and driving down error rates, with the long-run goal of releasing instant translation as a standard feature for Skype’s 300 milllion users. “Translation is something we believe ought to be available to everybody for free,” says Pall.

That raises an awkward question for professional translators like Karin. “Do you feel threatened by Skype Translator,” I asked her through the program. “Not yet,” was her translated reply, read aloud by her fast-developing, free digital rival.

Read next: Here’s Why Microsoft Is Giving Pirates the Next Windows for Free

Listen to the most important stories of the day.

TIME Education

Mistakes to Avoid When Learning a Foreign Language

English spanish dictionary
Getty Images

If you're serious about learning a language, you need to start looking for excuses to use it

Answer by Judith Meyer on Quora.

Thinking that going to a class will teach them the language

The teacher can only present material. Good teachers will present the material in an interesting way and bad teachers will present the material in a boring way, but you still have to internalize the material afterwards. There is not nearly enough class time to do the memorization in-class, and in a group class there’s not enough time for everyone to get enough practise either. You have to study and practice outside class in order to be a successful language learner.

Not following their interests

The teacher and the coursebook are a starting point, not an end-all. Don’t limit yourself to it, seek out things that interest you in your target language (or from your target culture). Don’t be afraid to think outside the box: apart from songs, movies, TV shows, books and newspapers there are also viral videos, funny advertisements, memes, blogs, forums, women’s/men’s magazines, cartoons, comic strips, video games, MMORPGs, recipes and Q & A sites like Quora. You should start looking for this kind of content immediately. Refer to translations at the beginning and then wean yourself off them.

Balking at the language being different

There will be plenty of things that work differently in your target language than they do in English. Some of them may seem (or be) illogical. Rather than having a grudge, be happy that you get to explore these new ways of expressing oneself, ways that are hidden to monolingual English speakers. If languages were a 1 to 1 translation of English, they’d be boring.

Finding excuses not to use the language, rather than excuses to use it

There are a ton of reasons not to use the language you’re learning. Maybe your level is too low. Maybe you don’t know any native speakers nearby. Maybe it’s a small language that doesn’t have much content online. Maybe it’s a dead language. These are all reasons, good reasons even, but they shouldn’t matter to you. If you’re serious about learning a language, you need to start looking for excuses to use it.

  • Greet your friends with a fresh Latin “Salvete! Quid agitis?” next time you see them. It doesn’t matter if they don’t understand. Teach them.
  • Convince your family that kung fu movies are best enjoyed in the original Chinese.
  • If researching something for history class, use the French Wikipedia in order to get an international perspective.
  • Never comment on international news without having read a local newspaper.
  • Play on the Spanish WoW servers and consider it time well spent.
  • Plan a German movie week. Invite friends.
  • Invent reasons to talk to foreigners you see around town, e. g. ask a department store worker where a product is even when you already know, welcome tourists to your city or swap information on favourite places.

If a language doesn’t excite you so much that you’ll grab at any excuse to use it, maybe you shouldn’t be studying it. Come up with 10 reasons to study the language and hang it somewhere visible.

Thinking that language learning has to be strenuous

Many people think that they can recognize effective programs by how hard and strenuous, exhausting and uncomfortable they are. By forcing yourself through them once a month, you feel like you’re really investing effort into your target language. In truth, the most effective programs are often the ones that don’t take particular fortitude to slug through. Learning grammar is rarely joyful, but at least it shouldn’t be something you dread. A positive state of mind will help a lot in making things memorable. Don’t hesitate to switch textbooks or to use multiple sources at once. Use interesting, authentic materials as soon as you can. Use the language.

This question originally appeared on Quora: What are the common mistakes of language learners?

TIME Ideas hosts the world's leading voices, providing commentary and expertise on the most compelling events in news, society, and culture. We welcome outside contributions. To submit a piece, email ideas@time.com.

TIME

Hear Guardians of the Galaxy’s Vin Diesel Say ‘I Am Groot’ in 15 Languages

He may wear it thin, but at least he switches it up

Vin Diesel demonstrated his proficiency as a polyglot back in July, when Marvel released a video of his Guardians of the Galaxy character Groot delivering his signature line in five languages. Now, in time for the movie’s DVD release on Dec. 9, he adds another 10 languages to his repertoire.

In this featurette released by Marvel, the baritone-voiced walking tree says “I am Groot” in Hindi, Kazakh and Hungarian, just to name just a few tongues. Marvel had Diesel learn the line in different languages, rather than hiring fluent voice actors, “to uniquely customize the moviegoing experience for international audiences.”

He may only say three words, but it’s safe to say he’s mastered them.

TIME Research

Speaking More Than One Language Could Sharpen Your Brain

Skip the sudoku, try learning French

Speaking more than one language does the brain some good.

A recent study found that bilingual speakers may actually process information more efficiently than single-language speakers. Researchers from Northwestern University, in Illinois, and the University of Houston used brain imaging to look at bilingual people’s comprehension abilities. They found that people who speak more than one language are comparatively better at filtering out unnecessary words than monolinguals, whose brains showed that they had to work harder to complete the same mental tasks.

The study, published in the journal Brain and Language, used functional magnetic resonance imaging (fMRI) to look at what’s called coactivation and inhibition in the brain. Coactivation is the ability to have both languages simultaneously active in the brain, while inhibition is that ability to select a correct language while hearing more than one at a time. The researchers studied 17 Spanish-English bilinguals and 18 monolinguals, and had them undergo tests that assessed their brains’ ability to eliminate irrelevant words.

For example, in one task the participants heard the word cloud and were then immediately shown four pictures. One of the photos was of a cloud, and another was a similar-sounding word like clown. The goal was to watch how quickly the brain could make connections to the correct word. Bilinguals were consistently better at the task.

The results create a bit of a chicken-or-the-egg scenario. Is a bilingual person better at such tasks because of their expertise in both languages, or are people with greater comprehension capacity better equipped to master multiple languages? It could be a mixture of the two. The researchers of the new study believe that being bilingual is a constant brain exercise. So instead of tackling a puzzle, why not give a new language a shot, if not solely for the brain challenge.

Read next: The Secret to Learning a Foreign Language as an Adult

TIME Brain

Want to Learn a Language? Don’t Try So Hard

If at first you don't succeed, trying again might not help you when it comes to learning languages.

A new study from MIT shows that trying harder can actually make some aspects of learning a new language more difficult. While researchers have known that adults have a harder time with new languages than children do, the latest findings, published in the journal PLOS ONE, suggest that adults’ stronger cognitive abilities may actually trip them up.

Children have a “sensitive period” for learning language that lasts until puberty, and during these years, certain parts of the brain are more developed than others. For example, they are adept at procedural memory, which study author Amy Finn, a postdoc at MIT’s McGovern Institute for Brain Research, describes as the “memory system we get for free.” It’s involved in tasks we learn unconsciously such as riding a bike, dancing, or subtle language rules. It’s a system that learns from observing and from experience; neural circuits in the brain build a set of rules for constructing words and sentences by absorbing and analyzing information—like sounds—from the world around them.

“The procedural memory is already in place for an infant and working well, and not interacting with other brain functions,” says Finn. However, as people age, another memory system that is less based on exploratory processes starts to mature, and control the language learning process. “As an adult, you have really useful late-developing memory systems that take over and do everything.”

In essence, adults may over-analyze new language rules or sounds and try to make them fit into some understandable and coherent pattern that makes sense to them. But a new language may involve grammar rules that aren’t so easily explained, and adults have more difficulty overcoming those obstacles than children, who simply absorb the rules or exceptions and learn from them. That’s especially true with pronunciation, since the way we make sounds is something that is established early in life, and becomes second nature.

“Adults are much better at picking up things that are going to immediately help them like words and things that will help them navigate a supermarket,” says Finn. “You can learn language functionally as an adult, but you’ll never sound like a native speaker.”

So how can adults remove their own roadblocks to learning new languages? Finn says more research needs to be done to determine if adults can ever go back to learning languages like children, but linguists are looking at a variety of options. A few include “turning off” certain areas of the brain using a drug or a technique called transcranial magnetic stimulation, which might allow adults to be more open to accepting new language rules and sounds.

Finn also hopes to study adults performing a challenging task while they learn a language, which is another way of distracting the cognitive portions of the brain from focusing on the new language, to see if that can help them to absorb more linguistic information.

TIME

Being Bilingual Keeps You Sharper As You Get Older

Picking up a second language, even later in life, can have benefits for the brain

People who speak more than one language tend to score higher on memory and other cognitive function tests as they get older, but researchers haven’t been able to credit bilingualism as the definitive reason for their sharper intellects. It wasn’t clear, for example, whether people who spoke multiple languages have higher childhood intelligence, or whether they share some other characteristics, such as higher education overall, that could explain their higher scores.

Now, scientists think they can say with more certainty that speaking a second language may indeed help to improve memory and other intellectual skills later in life. Working with a unique population of 853 people born in 1936 who were tested and followed until 2008-2010, when they were in their 70s, researchers found that those who picked up a second language, whether during childhood or as adults, were more likely to score higher on general intelligence, reading and verbal abilities than those who spoke one language their entire lives. Because the participants, all of whom were born and lived near Edinburgh, Scotland, took aptitude tests when they were 11, the investigators could see that the effect held true even after they accounted for the volunteers’ starting levels of intelligence.

Reporting in the Annals of Neurology, they say that those who began with higher intellect scores did show more benefit from being bilingual, but the improvements were significant for all of the participants. That’s because, the authors suspect, learning a second language activates neurons in the frontal or executive functions of the brain that are generally responsible for skills such as reasoning, planning and organizing information.

Even more encouraging, not all of the bilingual people were necessarily fluent in their second language. All they needed was enough vocabulary and grammar skills in order to communicate on a basic level. So it’s never too late to learn another language – and you’ll be sharper for it later in life.

TIME

What Does English Sound Like To Foreigners?

Ever wondered What does English sound like to foreigners?

Adriano Celentano’s “Prisencolinensinanciusol” is pure gibberish. The words don’t mean anything.

But the whole song was designed to sound like English — the way English sounds like to people who don’t speak it.

Ever done a gibberish imitation of a language you don’t speak?

Well, this is how non-English speakers mock you.

More of his work is here.

Join 45K+ subscribers. Get a free weekly update via email here.

Related posts:

What 10 things should you do every day to improve your life?

What five things can make sure you never stop growing and learning?

What do people regret the most before they die?

This piece originally appeared on Barking Up the Wrong Tree.

TIME Languages

Here’s How to Say ‘Hello’ In 21 Different Languages

From Croatian to Korean to Swahili to Swedish

Sick of greeting people with a boring old “hello” or “hey or “howdy there?” This infographic from LivingLanguage.com offers 21 other ways to say it. We definitely recommend busting these out to seem more worldly and cultured.

21 Ways to Say Hello

TIME Languages

Chineasy: A New Way to Learn Chinese Characters

The goal of my new book Chineasy is to allow people to learn to read Chinese easily by recognizing characters through simple illustrations and animations. By learning one small set of building blocks, students can build many new words, characters, phrases, and even sentences.

In the past year, I broke down thousands of common Chinese characters and analyzed how they are constructed. The process is just like a little boy breaking down lots of Lego models (a fire station, a boat, and maybe a space shuttle). This boy then starts classifying Lego bricks. He then realized that no matter how fancy and complicated the models are, they are all built out of the same set of Lego bricks.

In Chinese, most characters are built out of roughly 100 “building blocks.” I designed a program on my computer and then started prioritizing the most common and useful building blocks. In my computer program, I identified the correlations between each character until my screen was full of thousands of lines and tiny characters.

I then started using illustrations to help students memorize the building blocks. After knowing how to recognize those “building blocks” you can then carry on constructing loads more characters. I called those newly constructed characters “compounds,” meaning the characters which are composed of two or more building blocks.

Here are three examples: king, water, and cow.

  • King 王

    ShaoLan Hsueh—Chineasy/HarperCollins

    King is a very common surname in China. It is the Chinese equivalent of the name ‘Smith’ in Europe. Though we have illustrated King as a face, that character actually represents the universe: the three horizontal strokes represent Heaven, Man and Earth. The vertical stroke is the king who connects them.

  • King unillustrated 王

    ShaoLan Hsueh—Chineasy/HarperCollins

    The character for King 王 is often confused with the character for Jade 玉. As you can see both are very similar, except for the small line in the bottom right hand corner of the character. This confusion is compounded by the fact that in Simplified Chinese King 王 not Jade 玉 is a radical whereas in Traditional Chinese it is the other way around. Why is this? Well the Chinese language like all languages has been through 1000s of years of evolution. In China Jade is so highly valued that it was often associated with the King so it is unsurprising that the character was used interchangeably.

  • Water 水

    ShaoLan Hsueh—Chineasy/HarperCollins

    We all know what water is, we drink it every day, but it is important to realize that in Chinese, words can have many different meanings and uses especially when they are paired into phrases. In Chinese this character doesn’t only refer to water, it can also be used with other characters to refer to the qualities of water – clear, pure and liquid. When this character is used as part of a compound it transforms to look like this 氵 this character is known as ‘Three drops.’ A fun little phrase you can build with two of these characters is Buffalo 水牛 which is a combination of water 水 and cow 牛.

  • Water unillustrated 水

    ShaoLan Hsueh—Chineasy/HarperCollins

    Water is the source of all life. I think this is a beautifully fluid character that really represents its meaning. I used to remember this characters as different rivers joining together flowing to the ocean. Actually the original character from thousands of years ago actually depicted a river between two banks, but as with most Chinese characters the shape has changed considerably over the course of Chinese history.

  • Cow 牛

    ShaoLan Hsueh—Chineasy/HarperCollins

    Cow is a very common character in Chinese. It can be used to refer to bovine as a whole or be used as part of a phrase to be more specific. On top of this, you also see it on many menus (for beef!) and in any reference to prisons. “Why Prison?!” you might ask? Well, In ancient China cows were the main beasts of burden. At night they were kept inside to protect them from them predators, but the Chinese saw this as similar to indentured labor. So, a cow under a roof, which looks like this 牢 means ‘prison’ or ‘pen.’

  • Cow Unillustrated 牛

    ShaoLan Hsueh—Chineasy/HarperCollins

    The original character for cow from the Seal Script era looked like a cow’s heads with two horns. Over time the cow seems to have lost one of its horns, which is a shame. As with water, this character can also be used as an adjective which refers to the qualities of cows. When used in this format the character can mean ‘arrogant’ and ‘stubborn’. I have never known an arrogant cow, but stubborn seems apt.

Your browser is out of date. Please update your browser at http://update.microsoft.com