How Facebook Knows What You Look Like

That woman from that laundry detergent commercial, you’ve seen her before in the yogurt ad. And the clerk at the convenience store you hit up this morning, that’s right — he was the guy hitting on you at the bar last week.

Making split-second connections like these is something your brain does well. In fact, according to 2008 research out of the University of California San Diego, it only takes two quick fixations before we recognize the person we’re gazing at.

But when it comes to facial recognition, computers have to put in a lot more work to get the answer right. “Recognizing people’s faces is really easy for people to do just instinctively, but it’s actually really hard for a computer,” says Rob Sherman, Facebook’s deputy chief privacy officer. Facebook uses facial recognition technology to drive its “Tag Suggestions” feature, which helps the website ID you when a picture of you gets uploaded to the social network. And while the feature may seem to work at the blink of an eye, there’s a lot more going on behind the scenes.

First, Facebook’s computers analyze your profile photo and any other picture in which you’ve been manually tagged to find similarities between the images. In general, they’re looking at the dimensions of your face, including its shape, the distance between your eyes, and how far away your facial features are from one another. The site may also scan for the color of your hair.

“Essentially what we’re looking for is the unique characteristics of this particular person,” says Sherman.

Next, once you upload a photo to Facebook, the system figures out if there’s a face in the photo. If there are no faces, then it doesn’t make sense for Facebook to run facial recognition technology. But it’s also important to note that Tag Suggestions doesn’t get deployed for just any image of people uploaded to the social network. Instead, it’s only offered when the uploader and the photo’s subject(s) are already friends — doppelgängers on the other side of the world remain a mystery.

If faces are detected in your picture, Tag Suggestions whirs to life, analyzing the new image and looking specifically for your distinct characteristics, or those of your friends. “The goal is to figure out what are the things that differentiate one person’s face from another person’s face,” says Sherman. And if the feature does detect you or your friends, Facebook highlights the face and suggests to tag the image appropriately.

According to Sherman, Facebook’s technology was designed to streamline the photo tagging experience. The idea is that if you’re already friends with someone on Facebook, then you know what they look like (and they know what you look like, too). Tag Suggestions and facial recognition just makes it quicker and easier to tag them in a picture.

But not everyone sees this technology as a matter of convenience — some see it as an invasion of privacy. For instance, in Europe, concerns over facial recognition technology has delayed the rollout of Moments, Facebook’s new photo-sharing app. Moments uses the same facial recognition technology that powers Tag Suggestions, but it’s designed to help Facebook users share photos with each other. For instance, say you’re on vacation with your friends and you all take photos with your smartphones. Moments uses facial recognition to automatically share the group members’ photos with one another. That eliminates the hassle of emailing or texting images to one another after an event is over.

If all this talk of facial scanning and recognition has you hiding from camera lenses, there’s an easier way to opt out. “If people don’t want it, obviously we want to make it easy for them to turn it off,” says Sherman. To disable Tag Suggestions on your account, log into the social network and go to the Timeline and Tagging section of your settings, where you’ll see the last question: “Who sees Tag Suggestions when photos that look like you are uploaded?” Click on the “Edit” link to the right, and change the dropdown menu from “Friends” to “No One.” Once you do that, not only will the feature be disabled, but the social network will remove your record from Facebook’s database of similarities. This will also prevent your profile from being suggested to friends when they upload images to the site.

But still, Sherman says, Facebook’s records only work on its website. “Even though we are deleting it, it’s not really useful in any other system, and that’s one of the important points,” he says in the feature’s defense. And besides, being tagged in an image isn’t necessarily a bad thing.

“Tagging is a really important way to communicate on Facebook and interact with one another,” says Sherman. When you’re tagged in photos, you know those photos exist, and you can be alerted to conversations happening about them. Otherwise, people could be talking about you behind your back — or, more accurately, behind your face.