Virtual Reality and Eye Tracking: Sony’s Vision of the Future

After Nintendo’s “smash hit” Wii, Sony realized that raw horsepower wasn’t necessarily the be all, end all for a video game console. Jump to the beginning of 2013, when Playstation formed Magic Lab: a special R&D arm at PlayStation tasked with dreaming up the next generation of gaming experiences.

“We had the concept in 2012 for this group which would use technology to really explore new experiences,” said Richard Marks, Director of PlayStation’s Magic Lab. “We really focused a lot on technology in the past, and [now] we really want to focus more on the new experiences that technology enables. One of the things that we believe strongly in is actually prototyping things; we call it: experiencing engineering.”

Since joining Sony, Richard Marks has been responsible for the development of Sony’s PlayStation Eye, PlayStation Move controllers and now Sony’s foray into virtual reality: Project Morpheus, a wraparound headset designed to work with the company’s PlayStation 4 games console. The headset’s revelation came in tandem with Facebook’s high-stakes maneuver to put virtual reality on the map for non-gamers per its recent $2 billion acquisition of Oculus Rift.

So far, Sony’s touted Morpheus to game designers at shows like the Game Developers Conference and E3 to drum up development interest (though the technology’s still far from a commercial product — it currently has no release date). Morpheus’ display still has a few issues, too: there’s a stutter effect on some demonstrations caused by high latency, and Morpheus’ Field of View (FOV) doesn’t cover everyone’s vision completely. The technology also has various critics predicting that it lacks a mass market appeal. But then again Morpheus is only a prototype, as are the various Oculus Rift iterations.

“There’s a trade-off. There’s a fixed amount of resolution. So you can either give that to a really wide field of view or you can make [the resolution] feel higher, but the [field of view] narrower. We’re trying to get a good balance of that. Right now we’re still working on the issue of the display. Right now we have a great prototype system for our developers … [but] for the commercial system, we’re still working on that.”

And as a prototype, Project Morpheus is an amazing portal into what VR could look like for the future. The technology is so electrifying that creators and entrepreneurs outside the games industry are seeing VR’s potential, which is why Facebook put up the cash in the first place. For instance, currently Sir David Attenborough is creating a VR nature documentary, companies are looking into how VR can impact education and Hollywood is looking into virtual reality movies.

Whether VR will succeed as a mass market product or not remains to be seen; in the meantime, PlayStation’s Magic Lab is tinkering with its notion of what the future of gaming might look like.

During the launch of the PlayStation 4 in November 2013, Marks and fellow Magic Lab researcher Eric Larsen were demoing their eye tracking or “gaze tracking” technology. “A lot of different people are looking at how to track your eyes. Our focus is more on, if you can track your eyes, what do you do with it?” Marks said.

The technology has a lot of potential applications, like as a targeting assistant for shooter games, as Marks and Larsen demonstrated with the game Infamous: Second Son. One of the more interesting applications Marks noted is the ability to pick up on subtle, non-verbal communication cues. “Where someone is looking conveys a lot of information about what the person is interested in, what they intend to do, and it’s a very unconscious thing that people do,” Larsen said.

In the demonstration, the player interacts with a computer store merchant who’s trying to sell the player different products. The eye gazing technology detects what products the player is looking at and uses that information to decide what products to pitch the player. “You can make the characters smarter because they kind of react in a way that is more intelligent because they know what you’re looking at,” Marks adds.

On top of that, Magic Lab is also looking into biometrics, partnering with UC San Francisco to research brain waves as a feedback mechanism for how a game affects players.

Magic Lab, like Google X — responsible for the creation of Google Glass and Google’s Driverless Car — seems to be Sony’s take on “experiencing engineering” without the red-tape. Whether Magic Lab will create products with the same hype factor as Google X’s ideas is anyone’s guess, but if Morpheus is any indication, Marks and his team are off to a promising start.

Products Richard Marks Has Worked On

MORE: What Gaming Industry Professionals Think of Virtual Reality:

MORE: The History of Video Game Consoles – Full