How Video Game Makers Create Hyper-Realistic Worlds

Tommy Francois creates worlds. Virtual worlds, where gamers can do most of what they can’t do in real life. If you’ve ever wondered how today’s games look so realistic, Francois, the vice president of editorial at Ubisoft, is partially responsible for the photorealism in games like Assassin’s Creed, Far Cry, Watch Dogs, Tom Clancy’s The Division and most recently Ghost Recon Wildlands — and their artistic diversions from reality.

Building video game environments is a complex and delicate process, Francois tells TIME. “Realism is a dangerous word. We are making entertainment in fiction,” he says. “I often look at the worlds we build almost like onion peels. Each peel is a narrative opportunity, and every narrative opportunity is an opportunity for fiction, and characterizing either your characters, your stories, or your systems.”

The development process behind these games begins months before Ubisoft writes its first line of code. Francois hand-selects a strike team of writers, photographers, journalists, and developers to learn everything they can about the settings of a game. Some game environments are an amalgamation of multiple real-world locations, while others are directly based on specific cities or regions.

For Francois, that research is key to creating original gaming experiences. “If you’re only working from other people’s work, you become an extremely diluted version of someone else’s research process and it’s either impossible for you to innovate or to come up with your own stance on the world and create something original,” he says.

Next, a smaller team is sent on a series of weeks-long trips to gather footage and experience these places first-hand. “As much as I hate an airport, even 10 seconds in an airport is worth infinite time on Wikipedia,” he says. “The rhythm, the smell, the signs, the people, the colors, all of this stuff you just can’t get [any other way].”

For the recently released Tom Clancy’s Ghost Recon Wildlands, when the team arrived in Bolivia, they discovered that the buses have oxygen masks to help tourists cope with the high altitude. They also found that instead of coffee, residents commonly chew coca leaves, which also serves as a cure for altitude sickness. Both discoveries were new to the team and became details in the game.

The team was also able to uncover what Francois calls “the invisible parts of the iceberg,” or the understanding of the links between the systems in place in society, as they embedded themselves in the local culture and mores.

To supplement that experience, Francois’ strike teams take hundreds of thousands of pictures and hundreds of hours of video. They conduct interviews with locals and record sounds. All of that information is then entered into a digital database called the WTF, which stands for World Texture Facility. Back at the Ubisoft offices, this information is used as reference material by the game developers, with Francois working closely alongside them as they shape their worlds over the course of several years of development.

The trick to a successful final product is to seamlessly build a world while balancing the details with the game mechanics and storyline. Much is left on the cutting room floor, especially with storylines that might play it safe, relying on clichés and tropes that have worked in the past.

“Let me be super-blunt,” he says. “I think the industry kind of sucks when it comes to this. I imagine video games to be the children of all forms of entertainment. One of the things that we need to do is perhaps be less in touch with the reptilian part of our brain and be in touch with our frontal lobe — more empathy, different mechanics.”

Constraints aside, Francois is driven to create worlds that are relatable, games that help us understand the systems of the world we live in. His hope is that we, as players, can dive into his creations with an intensity we might not be able to experience in person.

New and emerging technologies could help him achieve these goals. For example, cloud computing could allow for complex weather patterns. It might also let him build his ultimate dream: games that transcend the traditional protagonist/antagonist model, similarly to what shows like The Wire were able to achieve where the perspective can shift between any character in the created world. Accordingly, the storyline and gameplay would shift to embrace the gray areas of our world, helping us to better understand archetypes of society with which we may be unfamiliar. Francois’s hope is, through that process, players will develop a sense of empathy and understanding that they will take with them into the real world.

Josh Raab is a multimedia editor at TIME. Follow him on Instagram and Twitter.