The Science Behind TIME’s New Apollo 11 Moon Landing Augmented Reality Experience

TIME this week launched TIME Immersive, a new iPhone and Android app that we’ll use to deliver groundbreaking augmented reality and virtual reality experiences. First up: the TIME Moon Landing experience, the world’s most accurate 3D re-creation of the Apollo 11 mission, which took place 50 years ago this month. Users can watch an approximately five-minute AR simulation of the Apollo 11 landing, narrated by TIME’s Jeffrey Kluger and featuring original NASA audio from the mission, then explore the surface of the moon on their own.

What makes the TIME Moon Landing hyper-accurate? At the experience’s core lies incredibly precise data meticulously collected over the last 20 years by John Knoll, the chief creative officer and visual effects supervisor at Industrial Light and Magic, a top Hollywood special effects company founded by George Lucas.

“I’m old enough to remember seeing the Apollo 11 landing live as a kid,” says Knoll, who gave his data to TIME. “That really left a big impression on me. In the years that followed, I was always fascinated with the space program.”

Knoll began collecting Apollo 11 landing data after stumbling upon a transcript of radio calls between the spacecraft and mission control. Those transcripts, he says, underscored the harrowing few minutes just before the “Eagle” lander touched down on the lunar surface, when it was running dangerously low on fuel. That moment, says Knoll, was largely glossed over in the Apollo 11 documentaries of his youth. “In reading the timestamped transcripts, this is white-knuckle time,” he says.

Knoll’s commitment to accuracy came in part from his disappointment with some Hollywood directors who pay lip service to scientific precision but abandon it in favor of what they or the studios believe is better storytelling. “I was very committed to making the re-creation as technically accurate as I could make it, in getting everything right about the motion of the spacecraft, the lighting conditions, the lunar terrain, where individual rocks and craters were,” says Knoll. “And to figure out if there were clever or sneaky ways to extract data from unlikely sources.”

To that end, Knoll relied on a handful of data sources, including NASA telemetry graphs, footage from a descent camera on the lunar module (LEM), and data from the Lunar Reconnaissance Orbiter (LRO), a probe orbiting the moon that was launched in 2009. He made up for shortcomings in the data with advanced computer vision techniques, including one process whereby the altitude of moon surface features can be estimated based on how bright or dark they appear in photographs.

“When you look at a photograph of the moon, and you see all that light and shadow, what you’re seeing is the orientation of the surface relative to the sun,” says Knoll. “If a surface is brighter, it’s because it’s inclined more towards the illuminance, and if it’s darker, it’s because it’s inclined more away. If you start on one end of an image, and if a surface is lighter than the average then it’s inclined up, so you accumulate the altitude, and if it’s darker, it’s declined, and so you decrement the altitude. By doing that, you can integrate an approximation of the terrain.”

Knoll hopes that the experience helps people better understand and take pride in the complexity of the Apollo project.

“I’m a big champion of science education, and people really understanding what we achieved,” says Knoll. “Those Apollo missions were great and amazing, and especially in these very divisive times, everyone regardless of their political affiliation can look back with some pride and look back at the accomplishment.”

The TIME Moon Landing experience was co-produced by TIME, John Knoll, the Smithsonian’s National Air and Space Museum and Smithsonian’s Digitization Program Office, Trigger, RYOT, and the Yahoo News XR Program. It is available within the TIME Immersive app, which you can download for iPhone in Apple’s App Store, or for Android in the Google Play Store. Look out for more TIME Immersive projects in the near future.