Inside Apple’s Plan to Give You a New Way of Experiencing the World

When Ralph Bishop opened a text on his iPhone, he didn’t discover an ordinary photo or video message. Instead, a virtual trail of arrows beckoned — signals from the sender teasing a virtual scavenger hunt of sorts. Bishop was able to see the trail as his iPhone’s camera viewfinder surveyed the area. He followed the secret pathway to the other side of the room, just beyond the staircase, where a message that read “Good luck!” awaited him.

Bishop, head of design at GIPHY, was demonstrating how the popular picture-sharing platform’s new app works. The app, called GIPHY World, will use Apple’s ARKit platform to make GIFs (a popular format for online images) appear in the real world when viewed through the phone’s camera. ARKit, which will launch with the the latest version of Apple’s iOS operating system for iPhones and iPads this fall, is a streamlined set of tools to help developers create augmented reality experiences.

Augmented reality or AR is a general way of describing software that utilizes your phone’s camera to create the illusion that objects or even fantastical creatures are secretly at arm’s length, capable of interacting with actual objects or space around you. It’s how mobile sensation Pokémon Go can make virtual monsters appear on sidewalks or in backyards, and why Snapchat can lay funky masks over your selfies. But developers and tech firms have struggled thus far to make the technology broadly appealing to audiences outside the occasional mania over a popular app.

That’s why the launch of ARKit is being closely watched as a make-or-break moment. With Apple‘s backing and an instant landing path to millions of iPhones and iPads worldwide, AR is poised to finally breakout. Apple CEO Tim Cook believes augmented reality will change how we interact with technology, telling The Independent in February “I regard it as a big idea like the smartphone.” As apps built on ARKit launch alongside iOS 11 in the coming weeks, that claim will be under the microscope.

Developers have had access to ARKit for several weeks, and in that time dozens of demos have surfaced. Furniture megastore IKEA, for example, plans to release a new augmented reality app called “IKEA Place” that lets you drop virtual furniture into a home. The app’s user interface is unlike that of a typical retail app, employing a camera viewer similar to Snapchat’s that tracks where you’re moving through a room so that it can accurately position furniture. IKEA’s been experimenting with augmented reality since 2013, but during my demo of its new app, I could already see ways in which it’s become more complex. Instead of placing a static sofa against a wall, the new app lets you see how that couch looks when it unfolds into a bed. And you can examine the furniture from any angle, even if you momentarily turn your camera away from the scene.

Apple’s push with ARKit is already attracting some developers that would never have tried augmented reality otherwise. Asher Vollmer, creator of the popular puzzle game Threes, is also developing an app that lets users digitally furnish a room. Vollmer posted a video clip of the tech online, which shows 3D chairs, sofas, tables and fireplaces with lifelike proportions being manipulated around an empty room. He plans to expand the demo into a full app and is talking to furniture companies, he told TIME. Much of Vollmer’s background is in developing mobile games, but he says he’d consider building other augmented reality apps with ARKit. “Now I’m really invested in this world,” he says. “It’s still very early so we’ll see.”

One of the biggest challenges for developers has been figuring out how to design genuinely useful AR apps, not just gimmicks masquerading as novelties. That’s why Laan Labs, the group behind Face Swap Live, has been using the Internet as a testbed to incubate ideas. A company demo video showing how ARKit can turn an iPhone into a virtual tape measurer, popularized on YouTube by Twitter account MadeWithARKit, was by far its most viewed, says Jason Laan, a partner at Laan Labs. The company is working on an ARKit-powered app that makes it possible to measure anything — from a piece of furniture’s dimensions to the diagonal length of a picture hanging on a wall — just by waving your iPhone’s camera over it. Although Laan Labs has developed at least 10 different concepts based on ARKit, with use cases including creating digital drawings and pointing a camera at a food item to discover food recipes (i.e. an avocado may become avocado toast), the firm is only expanding the tape measure demo into a full app at the moment. “It’s always hard determining what the most successful application of a technology is going to be,” says Laan.

Given augmented reality’s implicit playfulness, a natural area for the tech to flourish has been video games. Game maker Climax Studios plans to launch a 3D puzzler called Arise, which projects a lush, floating mass of land filled with ruins. The player’s job is to make sure a heroic character is able to explore the area by connecting gaps in the stage. To do so, you move your iPhone or iPad around until the half-circles marked on the map align, a task that requires examining the landscape from all angles. And studio Next Games is working with entertainment giant AMC on The Walking Dead: Our World, an augmented reality vamp on the post-apocalyptic zombie TV show that turns your environment terrifyingly hostile. During my demo, a life-sized enemy crawled out of a sewer cover virtually overlaying the floor a few steps away. And when fan-favorite character Michonne appeared, her features were detailed and convincing — she even blinks as you approach her.

Apple is hardly the only company providing tools to help developers build augmented reality apps. Alternatives like EasyAR, Wikitude and Kudan have existed for years. But developers say Apple’s solution makes it easier to create convincing experiences, since its tools are a part of the iPhone’s core operating system. Since ARKit is built into iOS, apps run more smoothly because there isn’t an added layer of image processing, says Barry O’Neill, the CEO of Touch Press. His company is building an augmented reality app based on Eric Carle’s renowned children’s book The Very Hungry Caterpillar. A crucial issue that ARKit solves, according to O’Neill, is keeping virtual objects firmly planted in place so that they don’t float askew. “Those things help the suspension of disbelief,” he says. “Which is a very important part of the augmented reality experience.”

This was indeed true upon trying the app: when the tiny red caterpillar inched toward O’Neill in the demo, it appeared to be actually traveling across the carpet. As I moved closer to the caterpillar, I noticed the lighting on its face changed as my shadow moved — another ARKit touch that makes these creations much more realistic.

Apple wasn’t the first and certainly won’t be the last major technology firm to place its bets on augmented reality. Google has been encouraging device makers and app creators to think about how AR should work on smartphones since 2014, when it unveiled its Tango depth-sensing system for mobile phones. On August 29, Google also announced its own alternative to ARKit called ARCore, a software development kit meant to help developers scale their AR apps to all Android phones, including ones that don’t support Tango. And between its HoloLens headset and the “mixed reality” viewers it’s worked on with its PC partners, it’s clear that Microsoft also sees a substantial future in augmented reality.

But smartphone-based augmented reality is ground zero given how intrinsic they’ve become to our lives, and in that sense, both Apple and Google have obvious advantages. It’s on the one hand easier for developers to create apps that work across most iPhones, since there are far fewer iPhone variants than the comparable galaxy’s worth of Android ones. “It’s not an issue of reach, it’s an issue of vertical integration,” says Tuong Nguyen, a principal research analyst covering consumer technologies at Gartner. “[ARKit] was built-in house and is optimized for [Apple’s] hardware.” On the flip side, Android phones that support Tango are equipped with better hardware for providing AR experiences. Two Tango-enabled phones, the Asus Zenfone AR and Lenovo Phab 2 Pro, include depth-sensing hardware that makes it possible to map a space in three dimensions — the iPhone includes no such components, and relies on its standard camera for AR.

It’s impossible to predict whether Apple or Google will dominate this still-nascent technology until more people start using it. The apps I took for a spin were demonstrated under ideal circumstances, with appropriate spacing and good lighting, and it’s unclear how they’ll work in less optimal settings. One thing is certain: a boatload of apps that use the iPhone’s camera to conjure new worlds inside the real one are about to hit the App Store. “Because it’s Apple, [ARKit] gets a critical mass of developers excited,” says Nguyen. “And that’s important because they’re the ones who are innovating.” ARKit’s success will depend on what developers manage to do with it, in other words. If these early apps are any indication, the results are sure to be fascinating.