I’m sitting behind a workbench in a white-walled room in Dania Beach, Florida, in the office of a secretive startup called Magic Leap. I’m staring wide-eyed through a pair of lenses attached to what looks like metal scaffolding that towers over my head and contains a bunch of electronics and lenses. It’s an early prototype of the company’s so-called cinematic-reality technology, which makes it possible for me to believe that the muscular beast with the gruff expression and two sets of swinging arms is actually in the room with me, hovering about seven feet in front of my face.
He’s not just visible at a set distance. I’m holding a video-game controller that’s connected to the demo station, and at the press of a button I can make the monster smaller or larger, move him right or left, bring him closer, or push him farther away.
Of course, I bring him as near as possible; I want to see how real he looks up close. Now he’s about 30 inches from my eyeballs and, though I’ve made him pocket-sized, looks about as authentic as a monster could—he seems to have rough skin, muscular limbs, and deep-set beady eyes. I extend my hand to give him a base to walk on, and I swear I feel a tingling in my palm in expectation of his little feet pressing into it. When, a split second later, my brain remembers that this is just an impressively convincing 3-D image displayed in the real space in front of me, all I can do is grin.
Virtual- and augmented-reality technologies used in movies, smartphone apps, and gadgets tend to underdeliver on overhyped promises with images that look crappy. Typically that’s because stereoscopic 3-D, the most commonly used method, is essentially tricking your eyes instead of working with the way you normally see things. It produces a sense of depth by showing each eye a separate image of the same object at a different angle. But since that forces you to look simultaneously at a flat screen in the distance and images that appear to be moving in front of you, it can make you dizzy and lead to headaches and nausea.
To be sure, stereoscopic 3-D has recently started getting better. The best system you can currently buy comes from Oculus VR, which Facebook purchased last spring for $2 billion; the $199 Gear VR, which was built in collaboration with Samsung and is aimed at software developers, lets you slide a Samsung smartphone into a headset to play games and watch videos.
But while Oculus wants to transport you to a virtual world for fun and games, Magic Leap wants to bring the fun and games to the world you’re already in. And in order for its fantasy monsters to appear on your desk alongside real pencils, Magic Leap had to come up with an alternative to stereoscopic 3-D—something that doesn’t disrupt the way you normally see things. Essentially, it has developed an itty-bitty projector that shines light into your eyes—light that blends in extremely well with the light you’re receiving from the real world.
As I see crisply rendered images of monsters, robots, and cadaver heads in Magic Leap’s offices, I can envision someday having a video chat with faraway family members who look as if they’re actually sitting in my living room while, on their end, I appear to be sitting in theirs. Or walking around New York City with a virtual tour guide, the sides of buildings overlaid with images that reveal how the structures looked in the past. Or watching movies where the characters appear to be right in front of me, letting me follow them around as the plot unfolds. But no one really knows what Magic Leap might be best for. If the company can make its technology not only cool but comfortable and easy to use, people will surely dream up amazing applications.
That’s no doubt why Google took the lead in an astonishingly large $542 million investment round in Magic Leap last October. Whatever it is cooking up has a good chance of being one of the next big things in computing, and Google would be crazy to risk missing out. The investment looked especially prescient in January, when Microsoft revealed plans to release a sleek-looking headset this year. HoloLens, which lets you interact with holograms, sounds as if it’s very similar to what Magic Leap is working on.
Behind the magic
Magic Leap won’t say when it will release a product or how much the thing will cost, beyond that the price will be within the range of today’s consumer mobile devices. When I press founder and CEO Rony Abovitz about such details, he’ll only smile and say, “It’s not far away.”
He’s sitting behind the desk in his office, which is just down the road from the Fort Lauderdale–Hollywood airport. The shelves are lined with toys and View-Masters—the plastic gadgets that let you look at pictures in 3-D. Abovitz, 44, is a bear of a guy with a kind smile, and when I meet him he’s dressed in black Nikes, a long-sleeved shirt, and slacks, his graying curly hair topped with a yarmulke. He’s thoughtful and composed, which I find somewhat surprising given that the only time I had seen him before was in a video of his talk at a TEDx event in 2012 in Sarasota, Florida. It featured two people dressed as furry creatures called “Shaggles,” Abovitz walking on stage dressed as an astronaut, and unintelligible rock music. Though the talk, called “The Synthesis of Imagination,” came off as performance art (perhaps even a mockery of a TED talk), he swears there is a coherent message embedded in it; figure it out, he says, and he’ll give you a yo-yo.
By day, Abovitz is a technology entrepreneur with a background in biomedical engineering. He previously founded Mako Surgical, a company in Fort Lauderdale that makes a robotic arm equipped with haptic technology, which imparts a sense of touch so that orthopedic surgeons have the sensation of actually working on bones as they trigger the robot’s actions. Mako was sold to a medical technology company, Stryker, for nearly $1.7 billion in 2013. By night, Abovitz likes to rock out. He sings and plays guitar and bass in a pop-rock band called Sparkydog & Friends. And as he tells it, Magic Leap has its origins in both the robotic-surgery company and his life as a musician.
Combining virtual reality with the physical world appealed to Abovitz even at Mako. Although the robotic-arm technology could give surgeons the sensation of touching their instruments to bones, Abovitz also wanted to let them see virtual bones as they went about this work. Over and over, he says, he tried out head-mounted displays made by different companies, but he was disappointed with them all. “They were all just complete crap,” he says. “You’d put it on and it would give you a headache and it was awful, and I was wondering, ‘Why is this so bad?’”
At the same time, Abovitz also wanted to take Sparkydog & Friends on a virtual tour. In U2’s 1987 video for “Where the Streets Have No Name,” the group, in a nod to an earlier move by the Beatles, plays an impromptu show on the roof of a Los Angeles liquor store. Abovitz yearned for his band to be able to do that, but virtually, and on a thousand rooftops at once.
About four years ago, he started mulling the problem over with John Graham Macnamara, a high school friend who had dropped out of Caltech’s theoretical physics program. They became captivated by the idea of displaying moving holograms like the one in Star Wars. Holograms—3-D images that can be viewed from many angles—are made by accurately re-creating light fields, the patterns made when light rays bounce off an object. But Abovitz figured it would cost a lot and take lots of time to project even low-resolution holographic images. At one point, he remembers muttering, “There is no display that can actually work.”
The next morning, though, he awoke with an idea: why bother with the painstaking steps needed to send a hologram out into a room for multiple people to see at once? Why not, instead, essentially make a hologram that only you see, doing it in a way that is natural for the eyes and brain to perceive, unlike stereoscopic 3-D? “We’re spending half a billion dollars–plus to effectively make nothing happen to you, physiologically,” Abovitz says.
The solution he and Macnamara and the rest of Magic Leap’s team have come up with is still largely under wraps, and on the record they avoid discussing how the technology works except in vague terms, citing concerns about competition. But it’s safe to say Magic Leap has a tiny projector that shines light onto a transparent lens, which deflects the light onto the retina. That pattern of light blends in so well with the light you’re receiving from the real world that to your visual cortex, artificial objects are nearly indistinguishable from actual objects.
If the company can get this to work in a head-mounted display, showing images near the eyes and consistently refocusing them to keep everything looking sharp, it will make 3-D images much more comfortable to view, says Gordon Wetzstein, an assistant professor of electrical engineering at Stanford who researches computational imaging and displays. “If they do what people suspect they do,” Wetzstein says, “it will be amazing.”
From virtual to reality
Magic Leap is working feverishly to get to that point. Since building its first prototype in 2011, the company has continued to shrink its technology down.
Already it works on something smaller than the unwieldy scaffolding I used. In another demonstration, using hardware on a cart, I can poke at a tiny flying steampunk robot, a character from a first-person-shooter game called Dr. Grordbort’s Invaders that Magic Leap is making with Weta Workshop, which created many of the special effects in the Hobbit movies. The robot can follow my finger around with surprising accuracy, right between the cubicles in Magic Leap’s office.
To judge from a look I get at a design prototype—a realistic-looking piece of hardware that’s completely nonfunctional—the company appears to be aiming to fit its technology into a chunky pair of sports sunglasses wired to a square pack that fits into your pocket. A somewhat similar image in a patent application Magic Leap filed in January suggests as much, too. The company won’t say for sure, though; Abovitz confirms that the headset will be a glasses-like wearable device, but I have to twist his arm to get him to agree to use even that hazy phrasing on the record.
Abovitz was enigmatic in his brief appearance on a TEDx stage in 2012. “A few awkward steps for me; a magic leap for mankind,” he said from inside his spacesuit.
It’s clear that getting the technology into that small form will be very hard. The smallest demo hardware I’ve seen at Magic Leap can’t yet match the experience of the bigger demo units. It includes a projector, built into a black wire, that’s smaller than a grain of rice and channels light toward a single see-through lens. Peering through the lens, I spy a crude green version of the same four-armed monster that earlier seemed to stomp around on my palm. In addition to improving the resolution of smaller units, Magic Leap will have to cram in sensors and software that will track your eyes and fingers, so you can control and interact with its virtual creatures—which themselves will have to incorporate real-life objects into whatever they appear to be doing.
That’s where last year’s half-billion dollars of investment come in. Magic Leap is hiring like crazy. It’s looking for software engineers for everything from eye tracking and iris recognition to the branch of artificial intelligence known as deep learning. It needs optical engineers, game designers, and other people who will dream up virtual objects to display. To give you a sense of where their minds might go, I saw ray guns and magic wands lying around the office. As its chief futurist, Magic Leap has hired the science fiction author Neal Stephenson, whose 1992 novel Snow Crash imagined a virtual world called the Metaverse.
The excitement of such quick growth is palpable at Magic Leap’s brightly decorated headquarters, where staid office trappings are punctuated by red high-backed love seats and yellow chairs. Employees energetically describe the games, sensors, and ray guns they’re working on.
With the massive investment last year, interest in the company has intensified. Abovitz says, “We went from ‘Does anyone care about this?’ to ‘Okay, people do care.’” Now he and the team are feeling the weight of these expectations. He says, “The inner 11-year-old—we want to blow that away.”
