How To Engineer Serendipity

I’d like to tell the story of a paradox: How do we bring the right people to the right place at the right time to discover something new, when we don’t know who or where or when that is, let alone what it is we’re looking for? This is the paradox of innovation: If so many discoveries — from penicillin to plastics – are the product of serendipity, why do we insist breakthroughs can somehow be planned? Why not embrace serendipity instead? Because here’s an example of what happens when you don’t.

When GlaxoSmithKline finished clinical trials in May of what it had hoped would be a breakthrough in treating heart disease, it found the drug stank — literally. In theory, darapladib was a wonder of genomic medicine, suppressing an enzyme responsible for cholesterol-clogged arteries, thus preventing heart attacks and strokes. But in practice it was a failure, producing odors so pungent that disgusted patients stopped taking it.

Glaxo hadn’t quite bet the company on darapladib, but it did pay nearly $3 billion to buy its partner in developing the drug, Human Genome Sciences. The latter’s founder, William Haseltine, once promised a revolution in drug discovery: After we had mapped every disease to every gene, we could engineer serendipity out of the equation. Darapladib was to have been the proof — the product of scientists carefully picking their way through the company’s vast genetic databases. Instead it’s a multi-billion-dollar write-off.

Big Pharma is hardly alone when it comes to overstating its ability to innovate, although it may be in the worst shape. By one estimate, the rate of new drugs developed per dollar spent by the industry has fallen by roughly a factor of 100 over the last 60 years. Patent statistics tell a similar story across industry after industry, from chemistry to metalworking to clean energy, in which top-down innovation has only grown more expensive and less efficient over time. According to a paper by Deborah Strumsky, José Lobo, and Joseph Tainter, the average size of research teams bloated by 48 percent between 1974 and 2005, while the number of patents per inventor fell 22 percent during that time. Instead of speeding up the pace of discovery, large hierarchical organizations are slowing down — a stagflationary principle known as “Eroom’s Law,” which is “Moore’s Law” spelled backwards. (Moore’s Law roughly states that computing power doubles every two years, a principle enshrined at the heart of technological progress.)

While Big Pharma’s American scientists were flailing, their counterparts at Paris Jussieu — the largest medical research complex in France — were doing some of their best work. The difference was asbestos. Between 1997 and 2012, Jussieu’s campus in Paris’s Left Bank reshuffled its labs’ locations five times due to ongoing asbestos removal, giving the faculty no control and little warning of where they would end up. An MIT professor named Christian Catalini later catalogued the 55,000 scientific papers they published during this time and mapped the authors’ locations across more than a hundred labs. Instead of having their life’s work disrupted, Jussieu’s researchers were three to five times more likely to collaborate with their new odd-couple neighbors than their old colleagues, did so nearly four to six times more often, and produced better work because of it (as measured by citations).

The lesson? We still have no idea how to pursue what former U.S. Defense Secretary Donald Rumsfeld famously described as “unknown unknowns.” Even an institution like Paris Jussieu, which presumably places a premium on collaboration across disciplines, couldn’t do better than scattering its labs at random. It’s not enough to ask where good ideas come from — we need to rethink how we go about finding them.

I believe there’s a third way between the diminishing returns of typical organizations and sheer luck. In Silicon Valley, they call it “engineering serendipity,” and if that strikes you as an oxymoron (which it is), perhaps we need to step back and redefine what serendipity means:

Any society that values novelty and new ideas (like our innovation-obsessed one) will invariably trend toward greater serendipity over time. The push toward greater diversity, better public spaces, and an expanded public sphere all increase the potential for fortuitous discoveries.

The flip side is that institutions failing to embrace serendipity will ossify and die. This is especially true in our current era of incessant disruption, as seen in rising corporate mortality rates and a surge of unpredictable “black swan” events. (Nassim Taleb’s advice for taming black swans, by the way? “Maximize the serendipity around you.”)

Finally, the greatest opportunities for engineering serendipity lie in software, which means we must take great care as to who can find us and how, before Google (or the NSA) makes these choices for us.

It’s no coincidence Silicon Valley is obsessed with serendipity. Everyone is familiar by now with the origins of the Post-it Note, Velcro, corn flakes, and Nike’s waffle sole, to say nothing of Teflon, Kevlar, dynamite, and vast swaths of modern chemistry and medicine. The Valley’s contributions include microprocessors and inkjet printers, while Steve Jobs didn’t discover desktop computing or the mouse until a reluctant visit to Xerox PARC in 1979 — which beget the Macintosh and everything after.

When Yahoo banned its employees from working from home in 2013, the reasons the struggling company gave had less to do with productivity than serendipity. “Some of the best decisions and insights come from hallway and cafeteria discussions, meeting new people, and impromptu team meetings,” explained an accompanying memo. The message from new CEO Marissa Mayer was clear: Working solo couldn’t compete with lingering around the coffee machine waiting for inspiration — in the form of a colleague — to strike.

Google and Facebook have gone Yahoo one better. Rather than sit back and wait for serendipity to happen, the search giant has commissioned a new campus expressly designed, in the words of its real estate chief, to maximize “casual collisions of the work force.” Rooftop cafés will offer additional opportunities for close encounters, and no employees in the complex will be more than two and a half minutes away from one another. “You can’t schedule innovation,” said David Radcliffe, but you can make introductions — as both Googlers and Mayer know well. The latter attributes the genesis of such projects as Gmail, Google News, and Street View on her watch to engineers meeting fortuitously at lunch.

Meanwhile, Facebook has hired architect Frank Gehry to build “the perfect engineering space: one giant room that fits thousands of people, all close enough to collaborate together,” founder Mark Zuckerberg explained. The goal of each company is the same: to create the best conditions for spreading the most valuable kind of ideas — the hunches locked inside our skulls until a felicitous combination of circumstances sets them free.

Mayer’s demand for proximity ignited a debate that’s still raging: What’s the best way to work, together or alone? Finally breaking her silence on the matter in the spring of 2013, she conceded “people are more productive when they’re alone,” then added, “but they’re more collaborative and innovative when they’re together. Some of the best ideas come from pulling two different ideas together.”

She’s right. (Not that Yahoo has many ideas to show for it.) We experience moments of serendipity daily, each with potentially huge payoffs down the road. But because we can’t predict which ideas will collide and fuse, we cling to boring productivity and efficiency. We not only run our lives but our entire economy this way, using GDP and even grosser statistics to measure progress that has never unfolded in a straight line. Life is emergent and unknowable — we’re just terrified to manage it that way. And because we only attribute our success to serendipity after the fact (if at all), we typically consign it to anecdotes (e.g. Post-it Notes), turning to them only when the numbers don’t add up. The problem is that more and more of the most important numbers — including patent applications, R&D budgets, and even economic growth — have stopped adding up.

We take the pace of innovation for granted. We assume that like Moore’s Law, the rate of scientific discoveries and inventions is smoothly accelerating. But we’re wrong. A growing body of research suggests the opposite is true; Eroom’s Law rules. That this is happening in nearly every industry means something deeper is at work — that the corporation itself is reaching its limits when it comes to invention. Like the long-dead societies he’s excavated, Joseph Tainter — who’s most famous for his book The Collapse of Complex Societies — believes companies have become too rigid and hierarchical to survive disruption, seeking only to discover what they already know. What’s missing is serendipity.

The same phenomenon that produced a gusher of new research papers at Paris Jussieu once produced the laser and transistor at Bell Labs and breakthroughs in linguistics and acoustics at MIT. It’s still happening in places like IBM’s Thomas J. Watson Research Center in Yorktown Heights, New York, where a chance meeting of a physicist and biologist in a hallway a few years ago led to a tiny microchip able to single-handedly sequence long strands of DNA. It’s no accident that the Watson Research Center produces more patents per year than any other building in the world, and IBM more than any other company.

In all of these cases, serendipity was responsible for the bridging of what the University of Chicago sociologist Ronald Burt calls “structural holes,” which appear when org charts and other formal structures create gaps in the informal network of experts floating through a company, campus, or city. In a landmark study a decade ago, Burt found that managers who straddled holes between teams and domains consistently produced better ideas than those who did not (and were rewarded accordingly). “This is not creativity born of genius,” he wrote. “It is creativity as an import-export business.” The easiest way to discover an idea, it would seem, is to borrow one.

Burt’s findings have been borne out again and again; in one study, a slight increase in serendipity generated more revenue and projects while speeding up their completion. (Contrary to Mayer’s mea culpa, it appears bumping into people makes you more productive, too.)

There’s a rich vein of research running through sociology, anthropology, network science, and management theory explaining how serendipity increases one’s “absorptive capacity,” i.e., our ability to recognize, assimilate, and put to use knowledge from outside our personal experience. Other studies demonstrate how successful firms harness serendipity to lower the costs and barriers to collaboration. And still others suggest that how we share “non-redundant information” across a social network is more important than the experience or credentials of any one person in the network itself, which explains how scattering scientists across a campus at random could vastly improve the quality of their work.

But perhaps the most interesting thing about all of these examples is that they were unintentional. Serendipity may not be luck after all — there is a hidden order to how we find new ideas and people — but we will never realize more than the tiniest fraction of its potential as long as we treat it that way. So how do we go further and actually plan for serendipity?

The first step takes place in our own minds. A few years ago, an Australian psychologist named James Lawley realized that no one had mapped the experience of serendipity before. Upon re-reading the letter in which the British aristocrat Horace Walpole coined the word in 1754, he noticed the fabled Three Princes of Serendip “were always making discoveries, by accidents and sagacity, of things they were not in quest of.” Today, all anyone remembers are the accidents. But equally important is sagacity, which the chemist Louis Pasteur famously called “the prepared mind.”

“What kind of mind is it?” Lawley asks. “One that thinks more systematically than simple cause-and-effect.” In other words, it’s a mind that’s open to the unexpected, to thinking in metaphors, to holding back and not jumping to conclusions, and to resist walls between domains and disciplines. It’s a mind that looks a lot like Joi Ito’s.

Ito is a former DJ, venture capitalist, and entrepreneur who moved to Dubai on a whim to get a better feel for the place. (That’s when he wasn’t traveling 300 days a year.) “My job was running around mostly making connections,” is how he describes it. That was before he was picked to run the MIT Media Lab, despite never finishing college himself.

Headlining a panel at 2013’s South by Southwest titled “The New Serendipity,” Ito talked about the qualities he’s cultivated within himself — being “antidisciplinary” and retaining his “beginner’s mind” — which he hopes will guide the Media Lab. “We aim to capture serendipity,” he said. “You don’t get lucky if you plan everything — and you don’t get serendipity unless you have peripheral vision and creativity.”

That’s also true for the next step, which is engineering serendipity into organizations. For all the talk of “failing faster” and disruptive innovation, an overwhelming majority of companies are still structured along predictable lines. Even Google cancelled “20 percent time,” its celebrated policy of granting engineers one day a week for personal projects. To capture serendipity, the company is looking at space instead of time — hence the design of its new campus, in which everyone is just a short “casual collision” away.

But how can we do a better job of bringing people together than installing bigger cafeteria tables, adding another coffee machine, or locking all the bathrooms but one? A start would be to tear down the walls preventing colleagues in one department or company from bumping into peers from another. That’s what AT&T has done with its worldwide Foundry network, where selected startups and entrepreneurs work alongside its own engineers as well as those from partners such as Intel, Cisco, and Ericsson. One of these startups, Intucell, improved AT&T’s call retention and throughput speeds by 10 percent and was later bought by Cisco for $475 million. In general, Foundry teams have cut the development time of new products from three years to nine months.

It’s telling that the Foundry outpost in Silicon Valley is stationed in downtown Palo Alto, where the chances of someone dropping in on their walk back from lunch are substantially greater than in some exurban office park. Cities are the greatest serendipity engines of all. They began life at crossroads as places to exchange goods and later ideas with others you would never encounter on the farm.

Only recently, we’ve come to recognize great ones for what they are — not as collections of skyscrapers (which China can build but can’t fill), but as the sum of their dense, rich, and overlapping networks of people. “They’re not a set of people, they’re not a set of roads; they’re a set of interactions,” says Luis Bettancourt, a physicist who describes cities as “social reactors.” Like the sun, they’re places where strangers collect, collide, and fuse — releasing tremendous heat and light in the process. What makes a city great, in other words, is how well its people are connected — to the city itself and to each other. And to make a city better, you have to engineer serendipity.

Which is what Tony Hsieh is trying to do in Las Vegas. Much has been written about the Zappos CEO’s flailing efforts to terraform downtown into a desert facsimile of Brooklyn or the Mission district, but his instincts are correct. He envisions every bar and coffee shop around the company’s downtown campus as an extension of its conference rooms, inviting strangers to work alongside his employees. He fervently believes blurring the line between the city and his company will make people in both smarter, happier, and more productive. “If you accelerate serendipity,” he says, “you’ll accelerate learning.”

To ensure that happens, he’s imported dozens of tech startups for his employees to learn from. Stipulated in their contracts is a promise to spend “1,000 hours per year of serendipitous encounters” downtown, searching for collisions and conversations. While it remains to be seen whether Hsieh can build a successful creative class company town, he’s right to believe the energies of the city are greater than any one company.

The final piece is the network. Google has made its ambitions clear — as far as chairman Eric Schmidt is concerned, the future of search is a “serendipity engine” answering questions you never thought to ask. “It’ll just know this is something that you’re going to want to see,” explained artificial intelligence pioneer Ray Kurzweil shortly after joining the company as its director of engineering.

One antidote to this all-encompassing filter bubble is an opposing serendipity engine proposed by MIT’s Ethan Zuckerman. In his book, Rewire, he sketches a set of recommendation and translation tools designed to nudge us out of our media comfort zones and “help us understand whose voices we’re hearing and whom we are ignoring.”

As Zuckerman points out, the greatest threats to serendipity are our ingrained biases and cognitive limits — we intrinsically want more known knowns, not unknown unknowns. This is the bias a startup named Ayasdi is striving to eliminate in Big Data.Rather than asking questions, its software renders its analysis as a network map, revealing hidden connections between tumors or terrorist cells, which CEO Gurjeet Singh calls “digital serendipity.”

IBM is trying something similar with Watson, tasking its fledgling artificial intelligence software with reading millions of scientific papers in hopes of finding leads no human researcher would ever have time to spot. Baylor’s College of Medicine used it this way to identify six new proteins for cancer research in a month; the entire scientific community typically finds one per year.

Baylor’s experiment — much like Paris Jussieu’s unintentional one — tells us something profound about the potential for new discoveries. Rather than compiling ever-bigger data sets or throwing more bodies at a problem, we need tools, organizations, and environments geared less toward efficiency — which is suffering from decreasing returns — and more toward what John Hagel III and John Seely Brown call “scalable learning,” in which serendipity is crucial.

So, what if we borrowed Ayasdi to power a social serendipity engine — one to identify who’s nearby, parse our hidden relationships, and make introductions? How would it work? We’d want it to be as easy as Tinder, which now owns half the mobile dating market. Next, we’d need context — why do I want to meet this person? Tinder works because its logic is binary: Swipe right or left. Everything else is harder.

That context exists somewhere in our data exhaust. For example, Relationship Science has mapped the connections between 3 million members of the 1 percent using publicly available information from more than 10,000 databases. Its customers use it to trace paths to their quarry via colleagues, corporate boards, and alma maters, with each link graded into strong, medium, and weak ties. Meanwhile, a startup named Rexter mines users’ email, calendars, and contacts to calculate the value of their connections and assign tasks accordingly. And, of course, there’s no shortage of sensors available — from smartphones to beacons to “sociometric badges.”

Now, take all of that and run it through Ayasdi’s digital serendipity engine. We could conceivably perform the equivalent of Baylor’s Watson experiment with the researchers of Paris Jussieu, plugging hundreds if not thousands of structural holes in months or even weeks, rather than fifteen years. What would we find then?

Usually, when I describe this vision, someone will reply, “But that isn’t serendipity!” I’m never quite sure what they mean — because it isn’t random or romantic? Serendipity is such a strange word; invented on a whim in 1754, it didn’t enter widespread circulation until almost two centuries later and is still notoriously difficult to translate. These days, it means practically whatever you want it to be.

So, I’m staking my own claim: Serendipity is the process through which we discover unknown unknowns. Understanding it as an emergent property of social networks, instead of sheer luck, enables us to treat it as a viable strategy for organizing people and sharing ideas, rather than writing it off as magic. And that, in turn, has potentially huge ramifications for everything from how we work to how we learn to where we live by leading to a shift away from efficiency — doing the same thing over and over, only a little bit better — toward novelty and discovery.

This essay was made possible with the generous support of the John S. and James L. Knight Foundation.

This article was originally published by The Aspen Institute on Medium