Making every failure count ... with data mining

20 years ago   •   1 min read

By Marcia Kadanoff

What might at first glance look like failure just may have a silver lining—but only if you know where to look. And knowing where to look requires more than mere serendipity, says columnist Michael Schrage, who prefers to turned Pasteur’s famous aphorism on its head: “The prepared mind favors chance.” Schrage cites the example of Merck, which was disappointed when its experimental antidepressant, “Substance P,” flunked its phase III clinical trials. Over the course of its extensive testing, however, researchers noted that ailing lab ferrets who were fed Substance P vomited much less than expected. Merck won approval to use Substance P as an antinausea/antivomiting medication in 2003, says Schrage, in large part because “capital-intensive innovators like Merck increasingly structure their research initiatives to ensure that such startling correlations trigger recognition and review.” In Schrage’s view, successful companies will need to depend on “prepared minds” to tease out statistical correlates in the massive dataspheres typical of biotechnology and other fields.

“These meta-analyses will become how prepared minds cultivate chance as well as exploit it. Innovators will spend less time designing clever experiments to generate data and more time scouring the data to generate hypotheses.” The future of innovation will increasingly be determined by the future of data-driven statistical techniques. The future of data-driven statistical techniques, however, depends on innovators who grasp that ferret vomit can be a source of inspiration. That’s not lucky serendipity; that’s good design.”

Technology Review Jul/Aug 2004

Spread the word

Keep reading