Inside the Wacky World of Weird Data: What's Getting Crunched

"Big data" is the term industry insiders use to describe a transformational change in computer analytics and business management. It means the slicing and dicing of enormous data sets to discover new—and often surprising—insights into the way the world works.

It's a red-hot field right now—because of twin revolutions going on in the amount of computer data available to study and the dramatic evolution of algorithms and analytics used to study that information.

Where computer scientists were once limited to mere gigabytes or terabytes of information, they're now studying petabytes and even exabytes of information. You don't need to know the math to know that's a colossal amount.

By one claim, all the words ever spoken by human beings ever could be stored in about 5 exabytes of data. And the amount of data in the world is growing exponentially—by another claim, 90 percent of the information in the world has been created just in the last two years.

At the same time, the tools to sift all that data are getting better as computer scientists refine and improve the algorithms they use to extract meaning from the deluge of data. Google, for example, recently unveiled a new version of its search algorithm, called "hummingbird," to provide ever more direct answers to complex questions.

All of that has cleared the way for a fast-growing group of start-up companies staffed with computer scientists and coders who are using this new microscope to examine the world. With names like "Kaggle," "Evolv" and "HaystaqDNA," they're trying to extract insights about the world around us in ways that have never been possible before.

Think of it as "Moneyball," but not just for baseball, for everything.

These new companies, side by side with researchers at universities like Harvard, MIT and Stanford, are taking the concept of what "data" is well beyond old-fashioned Excel spreadsheets. Forget tabs and columns. How about every pixel in every image of the world on Google Earth? That's data.

How about every post by every person on every Chinese social media site? That's data, too. Or what happens when you sort millions of job applicants not by the information on their resume, but simply by which Web browser they used to send in the application? That's also data.

And a lot of what these analysts are discovering about the world is, in some cases, deeply weird. Call it "weird data."

The computer scientists are finding patterns we never knew existed. Slicing Americans into demographic groups that no one has ever thought of. Finding signals in what has long been assumed to be just noise.

Want to know how many Republicans have solar panels on the roofs of their homes? You can do that. Do people with reverse-gender names—think of boys named "Sue"—shop differently than everyone else? You can find out. The companies are convinced there's money to be made in mining for these kinds of strange-but-true insights.

These firms are going beyond just looking at the past. With the growing field of "predictive analytics" these firms are learning how to tell the future, too. One start-up founder said he wants to be able to predict when a woman will have a baby even before she starts trying to get pregnant. There's an entire industry of baby-product companies that want to target them before their competitors do.

In the wake of Edward Snowden's NSA leaks, a lot of people are worried about just how much the government knows about them. But corporations and researchers are able to learn enormous amounts, too. And that raises a question: Should there be any limits on how much information private firms can gather about you?

Ironically, it's the Obama administration—which has stalwartly defended the NSA's metadata telephone collection program and other information gathering—that's conducting a study on exactly that. Called "Big Data and the Future of Privacy," the White House study is expected to be completed this spring.

"We are undergoing a revolution in the way that information about our purchases, our conversations, our social networks, our movements, and even our physical identities are collected, stored, analyzed and used," wrote White House counselor John