It probably comes as no surprise that the human race is generating more data than it knows what to do with. But the numbers behind this insane, exponential buildup of bits may still shock you: In a recent editorial, IBM's VP of supercomputing, Dave Turek, mentions that if we start in the year 2003 and go all the way back to the beginning of human history, we'll find that we as a species have created 5 exabytes (5 billion gigabytes) of information. In 2011, we began creating that same amount of data every 2 days, and by 2013, IBM and others expect we'll be doing the same every 10 minutes.
So yes, there is no question that we are now producing an unimaginable amount of data at an astronomical rate. The question we should now be asking – and the one that Turek and others are attempting to answer – is how are we going to keep up without plummeting into financial and mental ruin?
Here's the rub, which a lot of people these days tend to overlook: data, intangible as it may seem, takes up physical space, and the machines that occupy that space cost real money to maintain. A lot of money. For example, IBM currently estimates the costs of moving one exabyte of data for processing at $10 million.
So why, knowing this, have we kept chugging along under the assumption that everything will work out in the end?
"The traditional IT business model has rested largely upon the promise of ever-denser microprocessors ... That promise is failing us"
Up to now, most technologists have continued to operate with faith in Moore's Law, which states that the transistor density of computers will double roughly every 18 months, giving rise to dependably smaller and smaller vessels for electronic information storage. Moore's prediction was uncanny and it became axiomatic among computer scientists in the late 20th and early 21st centuries. More recently, however, many believe it's beginning to level off. Some even have outright declared it dead.
Turek points out that as chips increase in density, the amount of heat they generate also increases, which means more energy is necessary to properly cool the labyrinthine data centers popping up around the world. Some have suggested relocating data centers to the Arctic Circle, a solution which doesn't seem too attractive considering that we're already concerned about the lasting damage we're doing to the environment.
More sensible is Turek's suggestion, which pushes for machines and storage devices made from things like graphene and various heat-resistant nano-materials. Others like theoretical physicist Michio Kaku say that some time soon we're going to have to ditch silicon altogether for molecular or quantum computing.
There is another thing to consider, however: our ability to create data could also have a natural threshold of its own, and a study from last year suggests that it stems from our inability as humans to mentally process it all. The study found that in accordance with the Weber-Fechner law, the ability to generate new data is limited based on the type of data and the ability of humans to efficiently consume it. Sampling 600 million files of various types sourced from Wikipedia, the group of computer scientists found discrepancies based on whether the data is near-instantly absorbable, like an image, or requiring a more substantial time commitment, like video or text.
It's clear that we as humans are not really the greatest at managing the ridiculous amounts of data we produce. Fortunately there are plenty of potential solutions in the mix, but the most important question is whether we'll be willing or even able to perfect and harness those solutions before we find ourselves teetering on the edge of an information meltdown.