In the end, big data is really a backward science.
It’s a rear-view mirror tool. Think of a super computer with OCD tirelessly sorting through past events – even in real-time as they un-spool – one that seeks out repetitive patterns in the world, makes sense of them, then sells them to the highest bidder.
Most of us remember a way of life guided by homilies, intuition, and superstition. Now we have apps like Yelp, Match.com and Pandora that tap into the power of big data and help discover our hearts desire. Just for fun, check out the documentary attached to see how music discovery was done back in the days before Pandora came on the scene.
Big data is so big you need a machine to figure it out.
Data scientists now employ a branch of statistics called “machine learning” to analyze big data sets. And central to this is the algorithm – a computer program designed by humans, then set loose on big data to provide superhuman insights and automated response. Algorithms combined with big data allow us to model the weather, execute stock trades, decode the genome, even sell more toothpaste.
To understand the math behind machine learning see here and here. Or to ponder its impact on our lives, watch the TED talk below: