In the end, big data is really a backward science.
It’s a rear-view mirror tool. Think of a super computer with OCD tirelessly sorting through past events – even in real-time as they un-spool – one that seeks out repetitive patterns in the world, makes sense of them, then sells them to the highest bidder.
Most of us remember a way of life guided by homilies, intuition, and superstition. Now we have apps like Yelp, Match.com and Pandora that tap into the power of big data and help discover our hearts desire. Just for fun, check out the documentary attached to see how music discovery was done back in the days before Pandora came on the scene.
Big data is so big you need a machine to figure it out.
Data scientists now employ a branch of statistics called “machine learning” to analyze big data sets. And central to this is the algorithm – a computer program designed by humans, then set loose on big data to provide superhuman insights and automated response. Algorithms combined with big data allow us to model the weather, execute stock trades, decode the genome, even sell more toothpaste.
To understand the math behind machine learning see here and here. Or to ponder its impact on our lives, watch the TED talk below:
Big data is a by-product of our digital lives.
Experts and pundits have begun to use the term “digital exhaust” to describe the data which we as a global community throw off each day. Some of this data comes from academic centers (think CERN or the Human Genome Project). But the lion’s share flows from the digital footprints left by individuals, businesses and institutions through the day.
Here’s a rough breakdown provided by Deloitte:
- Web & social media – click-stream and web interaction data
- Machine-to-machine – data from senors, meters, “internet of things”
- Big transaction data – customer/partner transaction data
- Biometric – data includes fingerprints, genetics, retinal scans
- Human-generated – unstructured data like email, social posts, video
And for a deeper dive, see below:
Big data is truly big, and growing exponentially.
Statistics circulating online put things into perspective. IBM says that 90% of the data in the world today was created in just the last two years. CSC projects a 4300% increase in data production by 2020. IDC predicts that by 2020 the annual data creation will reach 40 Zettabytes – equivalent to 360 million years of BlueRay quality video or the data created if every US citizen wrote 3 tweets per minute for the next 598,867 years.
For more information see here, here, or here. Or watch commentary below by John Dodge of the Enterprise CIO Forum:
Data can be surprisingly beautiful.
Thanks to new data visualization techniques, and the artists and designers pioneering them, we can transcend traditional numeric tables and now see data in a whole new way.
Consider the Literary Organism project by designer Stefanie Posavec. Through path analysis and color coding she has transformed the narrative structure of Kerouac’s On The Road into a graphic rendition of both rich factual content and delicate beauty. Also, spend 20 minutes with Jer Thorp, NYT Artist In Residence, in the talk below where he shares his passion for the emerging field of data visualization. Beauty is truth!
Data has suddenly gotten sexy.
Data used to be considered boring. It was the domain of engineers (think protractors), organic chemists, accountants, actuarial scientists. Now, thanks to massive server farms, clever algorithms and machine learning, data has become the lifeblood of Silicon Valley and the foundation of huge personal fortunes.
To understand these changes read a recent article in the Harvard Business Review which chronicles the emergence of big data as “the sexiest job of the 21st century.” Even more telling, check out the recent TED talk below by Jameson Toole of MIT’s Human Mobility & Network Lab. The gleam in his eye speaks volumes about the irresistible allure of big data to scientists, entrepreneurs and speculators of our time.
Full disclosure – I’m a brand research executive and skilled in what is sometimes called “empathetic listening.” This means I’m an inveterate people watcher, instinctively put myself in others’ shoes and strive to figure out what makes people tick.
I’ve been reading up on trends in UX and brand design recently and find that an empathetic approach is highly prized at agencies as various as IDEO, frog design, Landor, and BBDO. Empathy is considered an essential component in up-front evaluation of user needs and, as well, is often leveraged in creative strategy and execution. Consider P&G’s amazing new campaign for the 2012 Olympic sponsorship, “road to glory”, that recognizes the role played by moms in supporting their young athletes. In this work:
- Empathetic listening drives insights that fuel creative/design strategy
- Empathetic messaging is at the center of the executional elements celebrating the games’ unsung heroes…moms
To check out the campaign click here. And look below to see another amazing example of empathetic listening driving empathetic creative – part of a pro-bono campaign by BBDO for the Ad Council.
Moral of the story – empathy opens doors to human understanding and is an irresistible force that breaks down barriers and brings people together.
God is in the details.
The existence of Higgs boson, a sub-atomic particle dubbed “the God Particle” because it explains the dynamics of mass, was confirmed on July 4th at the CERN labs outside Geneva. Peter Higgs first proposed it in 1965, but it took an additional 47 yeas, lots of math, and a $10 billion supercollider to prove it was real. Check these links for a deeper dive:
Amazing, really. Years of struggle to confirm a scientific principle that appears to have no practical value. The Economist put it this way, “The importance of Higgs belongs to the realm of understanding rather than utility. It adds to the sum of human knowledge – but it may never change lives as DNA or relativity have.”
The moral of the story – the thirst for knowledge is a strong, motivating force in human affairs.