According to IBM 90% of all data has been created within the last two years, and 80% of it is unstructured: documents, videos, images, e-mails, etc. No wonder Big Data is a Big Topic of conversation these days.
This all sounds amazing, but to me it feels like technology is just catching up to reality. Most of the world’s data has always been unstructured: thoughts and memories in people’s minds; carvings on stone; or printed on a piece of paper. The concept of structured data came along when databases appeared and data was required to fit neatly within distinct fields. Numbers, dates, names, and addresses were welcomed. Notes, comments, and documents were exiled to live either outside of the database or in unsearchable (and often seemingly unreachable) places within it.
Every time you log-on to your favorite donor management system you experience the consequences of this “Unstructured Data NOT Welcomed†legacy. You search for a person’s name, and if that person is mentioned in a note rather than having his or her own record, you are out of luck. You remember reading a great profile of the person, yet you can’t find it because it was created in MS-Word – one of the biggest prisons for unstructured data.
Philanthropy has been especially hard hit because so much of what it does can’t easily be structured. Think of all the visits with donors and the Contact Reports detailing how the visits went. How much value was lost as you shoe-horned your impressions, feelings, and insights into your database? For too many organizations the simple fact of a visit happening is all that is measured without regard to how it went.
What about all the external data purchased from screening vendors? Was it reduced to a few “attributes†so you could pretend it was integrated with your “database-of-record”? Speaking of DOR, can we make it RIP please?! Your database-of-record is only a record of quantifiable data like donations, and activities. It’s like a cash register – Counts your money, but has no ability to measure what is really making it ring.
And what about social media? According to Gartner 45% of spending on Big Data over the next few years will be driven by social analytics and intelligence. Where does all that data go?
On the mission side of the house where do you store information about impact? Where are the reports from the field? As donors demand more evidence their gifts make a difference, how much evidence is hidden within attachments: a technological innovation invented to keep unstructured thinkers (meaning most of us) at bay while engineers sought to bring structure to everything.
They have failed miserably. 80% of the world (and I would say even higher for those of us in philanthropy) is just beyond our reach. We are left to find our way based on transactional data, data which is misleading at best, and dangerous at worse.
It is not without some irony IBM, the company which first brought us the technology to separate the structured from the unstructured world, is the one now telling us the future belongs to those able to make sense of what their now out-dated technology cannot.
Big Blue has embraced Big Data. It is our opportunity to reclaim all the knowledge lost as we dutifully entered our feelings, emotions, and thoughts into unfeeling, unemotional, and thoughtless databases. It is our chance to free ourselves from searching, and begin finding what matters.
For philanthropy, Big Data provides a platform to enable missions to connect at a deeper level with donors, and to dramatically increase the impact on those it serves. That is an unstructured world I would like to live in – one I believe we have been living in all along.