Some Myths of Big Data

After Lawrence Hart recommended "The Myths of Innovation" by Scott Berkun, I was curious to find out what Mr. Berkun's ideas were on innovation. The book starts out with the concept of the years of hard work it takes for our innovative breakthroughs to occur; that the moment of genius is a surprise after many long hours and infinite combinations of possibilities.

As we consider "Big Data" and its promises of indexing and searching "innovation", I wonder how of this is marketing and how much is making a "significant positive change"? Big Data in healthcare has a long way to go before it can truly state that it works. What the Big Data push is doing is forcing the architects of the first generation of EMR designs to rethink how their data is designed, OCR for scanned results and forms, metadata of content, designs of the forms online and on paper, merging of transfer patient encounters, and the use of acronyms, keywords in forms. This is just a short list of hurdles that will need to be dealt with before Big Data can be fully useful.

Bad Data in EMRs

Indexing inconsistent and spotty information makes it easier to search, but the results would still need a lot of clean up. In other words, EMRs will need to be fully combed through and corrected to produce accurately results. One issue with EMRs is that the underlying organizational structures are hard to change because of the audits and regulations that control these changes. Also, when hospitals merge, older record numbers get merged in with newer one; at the master database level the numbers are unique, however at the patient record level all sorts of interesting things can happen where duplicates number occur and short term fixes are put in place...

Some EMRs allow for free text to be entered as metadata which describes the electronic or scanned in form; great, but have you ever tried to pull any type of consistent meaning out of these free text fields? What were they thinking??

Optical Character Recognition (yep, ICR and OMR too)

For hospitals that are not fully electronic, most of them, the issue of scanning and OCR'ing their forms and results is an ongoing struggle in that the forms and inconsistently filled out, doctors and nurse hand write their notes, and the design of the forms paper oriented. The technology behind ICR for hand writing is not good enough for recognition unless the hand written letters have boxes around them.

So, as Big Data permeates healthcare systems, at what point will it really help to bring about "significant positive change"? There are many years of hard work to get this point. If you drink the koolaid, index agents can intelligently sift through all of the noise and help find and report on the information that is required for upper management dashboards, however, I believe there will be many cycles of "reforecasting" Big Data effectiveness before it is ready for primetime.