Dr. Watson Says: ...the three to six man months is more than you will spend cleaning supply chain data, but not much more... What Do You Say? Click Here to Send Us Your Comments Click Here to See Reader Feedback

All the talk and hype around Big Data has put the pressure on managers in the supply chain to do more with the data you have.  But, what often frustrates managers is that the data you have is not clean enough or in the right format to answer the questions you want.

I was recently reminded that the problems we face in answering supply chain problems pops up in other areas as well:

First, in a recent Freakonomics podcast (around the 18:00 minute mark), Steven Levitt, who is also a partner in a consulting firm that answers questions with data- not unlike supply chain questions, made two interesting points:

The last point is the one that we see when we are pulling together supply chain data.  We’ll have different demand files for different divisions, different transportation data from different parts of the business, and different production data.  And, nothing will match up.  Creating a coherent data set will take work—the three to six man-months is more than you will spend cleaning supply chain data, but not much more.

Second, a New York Times article came out called “For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights.”  The article discusses the importance of data cleaning (or “data wrangling” or “data janitor work”).   Like the Steve Levitt quote, the article reminds us of the strategic value of data and the fact that it is difficult to clean it.  I liked the following sentence—it highlights that you have work to do before you get good answers:

The above podcast and article remind us that the problems we have with cleaning supply chain data are the same ones faced by other people in different industries.  In my experience, when doing a supply chain study, you should plan on spending about 60-70% of the total time developing a clean data set.

Final Thoughts:

You could read this article as a depressing reminder that we haven’t come all that far in our access to clean data.  However, I have a different outlook:  the need to answer questions with data won’t go away and access to new data sets won’t go way.  Instead of worrying about the difficulty of getting clean data, build skills on your team so you can create clean data sets and come up with new insights faster than the competition.