First Thoughts
  By Dan Gilmore  
  April 10, 2014  

Big, Big Supply Chain Data


I have promised for a while a column on so-called "Big Data” and the related category of advanced analytics, but have been waiting for my own thoughts to come together a bit on this before I obviously could hope to shed some light on the topics for readers, which I now feel prepared to do.

So let's start with this: In my opinion, there is a lot of humbug and hype out there right now relative to Big Data, and a lack of clarity about where the applications/opportunities are in the supply chain. That said, I think there really is something here, but largely still emerging.

At the same time, unquestionably big advances are being made in the area of analytics, and they are certainly getting more "advanced,” though again there are some definitional issues here about what is really new and what separates "advanced” analytics from the kind we have had for many years.

Gilmore Says:

Big Data is of little use without advanced analytics. But analytics of all sorts, whether traditional or advanced, can be valuably deployed with just regular data.

Click Here to See
Reader Feedback

And all this is further muddled by product and service providers of all types who see big dollars in Big Data and analytics, and hence are driving the hype and often not all that worried about precision of terms and concepts.

Clear as mud?

To me, Big Data implies that there is a data set that is so large, and with initially unclear relationships among the elements, that different sorts of analyses have to be performed on it versus how we would normally sift through the data.

So, for example, a couple of years ago we reported on a fantastic transportation scorecard system that Sears has built that makes all of its extensive data on transportation available in innumerable ways. That powerful system allows the company to understand performance, answer logistics cost and service questions and a lot more. (See Sears Builds Powerful Transportation Scorecard, and Shares Experiences on How to Get it Right.)

What Sears has created is one of the best if not the actual best such scorecard system out there in logistics, involving a substantial development effort. It validates the point that has been made for years that there is great opportunity in exploiting the data that transportation specifically and supply chain systems more generally spit out on a daily basis, but which few companies well leverage even today.

But to me, it is not Big Data. A lot of data for sure, but pretty straightforward and understood. While there was some trial and error developing the system, the information that managers and executives need was fairly well-understood. And I would add that I think the analytics, great as they were, were not "advanced” in the sense of something new and better than what we have had in the past. It is a business intelligence tool, albeit a great one.

There was a panel discussion on Big Data at the CSCMP conference last fall, of middling value. The panelists frankly struggled to differentiate Big Data projects from traditional ones, and led me to conclude that we are really talking about a continuum here, not sharp lines of demarcation. That in turn led me to conclude that everything in the middle of that continuum can be called Big Data or not. Just be aware. Some find advantage in putting a Big Data wrapper around traditional techniques.

But the panel had some useful discussions as well. For example, Ron Volpe of Kraft said his company (as are others) is working on tying together mountains of data from POS sales, promotional execution, social media and more to better understand their relationships, and if the resulting insight can lead to improvements in the ability to predict and then shape consumer demand.

Kraft did not know if this was possible, he said, but he believed the disadvantage a company in the consumer goods sector would be under from not breaking this code while its competitors did would be so huge that it was a major business risk to not pursue this Big Data effort.

Gary Whicker from JB Hunt gave one of the better examples of Big Data at work at the same session, describing how the carrier used it to reduce accidents by its drivers. The company uses on-board data recorders to capture basically every move a truck makes, connected to GPS data as well. That's a lot of data, as you can imagine. It then correlates what driver behaviors are most connected to accidents - for example, what they do driving onto off ramps from the highway.

It then incorporates that insight into driver training programs - and monitors drivers on a go-forward basis looking for those tending towards those dangerous behaviors, who then receive counseling. That's pretty interesting.

Penske Logistics was working on a concept five years ago or so that I will equate to something like statistical process control for large, complex supply chains. The idea was that as all a company's logistics events occurred throughout a day or process, you would in some cases start to see delays or variability early on that could be predictive of larger issues or failure later. That's interesting, and certainly related to my notions of Perfect Logistics. I am not sure if anything more came out from Penske related to this concept, but it was certainly Big Data-like.

One point that is important to note is that these Big Data systems and capabilities are not some genius savants such that you just need to push a button and magically hidden relationships between variables are discovered. What to look for has to be programmed, and that can be a big effort, and may have to be done several times until the right insight is finally achieved. Humans have to tell the machines what to look for amongst all this Big Data. Every case is different.

The "Internet of Things” may certainly offer interesting and valuable opportunities for leveraging Big Data as well - and that data will be really big indeed. Gartner just predicted there will be 26 billion connected things - pallets, machinery, and more - by 2020. This will be a radical change for many reasons, in part because today the source of most data that comes into the Internet is generated by people. Now things will be providing the data, and tons of it, about where they are, their environment, their condition, etc.

How will we make sense of all that data, and can value be obtained from analyzing it? In some cases the answer will certainly be Yes - and indeed some companies in asset-intensive industries are already doing so. But how to get there for many others is still unclear. What will be a very interesting dynamic, I believe, is when multiple parties all have visibility to this same data.

I will now make this observation: Big Data is of little use without advanced analytics. But analytics of all sorts, whether traditional or advanced, can be valuably deployed with just regular data.

So I am going to wrap it up here for this week. In part 2 of this series in a few weeks, I am going to look at the analytics side of this in more detail, and cover a bit of a great new book on this topic from our friend and SCDigest columnist Dr. Michael Watson.

If you are doing or thinking of anything cool or noteworthy with Big Data or analytics, I would love to hear about it, privately of course as required

What do you think of "Big Data?" Real deal, or too much hype? Where do you see the best applications in supply chain? Let us know your thoughts at the Feedback section below.



Recent Feedback

Happy to see you are trying to wrap your mental arms around this muddled mess of Big Data.

Every year we have more data – but how much of it is useful, and how much of it becomes information, is the real question.

Data becomes information the moment it answers the question asked.  Until then, it is just data, nothing more.  Better analytics may help us sift through the data faster, but the best of the analytics packages are constrained by something that technology will never improve; the ability to ask the right question.

Data is the fuel that drives execution improvement.  Information is the fuel that drives tactical improvement.  Questions are the inspiration that drives Strategic improvement.  We need data to measure and improve execution performance.  We need Information to decide on the tactical options available, and to develop new versions of tactics to deploy.  These are all good, but create marginal improvements.

Until we start asking better questions, as in the questions focusing on The Goal of the Supply Chain, we won’t improve our strategic strength of our supply chains.


David K. Schneider
David K Schneider & Company
Apr, 11 2014

Dan congratulations on your first part of 'big, big supply chain data'.

I would like to share one example of big data which I consider very helpful to avoid human casualties these days as to prevent catastrophic floods in particular risk regions.

In summary, programmed software analyze periodic data of mapped flood risk regions as depth level of nearby water reservations and watercourses (as rivers) and rain levels occurred and forecasted.

Afterwards, compare periodic data results with risk levels parameters and using this to plan contingency safety measures on these flood risk regions.


At high flood level risk would demand an evacuation plan.

Also would like to share a reflection of Dee Hock on his book ‘the chaordic organization’…would go as remember as …. “as time goes by data will begin to transform in information, information in knowledge, knowledge in comprehension and after long time could become wisdom”

I wish you good writing on this multiple perception theme.

Daniel Camargo
Senior Manager Consultant
Apr, 11 2014

I like and agree with your article completely.

I have had a vision since early 2008 that is so related to supply chain and technology, and the times and it starts off slowly and grows as people see the advantages it has not only in businesses but in employment and the economy.  It involves data proper usage, visibility, collaboration, and a new approach that actually isn't new for all.  I won't go into details in this email. 

I am industrial engineer and have been a supply chain consultant for 20 years.  I am also an inventor.

Shelley Jordan
Apr, 11 2014

Nice article and, I think, one of the more balanced ones I have seen recently on Big Data.  

There is definitely a definition problem around "Big Data".  To many folks it's just data that is too big to handle with their current infrastructure.  If their current infrastructure is Excel (as it is for many) that's not very big at all in absolute terms.

As the data volume increases it becomes increasingly difficult to handle it using conventional server tools too; Point of Sale data is an excellent example and one I am very familiar with.  Compared to most data sets a CPG can get their hands on, POS data is huge and they struggle to do anything with it other than some basic reporting.  Their analytic tools (or perhaps analytic skills) are not capable of handling this volume of data close enough to real-time so it does not get done.  This is a real, big problem and a lot of lost value.

Most big data proponents though would still argue that we are not talking about big data unless we are looking at high velocity, (relatively) unstructured information like log files, or social media feeds.  

As you mentioned, Kraft, I think, are working along the right lines and it is possible to get real value from POS data in improved, short-term forecasting accuracy.  I think the jury is still out on just how much value can be derived (for supply chain purposes at least) from social media data and sentiment analysis.  My gut-feel is that it may help explain why sales suddenly changed at retail but I don't think it will help much in forecasting that change.

FYI - I've already got Dr Watson's book and I look forward to reading it on my next flight.  If it's to the same high standard as his last one on network optimization, it should be a good read.

Andrew Gibson
Crabtree Analytics
Apr, 11 2014

I read your article and find it very insightful. I am compelled to agree with your point of view for various reasons.

In my opinion, all of us have the data at hand on consumer behaviour and are trying to plug in an equation that would propel big data into business the way software applications on Apple and Android offered quantum growth in application development. That being said, the difficult component would be in arranging consumer goods and services in a matching fashion while not jeoparadising the current business.

Was great reading your review again.

Business Consultant
Apr, 13 2014

I completely agree that the value from Big Data is of little use without advanced analytics.  The value from Big Data comes from gleaning insights in time to take smarter actions. 

A key distinction related to Big Data is about the nature of the data itself.  Unlike traditional applications of performance management that might provide KPI scorecards over large amounts of structured data, Big Data includes unstructured data such as social media chatter.  You alluded to this as well.  To get insights out of unstructured data and structured data we need algorithms that cluster and detect patterns in the data and then machine learning to decipher relationships between these patterns.  It is this process of putting some structure into unstructured data in large scale that distinguishes advanced analytics as applied to Big Data.

Today, the Internet of Things is largely dominated by simple use cases where a sensor detects some parameter to be in a certain range and triggers a message across the internet causing some action to be taken.  A car signalling the dealership that it needs maintenance is one such example.  In my opinion these use cases are the tip of the iceberg and much more interesting sense and respond examples will follow.  I believe that sensors will soon go beyond physical sensors to also include logical sensors - such as a statistical process control chart showing an out of control trend for a key metric.  There will be sensors on unstructured data as well - such as a product's positive or negative sentiment trending beyond defined thresholds.

Big Data in supply chain will be all about turning structured and unstructured data into insights and then using those insights to make timely and profitable execution decisions across the supply chain.

Adeel Najmi
Chief Science Officer
JDA Software
May, 02 2014

Was there a part 2 of this series. Is there more clarity in some of the applications of Big Data in the supply chain? I would imagine Retail would a good candidate, given the push for omni-channel, Ecommerce and the amounts of data it generates.

Are there any innovative companies that are further ahead in developing solutions in this area?

Kumar Rajagopalan
Director of IT, Merch
May, 28 2015