I have promised for a while a column on so-called "Big Data” and the related category of advanced analytics, but have been waiting for my own thoughts to come together a bit on this before I obviously could hope to shed some light on the topics for readers, which I now feel prepared to do.

So let's start with this: In my opinion, there is a lot of humbug and hype out there right now relative to Big Data, and a lack of clarity about where the applications/opportunities are in the supply chain. That said, I think there really is something here, but largely still emerging.

At the same time, unquestionably big advances are being made in the area of analytics, and they are certainly getting more "advanced,” though again there are some definitional issues here about what is really new and what separates "advanced” analytics from the kind we have had for many years.

And all this is further muddled by product and service providers of all types who see big dollars in Big Data and analytics, and hence are driving the hype and often not all that worried about precision of terms and concepts.

Clear as mud?

To me, Big Data implies that there is a data set that is so large, and with initially unclear relationships among the elements, that different sorts of analyses have to be performed on it versus how we would normally sift through the data.

So, for example, a couple of years ago we reported on a fantastic transportation scorecard system that Sears has built that makes all of its extensive data on transportation available in innumerable ways. That powerful system allows the company to understand performance, answer logistics cost and service questions and a lot more. (See Sears Builds Powerful Transportation Scorecard, and Shares Experiences on How to Get it Right.)

What Sears has created is one of the best if not the actual best such scorecard system out there in logistics, involving a substantial development effort. It validates the point that has been made for years that there is great opportunity in exploiting the data that transportation specifically and supply chain systems more generally spit out on a daily basis, but which few companies well leverage even today.

But to me, it is not Big Data. A lot of data for sure, but pretty straightforward and understood. While there was some trial and error developing the system, the information that managers and executives need was fairly well-understood. And I would add that I think the analytics, great as they were, were not "advanced” in the sense of something new and better than what we have had in the past. It is a business intelligence tool, albeit a great one.

There was a panel discussion on Big Data at the CSCMP conference last fall, of middling value. The panelists frankly struggled to differentiate Big Data projects from traditional ones, and led me to conclude that we are really talking about a continuum here, not sharp lines of demarcation. That in turn led me to conclude that everything in the middle of that continuum can be called Big Data or not. Just be aware. Some find advantage in putting a Big Data wrapper around traditional techniques.

But the panel had some useful discussions as well. For example, Ron Volpe of Kraft said his company (as are others) is working on tying together mountains of data from POS sales, promotional execution, social media and more to better understand their relationships, and if the resulting insight can lead to improvements in the ability to predict and then shape consumer demand.

Kraft did not know if this was possible, he said, but he believed the disadvantage a company in the consumer goods sector would be under from not breaking this code while its competitors did would be so huge that it was a major business risk to not pursue this Big Data effort.

Gary Whicker from JB Hunt gave one of the better examples of Big Data at work at the same session, describing how the carrier used it to reduce accidents by its drivers. The company uses on-board data recorders to capture basically every move a truck makes, connected to GPS data as well. That's a lot of data, as you can imagine. It then correlates what driver behaviors are most connected to accidents - for example, what they do driving onto off ramps from the highway.

It then incorporates that insight into driver training programs - and monitors drivers on a go-forward basis looking for those tending towards those dangerous behaviors, who then receive counseling. That's pretty interesting.

Penske Logistics was working on a concept five years ago or so that I will equate to something like statistical process control for large, complex supply chains. The idea was that as all a company's logistics events occurred throughout a day or process, you would in some cases start to see delays or variability early on that could be predictive of larger issues or failure later. That's interesting, and certainly related to my notions of Perfect Logistics. I am not sure if anything more came out from Penske related to this concept, but it was certainly Big Data-like.

One point that is important to note is that these Big Data systems and capabilities are not some genius savants such that you just need to push a button and magically hidden relationships between variables are discovered. What to look for has to be programmed, and that can be a big effort, and may have to be done several times until the right insight is finally achieved. Humans have to tell the machines what to look for amongst all this Big Data. Every case is different.

The "Internet of Things” may certainly offer interesting and valuable opportunities for leveraging Big Data as well - and that data will be really big indeed. Gartner just predicted there will be 26 billion connected things - pallets, machinery, and more - by 2020. This will be a radical change for many reasons, in part because today the source of most data that comes into the Internet is generated by people. Now things will be providing the data, and tons of it, about where they are, their environment, their condition, etc.

How will we make sense of all that data, and can value be obtained from analyzing it? In some cases the answer will certainly be Yes - and indeed some companies in asset-intensive industries are already doing so. But how to get there for many others is still unclear. What will be a very interesting dynamic, I believe, is when multiple parties all have visibility to this same data.

I will now make this observation: Big Data is of little use without advanced analytics. But analytics of all sorts, whether traditional or advanced, can be valuably deployed with just regular data.

So I am going to wrap it up here for this week. In part 2 of this series in a few weeks, I am going to look at the analytics side of this in more detail, and cover a bit of a great new book on this topic from our friend and SCDigest columnist Dr. Michael Watson.

If you are doing or thinking of anything cool or noteworthy with Big Data or analytics, I would love to hear about it, privately of course as required

What do you think of "Big Data?" Real deal, or too much hype? Where do you see the best applications in supply chain? Let us know your thoughts at the Feedback section below.