Dr. Watson Says: Knowing about precision and recall will help you guide your team to build the best models for your business. What Do You Say? Click Here to Send Us Your Comments Click Here to See Reader Feedback

This was co-authored with Sam Hillis and Sara Lewis of Opex Analytics.  Their biographical information can be found at the end of the column.

Recent advances in computing and availability of data have driven an explosion of enthusiasm around predictive analytics. As these algorithms and technology spread through companies and industries, it has become increasingly important to have knowledge of key concepts that extend beyond buzzwords.

There are two standard types of predictive analytics.  The first type, which you are likely familiar with, is regression.  In regression, you are predicting a numeric value.  You have seen regression in action when forecasting sales for the next month or year.  The other type is classification.  You are likely to start to encounter this more as your organization embraces analytics. Classification is predicting a category or whether an event will happen. 

Classification algorithms are likely to help the supply chain with the following types of problems:

Based on the types of problems we are looking at, it’s clear that these predictive analytics algorithms are soon going to play a more important role in the supply chain.  But, how do you work with the technical team to make sure the model is as good as it can be for you? With regression, we know to ask about the forecast errors and statistical significance of the variables.  With classification algorithms, we have new metrics to ask about:  precision and recall. To explain precision and recall, we have found the fishing example to be helpful.  In this example, we have a pond of fish and we know the total number of fish within.  Our goal is to build a model that catches red fish (we may say that we want to ‘predict’ that we catch red fish).  In our first test we have a model that consists of two fishing poles with bait made from a recipe based on scientific analysis of what red fish like.  The precision metric, is about making sure your model works accurately (or that the predictions you make are accurate).  With our fish example, this means that the fish caught with the special bait are, in fact, red.  The following test shows great precision—the two fish caught were both red.   We were trying to (or predicted we would) catch red fish, and all of the fish that we caught were red.

There is one small problem here though.  We knew there were a lot more fish in the pond and you might notice that when we looked closer we also found a lot more red fish that we didn’t catch.  How can we do a better job of catching more of the red fish?

Here is where our other measure, recall, comes into play.  Recall awards us for catching more of the red fish.  In our first model we didn’t do a very good job of this.  So although our precision was good our recall was not so good.

Knowing this we decide to develop a new model using a fishing net and a new bait recipe.  The picture below shows the result of our first test with this new model.  We caught more of the red fish!   Our recall has definitely improved. 

This is the fundamental trade-off between precision and recall.  In our model with high precision (most or all of the fish we caught were red) had low recall (we missed a lot of red fish).  In our model with high recall (we caught most of the red fish), we had low precision (we also caught a lot of blue fish).

When building a classification model, you will need to consider both of these measures.  Trade-off curves similar to the following graph are typical when reviewing metrics related to classification models. 

For a given model, it is always possible to increase either statistic at the expense of the other. Choosing the preferred combination of precision and recall can be considered equivalent to sliding a dial between more or less conservative predictions (i.e. recall-focused vs. precision-focused). It is important to note that this is for a given model; a better model may in fact increase both precision and recall.

In choosing the correct balance of precision vs. recall you should take a closer look at the problem you are trying to solve.

Let’s relate this back to our supply chain problems, if we are predicting truck driver accidents, we may want a high recall (and be ok with low precision).  That is, we want a list that captures all the high risk drivers.  We can then do extra training and extra monitoring.  And, we are ok that this list may also include a lot of drivers who wouldn’t have had accidents anyway.  Our money spent on training and monitoring these already good drivers is worth it if we prevent just one severe accident.

On the other hand, if we are predicting stock outs, we may go for precision.  If 200 of my 5,000 items will stock out next month, I may want high precision.  That is, I would be happy if you gave me a list of the high precision list of 60 SKUs most likely to stock out.  I’ll expedite and take extra measures with these 60 SKUs.  I’ll still miss 140.  But, that is better than the model giving me a list of 600 SKUs.  The list of 600 has most of the 200 in there, but I’m spending time and money on 400 items where there wasn’t going to be a problem.  

Final Business Thoughts

Managers will soon see more and more classification problems.  Knowing about precision and recall will help you guide your team to build the best models for your business.  

Extra Material:  Details on the Calculations

To truly understand the calculations, we need to understand the following conditions: 

• True Negative: we predicted that we wouldn’t catch a blue fish and we didn’t
• False Negative: we predicted that we wouldn’t catch a blue fish and we did
• True Positive: we predicted that we would catch a red fish and we did
• False Positive: we predicted that we would catch a red fish and we didn’t

Precision is the percent of times that we predict an event will occur and it actually does; for example, with the net we catch 17 fish, but only 6 are actually red giving a precision of 6/17 ≈ 35%: (true positives / (true positives + false positives)).  Recall is the percentage of events that occur that we predict will occur; again, with the nets we catch 6 of the 8 total red fish giving a recall of 6/8 = 75%: (true positives / (true positives + false negatives)).

Sara is also co-author of the book “Supply Chain Network Design,” which was published in 2012 and is being used by practitioners and a number of universities. She is a frequent guest lecturer on the topic of Network Design at several prominent U.S. Universities.

​Sara Lewis graduated from Penn State University with degrees in Business Logistics and Management Information Systems. - See more at: http://opexanalytics.com/about-us/#sthash.tECjnh4D.dpuf