So how do you evaluate alternatives between different options, especially between different vendors for technology, consulting, outsourcing, etc.?

If you're like most of us, you usually use some form of "weighted average" comparison. That is, you select a number of criteria (functionality, price, viability, etc.) and assign weights to each category. Score each vendor candidate, multiply by the weight factor, add them up and declare a winner. Companies often use this method to select a consulting firm to work with, only to have the consultant then use a weighted average methodology to evaluate supply chain software vendors.

I used this approach to select vendors myself a number of times, and have been on the other side of it even more so. I've also worked with lots of consultants I respect deeply who use this technique.

Nonetheless, it's a pretty lousy way to select a vendor. At least that's what Doug Hubbard, of Hubbard Research says, and he makes a pretty good case. I've known Doug for several years, and he's one of the smartest guys around. His specialty is helping companies make better decisions around IT investments using decision science principles and techniques (probability analysis, risk-return, etc.). I actually talked to Doug regarding our soon to be completed work on ERP versus best-of-breed supply chain applications, and we wound up on this topic during one of many tangents. I thought it was worth spending some time with his thinking on these pages.

Weighted average approaches "are probably the least likely technique to improve decisions," according to Hubbard. Why?

	The fundamental concept is flawed: Among many examples, the weightings aren't factually based - who says, for instance, that "price" should get a weighting of 30%? The ratings (say a scale of 1-5 for each category) don't really reflect the differences in vendors. Is a vendor that receives a 4 really twice as good in that category as one which gets a 2? That's how the math comes out. Hubbard says tests have proven that changing the score range (say from 1-5 to -2 to +2), even though the math works out the same, significantly changes the way people will score vendors.
	Conflicts of interest : People completing the scoring often have biases and vested interests in the outcome.
	People frequently override what the answer the method gives to get the answer they want : Any of us who have been around have seen it - if the weighted score doesn't give the answer that someone hopes, either the ratings or the weighting are changed to get a new answer.
	People really don't trust the answers they get : It's really just taken as one input to the decision process.
	It often doesn't include real decision criteria : Let's face it - we often choose vendors because we just feel more comfortable with them or their team. But you never see this as one of the categories in a weighted average, along with many others. Risk, a very important factor, is often not included explicitly.

So, while the effort to fill out the weighted average spreadsheets may in fact elicit some useful information, the formula itself really has little practical value as a decision tool.

What's the alternative? Well, there's the rub. For most of us, flawed as it may be, the weighted average approach at least provides some framework for making comparisons. Hubbard in fact does have a "better way," which you can learn about on his web site. We're also happy to announce that you will soon find out more on these pages, as Hubbard is going to be providing a regular column for SCDigest on how to make better decisions about technology. I know you'll enjoy it.

Do you think weighted average methods lead to good answers in evaluating vendors? What are better alternatives?  Let us know your thoughts.