Methods for scoring Cost and implications of relative scoring

My last blog showed that if we combine the financial and non-financial elements of a bid evaluation using a Cost criterion with a weight, we need to carefully consider how we will prevent unaffordable solutions from winning the competition.

This is because if we don’t allocate sufficient weight to Cost – to ensure that an unaffordable solution will lose sufficient points that it cannot catch up even if it performs very strongly in the non-financial evaluation – we need another way of preventing it from winning.

This could be a maximum acceptable cost, but this can be problematic for many reasons.

But there is another implication of using a weighted Cost criterion that is the subject of this blog.

It’s that if we use a weighted Cost criterion we will need a mechanism for converting the costs of each bid – which will be measured in pounds, dollars or some other currency – to scores so that they can be combined with the non-financial scores to arrive at an overall score.

This blog will explore some different options for the scoring of Cost and their implications.

Converting Costs (in currency) into scores

There are a great many methods that can be used to convert Costs (in currency) into scores.

Stilger, Siderius and Van Raaij identified 38 distinct methods for combining financial and non-financial factors and a number of different ways of scoring Cost in their paper published by the Journal of Public Procurement (Spring 2017, Volume 17).

This list was by no means exhaustive, but it did include most of the methods that we’ve seen being used by our clients.

Given the range and variety of possible approaches, it wouldn’t be practical to explore all of them in this blog.

But it is useful to look at the different types of approaches and some of their characteristics and implications.

converting costs into scores.png

One major difference between cost-scoring methods is whether the way that each bid’s Cost is scored depends on the Costs of other bids.

If the way that a bid’s Cost is scored does not depend on other bids’ Costs we call it absolute scoring; if it does depend on other bids’ Costs we call it relative scoring.

What about relative scoring methods?

One commonly used relative scoring method gives full marks to the cheapest bid and a proportional score to the others that are more expensive – so a bid that is twice as expensive as the cheapest gets half the marks.

Another relative scoring method gives full marks to the cheapest and zero marks to any bid that costs twice the cheapest or higher.

The use of any type of relative scoring method comes with significant drawbacks.

Using a relative scoring method means that a buyer will only know how a particular Cost will score after it receives the bids, knows the Costs of all bids, and therefore how the scoring method works in detail.

This means that this buyer will not know at the time they issue their invitation how they would end up scoring the hypothetical Costs of various bids and therefore what the difference in scores would be, for a given difference in Cost.

For example, the buyer cannot know what difference in score would result from one bid being £1m more expensive than another.

This makes it much more difficult for the buyer to ensure that the scoring method is going to have the desired outcome and means that the buyer (in theory at least) cannot work out in advance which of two solutions is better that the other.

It also stands to reason that each bidder cannot know at the time they submit their tender how much their customer would value cheaper or more expensive solutions they could put forward, because they too do not know (with certainty) how a difference in Cost would translate into a higher or lower score.

The implication of this is that bidders may not be able to work out which of their options is the best to offer to their potential customer.

Let’s look at an example to show this in action:

Let’s suppose that we are running a competition where the technical criteria are weighted 60% and cost is weighted 40%.

We receive three bids and evaluate them technically.

The three bids with their technical scores (out of 60) and their prices are shown below.

technical/quality 41/60 blue
technical/quality 54/60 purple
technical/quality 32/60 pink
price $50m blue
price $80m purple
price $40m pink

Now let’s suppose that we have chosen to use the relative cost scoring method using the formula below, that gives top marks to the cheapest and a proportional score to the rest.

price of cheapest solution / price of solution being scored

We can now score the prices of each solution (out of the 40 points that have been allocated to price) and calculate the overall scores.

technical/quality 41/60 blue
technical/quality 54/60 purple
technical/quality 32/60 pink
price $50m blue
price $80m purple
price $40m pink
price score 32/40 blue
price score 20/40
price score 40/40 pink
overall score 73/100 blue
overall score 74/100 purple
overall score 72/100 pink

We find that our green bid has won the competition, beating both green and blue.

But what if the blue bid was never submitted or is withdrawn?

This may appear academic or very unlikely, but bear with me for now and it’ll become clear why this is an issue…

Now we only have these two bids – and it’s important to note they are exactly the same solutions as before.

technical/quality 41/60 blue
technical/quality 54/60 purple
price $50m blue
price $80m purple

Again, we can use our published cost scoring function to score the prices and calculate an overall score.

technical/quality 41/60 blue
technical/quality 54/60 purple
price $50m blue
price $80m purple
price score 40/40
price score 25/40 purple
overall score 81/100 blue
overall score 79/100 purple

Now we find that the blue bid has won the competition and has beaten purple.

Let’s think about this for a moment.

From the buyer’s perspective, at the time they published the invitation (including the scoring method) they could not know whether a blue solution was better than a purple solution, or a purple solution was better than a blue solution.

It depends on whether they got offered a pink solution – which was never going to win anyway!

So, the winner of the competition is not determined by the buyer’s preference for whether blue or purple is a better solution – it’s determined by the solutions that each bidder chooses to offer.

And the bidder’s perspective?

Let’s also consider the bidders’ perspective. OK, they cannot know for sure how their solution will be scored, which doesn’t sound too bad (from a buyer’s perspective).

But they also might not know which is the best solution for them to bid with.

Imagine that the blue and pink solutions were actually two different options that a single bidder was trying to choose between to bid with.

This bidder cannot know in advance which is better, and their choice will may be informed by a pure guesswork or a combination of assumptions and intelligence about their competitors as well as game theory analysis.

So, what the buyer ends up receiving from each bidder may be a solution determined partly by guesswork and game theory that may not be each bidder’s best solution to the buyer’s requirement.

And the buyer chooses one of those as the winner in a way that means they cannot, in advance, always know which of two solutions would be the winner.

What about other relative cost scoring methods?

Although the example above is based on a particular relative cost scoring method, the same effect can be demonstrated in any relative scoring method, whether it’s applied to cost or technical.

So other relative cost scoring methods, such as comparing costs to an average, also come with the same implications.

In my next blog I’ll continue to look at different types of cost scoring method and examine the difference between linear and non-linear methods.

Sign up to our newsletter to stay up to date on the latest articles, blogs, events and more.

Our AWARD® Software supports complex strategic procurements