Estimated reading time: 8 minutes
Introduction
Evaluation criteria reflects a lot of different factors important to your procurement. Some will be closely tied to the performance or capability of the goods, equipment, or services you’re buying. Some are more about whether you’ll be able to get the goods or service on time and whether they will remain available to end users. Some criteria are about risk mitigation, that could be delivery and quality related risk. Or it could be commercial and financial risk, perhaps by including criteria to do with establishing acceptable contract terms and conditions. Some criteria like social value, local economic benefits, and environmental protection, are there to measure the benefits to more remote stakeholders like local communities, the nation and even the world’s population. Finally, the cost of the goods and services, which might just be acquisition cost or could be the cost of acquiring, maintaining, and operating the solution throughout its useful life.
In this blog we’ll be focussing on just one of the key elements, ‘What’s difficult about weighting?’, drawn from the recently released ‘Weighting your evaluation criteria’ webinar. You can watch the full recording free by completing the form at the bottom of this page.
Quick links
What are the most important parts?
One reason it’s difficult is because stakeholders have different views on what’s important when it comes to evaluation criteria.
Some of your criteria:
- Will be about enduring differences in the utility of the goods or services you’re buying.
- Will be about cost. This could be as simple as considering the upfront purchase cost or it might be more complicated if including the cost of owning and operating goods or if drawing down services over an extended period.
- Will be about minimising risk. This might focus on the availability of the goods or services you’re buying.
- May focus on financial risk.
- May address things like the legal and reputational risks when choosing their suppliers.
- Are likely to be about commitments made to initiatives such as social value and local or regional economic benefits.
So, when weighting, you’re trading off against each other – short term utility, longer term utility, short term costs, long term cost, risks, and opportunities, as well as certainties and possibilities. Of course, these various factors impact on individual stakeholders in very different ways. The increasing focus on social value and environmental factors, means you need to factor in the needs of stakeholders with interests further disconnected from the procurement team, such as local communities, disadvantaged groups, and the wider population. Reconciling these distinct stakeholder needs is not going to be easy.
Biases
Weighting is also difficult because people are strange social animals who are being asked to make judgements. Weighting involves social interactions, as such, it’s open to the perils of interpersonal conflict, power struggles, and status games. You sometimes fall in line with the highest-ranking person in the room or keep quiet if you disagree. Sometimes you’re so passionate about an issue that you can’t imagine any other perspective. Sometimes you trade off some favour you owe or need to advance your position.
Putting those challenges to the side, weighting also involves the application of judgement by individuals. That means that you’re opening yourselves up to biases and noise in your decision-making process and you sometimes have a hard time distinguishing between different levels of performance. Sometimes you have a solution in mind and want to reverse engineer a process to get to that and sometimes you oversimplify a problem.
Both issues, social interactions and judgement make weighting a challenge. Weighting will likely involve quite a few individual people representing your stakeholders. It’s a social process involving application and testing of individual judgement and reconciliation of disparate and conflicting views.
You need to be aware of the various social and psychological effects that can influence the outcome.
Scoring Criteria
Another thing that makes weighting difficult is having to consider how the criteria are scored when judging their relative importance.
Firstly, your criterion may be scoring an “objective” level of performance that’s above a “threshold” level of performance.
When weighting criteria, you also need to be aware of how they’re scored. This is because, depending on how sensitive your measurement of the criteria is, you may want to weight it differently.
It’s easy to lose track of what criteria you’re measuring and how they’re scored when engrossed in a weighting workshop debating the relative importance of criteria you care deeply about. It helps to have an experienced facilitator running your workshop, or alternatively to allocate someone the role of making sure this is constantly in everyone’s mind.
Managing expectations
Weighting is often difficult because you must work around expectations and constraints or, spend time persuading others that you need to do things differently.
Past procurements influence and provide a model for future ones. This might take the form of a formal criteria framework with pre-determined weights or might just be an expectation regarding the weight of a particular criteria. Or it might be that stakeholders are expecting a particular method to be used when weighting the criteria.
Either way, it’s important to not underestimate the time and energy needed to do things differently and overcome the natural friction to change. If change is necessary, it’s important that stakeholders are identified and engaged effectively, to ensure they understand the need for change and the advantages of new approaches. Alternatively, it may sometimes be best to recognise that change is not practical and work around it.
Many organisations will have a traditional or standard approach to this but there are many, many other methods of combining cost and quality.
Each of the methods for combining cost and quality have deep implications for how bidders are motivated and for what you need to think about when weighting cost.
Robust basis of comparison
Weighting is difficult because it’s not necessarily obvious what you should take into consideration when weighting your criteria.
It’s obvious that somehow, you need to consider their underlying importance, but what does this mean in practice? It must be something to do with how much each criteria supports the achievement of the procurement aims, but how do you make this objective and consistent when you’re dealing with criteria of very different natures?
There are also a few other things that you might want to consider.
For example, do you need to consider how much each criterion differentiates between bidders? We’ve seen that increasing differentiation might be important, but how much should it influence the weight of the criteria? It might seem obvious that you should increase the weight of criteria that differentiate more and decrease the weight of criteria that are unlikely to differentiate, but there’s a potential catch-22 here, as reducing the weight of criteria where you expect all bidders to perform well might remove the incentive for them to do so!
We might also want to think about and reflect in your weights, the nature of the information available to bidders and the robustness of the commitments they can make. Some criteria concern things that bidders can make robust commitments to, enabling us to track these commitments and hold bidders to account for more easily. Other criteria are the opposite, they concern things that are difficult or impossible to track under contract. Some criteria might examine highly objective, easily verifiable performance data; others might examine more subjective subjects where the bidder’s response is less easily verified.
Potentially time is another factor. Some criteria concern issues that are likely to be an immediate concern after award of the contract. There might be no time for bidders to adapt and improve their solutions before the impact of a poor proposal is felt. Or there might be lots of time for bidders to improve their proposals or engineer workarounds, meaning the impact of a poor proposal might be somewhat mitigatable under contract.
So, if you want your weights to reflect these sorts of considerations and still be objective, you will need to carefully agree the basis on which you will compare the importance of all your criteria.
How do you know if your weights are correct?
Weighting is difficult because you need a way of knowing when you have got it right.
We’ve seen that your weights are important because they will influence what solutions you’re offered and determine which of them go on to win your competition.
So, it’s important that you get them right.
And in public procurement you must declare them up-front, so you need to be confident in them at the point you publish your Invitation to Tender or Invitation to Negotiate.
But how will you know when the weights are right? “Knowing they are right” implies that you know the weights will influence bidders in the way you want and that they will result in the best solutions, hypothetical solutions at this point, scoring the highest. The hardest part of this is getting agreement on which of solutions are the best, your stakeholders may have very different views on what is important, but it can also be difficult to work out what to test.
Sensitivity analysis is normally done using scenarios, each of which is a hypothetical solution with a specific level of performance against each criterion. You ensure these scenarios test a range of realistic solutions as well as unlikely outliers and normally choose them to answer specific questions of concern.
Constructing scenarios has its complications. To make them realistic you will need to consider things like whether there is correlation between performance on different criteria. This can be positive correlation where solutions are likely to be either strong or weak against a set of criteria or negative correlation where strength against one criterion implies weakness against others.
It’s worth noting that to some extent, if you have weighted your criteria using appropriate methods, you may already have some confidence that the best solutions will score the highest because you will know that each part of your scoring model is appropriately in balance with all the others. But some methods for weighting are better than others at this. So, it’s likely that however you have weighted your criteria, you will still want to undertake some sensitivity analysis to be sure that your criteria and weights will work in the way intended.
This blog drew on just one of the key elements from the recently released ‘Weighting your evaluation criteria’ webinar. You can watch the full recording free by completing the form below.
