This page contains an introduction to some of the statistical methods we use when evaluating charities, in order to ensure that our assessments are as complete and accurate as possible.
Future events and probability
When we assess a charity, we are looking at the impact that further donations will make. However, once we have worked out what a program is meant to achieve, there are also risks which need to be taken into account: Corruption, misappropriation of resources and simple administrative error, to name a few. We need a way to reflect the uncertainty created by these risks in our assessment.
To show how we do so, consider the example of charities A and B, both of which are running new campaigns:
- If charity A’s campaign is successful it will do 20 QALYs of good for every £1,000 donated. Although the campaign is new, it is based on past successes and is well researched. We assess the chances of its success at 90%.
- If charity B’s campaign is successful it will do 30 QALYs of good for every $1,000 donated. The campaign is ambitious, but there are a number of risks which could cause it to fail. We assess the chance of success at 50%.
To compare these campaigns, we multiply the potential good done by the probability of it happening to get our final figures:
- Charity A: [20 × 90% =] 18 QALYs per $1,000 donated.
- Charity B: [30 × 50% =] 15 QALYs per $1,000 donated.
As a result, we can see that at this point charity A is the better prospect, even though it’s campaign is less ambitious. Of course, if charity B is successful with its campaign, then the chances of it continuing to be successful will rise, and the calculation will change.
Regression to the mean
Since we are looking for the most effective charities, we are naturally drawn to the ones which have had a high degree of recent success. However, we have to be cautious about this, especially if the success has only been for a relatively short period of time.
The reason to be careful is that there will always be a random element to success. In any given time period, different charities will have better or worse luck. If we simply recommended the charity with the best recent figures, it could well be that they have just had a lucky few years. If so, over the next few years they are likely to be less successful and “regress to the mean”.
In order to be sure that we recommend charities with effective methods rather than just those with recent good luck, we look for long-term success and frequently re-evaluate them to check that they are still as effective as originally believed.
Measurement and error
The difficulties with measuring effectiveness mean that there might be errors in the data we consider in order to assess charities. Again, this creates the risk that an effective-seeming charity is actually just one where the measurement errors came out in its favour. Unlike the problem discussed above, these errors may well be the sort to re-occur year after year.
In order to combat this, part of our research is focussed on working out the extent of these measurement errors. In effect we work out the ‘margin of error’ around our results. If the margin is particularly wide and the difference in effectiveness between different charities is particularly small, then there is a risk that the apparent differences are nothing more than measurement errors.
Fortunately, this is not the case for our recommended charities. They appear significantly more effective than the other charities we have considered, and although there is a margin of error, it is relatively small. As a result, we can be sure that they really are more effective, and we can recommend them with confidence.
The conclusions we reach using this information is that there are four standout charities we recommend. Find out about them here