Owen Cotton-Barratt is Giving What We Can’s director of research, and also works on the Global Priorities Project (GPP)  within the Centre for Effective Altruism. The GPP works to try to find out how we should compare the different causes that we could be working on. This is closely tied to Giving What We Can’s aim of finding out how individuals can help people the most with their donations.
Owen’s current project is developing a framework to treat problems of unknown difficulty. It’s a work in progress, but he wanted to see whether the kinds of results the model produced would be useful by working through a specific example – the value of medical research. In this post I present a brief sketch of both his model and his results. Note that this work is very preliminary, and my statement of it is rough!
You need to build a simplified model that can start to give us a handle on what’s going on, but that gets the major effects approximately correct. This model aims to find a best guess for the average benefit (measured in Quality Adjusted Life Years (QALYs)) produced by an extra unit of resources (measured in dollars) invested in medical research, as well as some idea of how robust this estimate is. It starts by finding the general shape that returns to medical research would have, and then uses rough numbers about the world’s actual research spending and life expectancy increases to find out how many QALYs we might expect to generate per million dollars we put into medical research.
In research in preparation, Owen argues that we should think the beneficial knowledge from a research area will scale approximately logarithmically, in expectation, with the resources invested. If the research area has many small discoveries rather than a few large discoveries, its development should also exhibit roughly logarithmic behaviour. In the case of medical research, this appears to be a reasonable assumption.
This means that a doubling of resources is equally good regardless of starting point. So money would have diminishing marginal value when put into research, as we might expect. Of course, this model does not hold for arbitrarily large values, but it is a fair approximation over the range we’re interested in.
The very rough model above seems to indicate that the knowledge generated by medical research varies logarithmically with money put in. That’s an interesting result, but what we really want to know is how many QALYs we could generate by donating a certain amount to medical research. To assess that, we need to ask how useful extra knowledge is. At first glance, it seems like it generates health benefits in a stream of constant size into the future, so even a small amount of extra research today will produce very large numbers of QALYs.
However, we should assess the value of extra research by the counterfactual where we don’t do this research. Whereas the obvious counterfactual considers the research never happening, the correct counterfactual is generally that the research will happen, but later. Of course the resources that would have been used to undertake this research later will now be freed up, often for further research. But we get diminishing marginal returns from research as the low-hanging fruit are taken, so although there will be a stream of counterfactual benefits going into the future, it will be decaying and its total size may well be finite.
To figure out the actual value, we need some numbers about current medical research. The numbers below are rough, as this is a proof-of-concept calculation. Where they aren’t sourced, they’re Owen’s best guesses. Putting more work into getting good data to use in the model could be a good way to produce more accurate outputs.
You might think that the benefit that medical research produces in a year will continue every year from then on, because once you’ve found a vaccine or cure you can continue to use it. However, we shouldn’t be comparing the case of us finding the vaccine with the case in which it was never found – chances are it would have been found, just later on. On the other hand, the money that would have been used to find that vaccine later on, can now be used to fund other medical research. What we have effectively done is moved medical progress as a whole forward. Therefore, the benefit from funding medical research does last into the future.
In order to find the total value of funding medical research, we need to know how long into the future our funding continues to be a benefit. Since medical funding is increasing each year, the benefit from a particular donation decays over time. How quickly this happens depends on how the benefits from medical research vary with resources put in. Using the result that this is log-linear gives that the benefit of research today decays at a discount rate equal to the rate of increase of medical funding. With a 3% annual increase, the sum of the infinite series generated is equal to 33 times the benefit generated in one year without accounting for this.
The total benefit produced by the $600billion put yearly into medical research is the number of QALYs benefit in a year (around 2 million), times the effective number of years the benefit continues for (33.3). Therefore, the total benefit is around 70 million QALYs. That means per million dollars put in, 110 QALYs are produced. That is $8,500 per QALY. Assuming that the error in our estimates by component are independent, Owen also worked out a rough 90% confidence interval, of between $2,000 and $36,000 per QALY. Accounting for model uncertainty might make it a little wider again.
The result we got – that the return of medical research is around $8,000 per QALY - is very rough, and has a high variance. However, it does give us an idea of its value, which is very useful. There are two particularly interesting things it tells us: This estimate indicates that we are under-investing in medical research. The NHS is typically happy to fund treatments up to around $50,000 per QALY. If medical research on average is more cost-effective than that, rich countries should be investing more in it.
The estimate indicates that medical research as a whole is not typically as effective as the very best direct interventions (like providing people with bed nets). Despite the initial appeal of research as something which will have far ranging ramifications, on average deworming children is more effective.
However, our estimate has been for the average effectiveness of medical research. It is quite plausible that particular research efforts are highly cost-effective, particularly if they are in fields which are likely to be neglected because the beneficiaries would be very poor (for example, developing a vaccine for malaria or schistosomiasis). With further work we could begin to make comparisons of these areas.