Giving What We Can no longer conducts our own research into charities and cause areas. Instead, we're relying on the work of organisations including J-PAL, GiveWell, and the Open Philanthropy Project, which are in a better position to provide more comprehensive research coverage.
These research reports represent our thinking as of late 2016, and much of the information will be relevant for making decisions about how to donate as effectively as possible. However we are not updating them and the information may therefore be out of date.
Cause Area: Medical Research
Note that any attempt to estimate the returns to research funding will depend on which disease is funded, and the type of funding given. All of these estimates are for a particular disease/funding type combination, and this may explain some of the variation between them.
This report models the cost-effectiveness of research into both drugs and vaccines for Malaria, HIV/AIDS, and Tuberculosis, through Product Development Partnerships (PDPs). PDPs are collaborative funding mechanisms between the private and public sector in order to develop new products for treating diseases. The model uses a cost effectiveness analysis, outputting $/DALY, is ex ante, but estimates the average effect.
I can only access a summary paper (although I’ve requested the full version), so I cannot explain the full model used, but their approach can be summarised as follows: The analysis focuses only on the direct health impact through new products developed, and doesn’t regard spillover effects from new knowledge, or the impact of advocacy work that PDPs carry out. The research process is modelled with a Markov model (a form of modelling that can account for uncertainty). This assesses the probability that drugs will get through various stages of the development process, and the costs of each stage of development, to assess the total expected cost of developing a new drug or vaccine. The model assumes that research has an impact by making healthcare spending (spending on treatment and prevention, rather than research) more efficient, so that more DALYs are averted per dollar of health spending (current public health spending is assumed to be maintained).
The cost-effectiveness of vaccines ranges from $12-$107 per DALY, and the cost-effectiveness of drugs ranges from $12-$17 per DALY. The variation between results depends primarily on public health purchasing policies. They claim that under sensitivity analysis, where they vary certain key assumptions, research remains cost-effective, and the key conclusions are not affected.
First, the paper does not account for spillovers and interactions between programmes. Second, it only accounts for disease burden in three World Health Organisation (WHO) regions. These two factors may mean that research is actually more effective than this study suggests. Third, the model inherits uncertainty from its estimates of the cost of each stage, and its estimates of the probability of a potential drug passing each stage. Fourth, the study is now 9 years old, and therefore uses outdated burden of disease figures. Finally, I have significant uncertainty about how accurate the model’s estimates are likely to be because I don’t have access to their full methodology, and so I can’t assess what factors they are accounting for. In particular, it is difficult to assess how they estimate the cost of research.2 This increases the uncertainty surrounding the estimate.
This paper estimates the cost-effectiveness of encouraging vaccine development for malaria, HIV, and TB. Vaccine development would be encouraged through an Advanced Market Commitment (AMC): a legal commitment to purchase a large number of vaccines at a certain price, which is designed to create a market for vaccines that would encourage industry to invest in research (see Appendix 2 for more details). It estimates the average, ex ante return.
Taking the market for a malaria vaccine as an example, the method is as follows. The total market needed to incentivise drugs companies to invest in vaccine development is estimated as the current mean market size for current drugs in all categories. This is estimated to be around $3bn, once some downward adjustments have been made, although this figure strangely leaves out a significant upward adjustment which the authors discuss as necessary.4 The paper then assumes that the cost of capital (interest rate) is 8%. DALYs are discounted at 3% per annum (that is, a DALY one year in the future is valued 3% less than a DALY today, and so on). It is assumed that 3 doses of the vaccine will provide protection for 5 years, and then be 60% effective. It is assumed that delivery costs are $0.75 per dose, and that it is only distributed in countries where the vaccine is cost effective (that is, where the vaccine provides returns of less than $100/DALY). Finally, they assume that vaccine uptake is 5 percentage points higher than that of an existing 3 dose vaccine.5
The results are as follows: For Malaria, assuming that the AMC means the creation of a malaria vaccine where there otherwise would not have been a vaccine at all, returns are $15/DALY. The equivalent figure is $17/DALY for HIV, and $30/DALY for a TB vaccine. Making the alternate assumption that the discovery (and implementation) date of the vaccine is only brought forward by 10 years, the cost per DALY is $23 for malaria. And if we assume that the vaccine is brought forward by 2 years, the cost effectiveness is $80-90/DALY. The cost per DALY also increases if you assume that a larger commitment is needed to incentivise vaccine research.6
Slightly less encouraging results have been developed by other organisations, using similar methodologies. The IAVI, using a similar method, calculates average cost-effectiveness under four different efficacy scenarios. When assuming that an AMC stimulates development of an HIV vaccine that would otherwise never have been introduced, they find that the price per DALY ranges from $21-$67. Making the more realistic assumption that an AMC only brings forward vaccine production (by 10 years), cost-effectiveness ranges from $26 to $96 per DALY.7 This study only examined HIV.
Many of these limitations apply not just to the ‘Making Markets’ paper, but also to the related estimates from IAVI, and the estimates of MVI and Tremonti. For simplicity, I have focused on the ‘Making Markets’ paper, but these limitations can generally be read as applying to the other papers too.
First, it should be noted that drug industry research and development costs seem to rise quickly over time, and on average increase at an above-inflation rate. This means that the market size needed to incentivise firm investment is also likely to increase at an above-inflation rate, which may mean that the estimate of average returns to a new drug is an underestimate.8 Second, developing the vaccines mentioned is likely to be harder than developing a normal drug. This will mean that a larger market size than usual will be needed to incentivise vaccine development. For instance, malaria, as a parasite, presents special challenges, and other tropical diseases present similar special challenges.9 Third, we should note that the study is now ten years old, and the burden of disease and funding landscape is likely to have changed significantly in this time, which may invalidate the results. Fourth, the estimate does not account for the fact that a vaccine will lower the burden of disease, and therefore reduce healthcare costs in the country where the vaccine is distributed. Nor does it account for broader epidemiological or productivity effects. This may lead the study to give a downward-biased estimate of the firm’s cost effectiveness. The fourth point could make research funding more effective than the study estimates, but the first two points counteract this effect: overall, it is unclear in which direction the study is biased, from these points.
Finally, we should note that this study has received strong criticism from Andrew Farlow, an economist at the University of Oxford. First, Farlow argues that, when we account for the risk of investing in vaccine development, pharmaceutical firms face a higher cost of capital - higher than 15%, and perhaps around 25%. This higher cost of capital greatly increases the size of APC needed. Second, Farlow notes that public funding may partially crowd out private funding, which he estimates may reduce the impact of a given AMC commitment by around a half. Accounting for these two factors, he argues that $1bn of AMC funding would only increase private-sector spending by $66.5m. He uses this to argue that direct funding of vaccine research may be more efficient (that is, he prefers ‘push’ funding to ‘pull’ funding: see Appendix 2 for more details), and also to argue that the terms (most importantly, the value) of an AMC should be set later on, when more is known of the appropriate size for the AMC, and when scientific risk is lower.10
In another paper, Farlow and co-authors level a more serious criticism: they allege that the value of AMC was estimated to be higher than $3bn in previous reports, but that it was lowered for reasons of ‘political expediency’: that is, to gain support for the policy. They also argue that a case study unfavourable to the report’s conclusions was deliberately excluded from later drafts.11 More generally, they argue that the appropriate value of the AMC rests on important scientific questions, which the ‘Making Markets’ paper does not address. Relatedly, the authors argue that the sample on which the $3bn figure is based does not reliably extrapolate to global vaccines, which present very different challenges. Another point that Farlow et al make is that the authors of the original study failed to account for the variability in manufacturing costs, which could be so high that the AMC failed to incentivise research at all. They also point to technical difficulties relating to intellectual property and liability. In addition, they argue that an AMC is less suitable for incentivising small biotech companies, and developing-world research companies. They also point out that the mechanism is largely untried, and that there is therefore not enough evidence to divert funds from other mechanisms to this project. Finally, they point to difficulties in specifying the appropriate minimum technical requirements for an acceptable vaccine.12 It is worth noting that throughout, the focus is on showing that AMCs are not the most efficient mechanisms for ensuring product development, rather than on showing that funding vaccines is in general not cost-effective: indeed Farlow et al. conclude by calling for continued funding, through mechanisms other than AMCs. Nevertheless, these comments lead me to think that the ‘Making Markets’ and related papers may significantly overestimate the cost-effectiveness of AMCs for the ‘Big three’ diseases. The estimates should be treated with caution.
The Copenhagen Consensus Center has produced a paper that carries out an average, ex ante cost-benefit analysis for HIV vaccine research development.
It assumes that there will be a 50% efficacious vaccine for HIV by 2030 (arguing that this is a conservative estimate), which will be distributed to the general population aged 10 to 49. The cost of full vaccination is assumed to be either $60 or $150.14 DALYs are discounted at a rate of 3% or 5% per annum, and valued at $1000 or $5000. It is assumed that the average infection occurs at the age of 25, and that infection results in 29 years of life lost without treatment, or 15 years of life lost with treatment. The present value of health loss is calculated. The calculation accounts for the savings in health costs that will arise from fewer people contracting the disease, and so fewer needing antiretroviral therapy, which costs $500 per year.
Three scenarios are considered. In scenario 1, other treatments lead to eradication before vaccine development is complete. In this case, the vaccine has no impact. The authors place a probability of 0.1 on this scenario occurring. In scenario 2, better use of existing treatments reduces the disease burden somewhat by the time a vaccine is ready for deployment. This is estimated to be the most likely scenario, with probability of 0.5. In scenario 3, current trends in disease burden continue, leading to very little reduction in the disease burden (probability: 0.4). These scenarios are weighted by their probabilities in the calculations.
Finally, two alternate assumptions about funding are made. In one scenario, an additional $17.1bn is assumed to be sufficient to create a vaccine with the specifications above, and the average impact of this funding is calculated.15 In the second scenario, the marginal impact is considered, by assuming that an extra $100m/year brings forward availability by 0.4 or 1 years).
Under the first assumption, which examines average returns, benefit:cost ratios are between 3:1 and 48:1. If we instead make the second assumption, to estimate marginal impact, benefit cost ratios are between 4:1 and 106:1.16
The calculation ignores related infection costs, the costs of caring for AIDS orphans, productivity effects, or the burden of disease outside of sub-saharan Africa. These factors suggest that the estimate is downward biased.
However, one commentator on the paper, Forsythe, notes that the paper fails to account for disinhibition costs (increases in unsafe sex, because people think that they’re immune to HIV). This may mean that it overestimates the health benefit from a partially effective vaccine. Forsythe also notes the need to account for advertisement costs, particularly to overcome stigma related to a vaccine for a sexually transmitted disease. He also notes a need to account for liability costs. Forsythe is also somewhat more pessimistic about the efficacy of the projected vaccines. All of these factors would bias the estimate of effectiveness upwards.17 These various considerations mean that there is a lot of uncertainty associated with the estimate, but that it is not clear whether the estimate is biased upwards or downwards.
However, one of the most significant limitations of this study is that it is a cost-benefit analysis rather than a cost-effectiveness analysis, so it is not comparable to other estimates that are in a $/DALY form.
This paper assesses the average ex ante cost-effectiveness of funding research into new Tuberculosis vaccines.
The analysis focuses on the two regions which bear the bulk of the TB disease burden: Asia, and sub-Saharan Africa. The impact of the vaccine is estimated using an epidemiological model. Three product profiles for a potential vaccine are considered. The first, ‘prime’, considers an improved initial TB vaccine, with cost of $0.50-$2. The second, ‘booster’, considers a booster vaccine that extends the efficacy of the existing vaccines, with costs of $5-$10 per dose. The third scenario, ‘prime + booster’, considers a combination of the improved initial TB vaccine with the booster vaccine, with costs for the program scaling in line with the component costs. It is assumed that developing one vaccine costs between $600m and $800m. Adjustments are made for administration and distribution costs.19
The study finds social returns of $6-$10 per DALY for the ‘prime’ program in sub-Saharan Africa, or $5-$16 per DALY for the same program in Asia. Returns for the ‘booster’ program are $21-$26 per DALY in sub-Saharan Africa, or $18-$235 per DALY in Asia. For the combined ‘prime + booster’ program, returns are $21-$23 per DALY in sub-Saharan Africa, or $105-$136 per DALY in Asia.
First, this paper relies heavily on its assumption of the cost of developing new vaccines: the costs it mentions are roughly the average development costs for all pharmaceutical products, and this may not be an appropriate reference class. Moreover, there may be specific challenges relating to Tuberculosis, which make vaccine development more difficult.20 Second, the model also relies on its epidemiological model, which I do not have the expertise to assess. Therefore, I can’t tell whether this study accounts for co-infections, healthcare costs, or productivity effects. Third, the model assumes that a new TB vaccine would not be produced at all without non-private investment, and this might not be the case: it may be that industry would develop a vaccine anyway, but that donations speed the process up. If this is the case, the effectiveness estimates are likely to be upward biased. This is related to the study’s focus on the average, rather than marginal, donation. Finally, the study is somewhat dated, and the burden of disease or funding situation may have changed since it was published.
This paper attempts to estimate the marginal, ex ante cost-effectiveness of funding medical research into a wide range of low-income diseases, tracking uncertainty throughout the process.
It uses a model designed to estimate the returns to extra funding of problems of unknown difficulty, of which research is one, since we don’t know how many resources will be needed to create a vaccine. It models marginal funding as bringing forward the date of technological discovery, and therefore shifting forward the rollout of the vaccine/drug. This in turn means fewer people contract the relevant disease. The model assumes that ‘push’ funding shifts forward the entire time-profile of the disease, so that the gain of shifting forward vaccine discovery by a year equals the current annual disease burden. Costs of distribution were not accounted for. This model uses estimates of funding, which were drawn from the G-FINDER project, and estimates of disease burden, drawn from WHO and Global Burden of Disease studies. Adjustments were made for potential biases in these figures, and estimates were made of the uncertainty surrounding the figures. It was assumed that the elasticity of research with respect to marginal funds had a mean of 0.35, a conservative estimate.
The study found median returns of $72/DALY across all diseases studied, but also found that 9 of the diseases studied exhibited returns of $50/DALY or better. For instance, research into Diarrhoeal disease was estimated to have returns of $8/DALY, whilst Salmonella research had returns of $14/DALY. Estimates for the ‘Big Three’ were $91/DALY for malaria, $45/DALY for tuberculosis, and $71/DALY for HIV/AIDS. Mean returns were estimated to be significantly higher, but 95% confidence intervals tended to be roughly ten times higher and lower than the central estimates, indicating significant modelled uncertainty.
The model also accounts for the positive impact of displaced health spending, and of displaced health research spending (since, post-eradication, neither health spending to control the disease, nor research spending to develop new technologies is needed). The combined impact of these effects is plausibly very large, but highly uncertain, so is excluded from the central estimates.22
The omission of the two effects just described tend to bias the effectiveness downwards, but this may be necessary to limit the amount of uncertainty in the model. The model also assumed that funding made now does not affect funding decisions made in the future. Whilst this is questionable, it could fail in either direction, and therefore it is unclear in which way this would bias the estimate. The model also assumes that shifting research funding forward by a certain time period shifts eradication forward by the same time period. This relies on the assumption that diseases will eventually be eradicated, which seems reasonable to a first approximation. It also relies on the more questionable assumption that it will be new control methods, rather than a more widespread use of existing control methods, which will ensure eradication. Finally, this assumption relies on the idea that increasing funding shifts discovery forward logarithmically (so that each unit of funding brings discovery forward somewhat): whereas it may be that increases in funding do not significantly affect discovery times. This will vary depending on the disease, but it is likely that this factor causes a significant overestimation of effectiveness in several cases. More generally, the analysis was very abstract, and failed to account for disease-specific considerations. This should increase the uncertainty we have with regards to the estimates.
Nevertheless, a key advantage of this model is that it is a marginal, ex ante cost-effectiveness analysis, something that no other existing study that I could find fulfilled.
I turn now to some alternative approaches to estimating the returns to research, focusing on quick, speculative calculations.
GiveWell has previously carried out a rough cost-effectiveness calculation for research into cancer.24 I replicated this estimate, instead considering the set of tropical diseases that were studied in the Global Priorities Paper discussed above. The result is an average, ex ante cost-effectiveness estimate.
My approach was to retain GiveWell’s model, and to maintain their parameters, but to replace estimates of the disease burden and spending with data from the WHO, and the G-FINDER project. The model assumes that the entire DALY burden will eventually be eliminated, and that new research will be responsible for around half of the benefit.25 It was assumed that benefits would lag costs by 50 years.26 The benefits in DALYs, and the costs were then discounted, at a 3% annual rate, and accounting for population growth. The ratio of costs to benefits was then calculated, to give a cost-effectiveness ratio.27
The results were that only three of the diseases studied failed to reach the WHO cost-effectiveness barrier of $100/DALY, whilst four had cost-effectiveness ratios of less than $10/DALY. The ‘Big-Three’ had cost-effectiveness in the range $44-$52/DALY. Please see the separate spreadsheet for full results.28
The model is relatively simplistic. Like the Global Priorities Project report, discussed above, it fails to account for the particular scientific challenges that different vaccines pose, or for the possibility that current control methods, rather than new research will lead to eradication. This complex set of questions is subsumed under the assumption that research will eliminate half of the disease burden. It is unclear in which direction this is most likely to fail, for each disease, but this in itself introduces a huge amount of uncertainty to the analysis. Second, it may be that the G-FINDER measures of research spending are downward biased, since they may fail to account for all research spending (this would cause the effectiveness to be overestimated, the rest being equal).29 Third, the lag to benefits may be an overestimate, since other studies tend to assume a shorter route to benefit, and some of the studies cited above assume benefits in ten to fifteen years. If this is an overestimate, the cost-effectiveness may be underestimated. Fourth, the discount rate may not be appropriate: in particular, we might find it inappropriate to discount DALYs. Finally, we should note again that this study gives an average, rather than marginal, estimate of impact, and that the marginal impact of donations may differ significantly from this estimate.
I have requested to see the calculations behind the cost-benefit analysis of this paper, in order to attempt to convert it into a cost-effectiveness analysis. However, I have not received further details, so I cannot carry this calculation out.
The section justifying the scope of this paper provided reasons for thinking that this problem was likely to be important (by developing needed technologies which could improve health and economic outcomes), neglected (in that this set of diseases receive less funding than other disease do), and tractable (in that progress is being made, and the disparity in purchasing power means that treatment for these diseases, once developed, can be highly effective).
I then went on to examine in detail a set of reports that attempted to estimate the cost-effectiveness of research into these diseases (plus a cost-benefit analysis from the Copenhagen Consensus Paper). I focus now only on the cost-effectiveness analyses, since they are more directly comparable.
| Name of paper | Disease studied | Average/Marginal | Ex ante/ex post | Proposed funding mechanism | Cost effectiveness estimates ($/DALY) |
|---|---|---|---|---|---|
| Donor investment choices for Global Health30 | Malaria, HIV/AIDS, and Tuberculosis31 | Average | Ex ante | Push | 12-107 |
| Making Markets for Vaccines32 | HIV/AIDS | Average | Ex ante | AMC | 17-90 |
| Tuberculosis | Average | Ex ante | AMC | 30-90 | |
| Malaria | Average | Ex ante | AMC | 15-90 | |
| IAVI study33 | HIV/AIDS | Average | Ex ante | AMC | 21-96 |
| Tuberculosis Vaccines: the case for investment34 | Tuberculosis | Average | Ex ante | Push | 5-235 |
| Estimating the cost-effectiveness of research into neglected tropical diseases35 | Diarrhoeal diseases | Marginal | Ex ante | Push | 8 |
| Salmonella | Marginal | Ex ante | Push | 14 | |
| Malaria | Marginal | Ex ante | Push | 91 | |
| HIV/AIDS | Marginal | Ex ante | Push | 71 | |
| Tuberculosis | Marginal | Ex ante | Push | 45 | |
| Model adapted from GiveWell’s analysis36 | Malaria | Average | Ex ante | Push | 44 |
| Tuberculosis | Average | Ex ante | Push | 52 | |
| HIV/AIDS | Average | Ex ante | Push | 52 |
This table at least allows all of the reported results can be seen in the same place. However, in many places, the results are likely to be biased in predictable ways, as I will now go on to discuss further. Moreover, all of the estimates are highly uncertain, and the summary table fails to capture this. Finally, the summary table omits some of the results from the last two studies, since there are so many results that they would clutter the table.37
In Section 6 I will give some general reasons for being very cautious about the estimates given in all of these papers. However, as discussed in section 4, there are several biases and uncertainties that only affect some papers and not others. In this section, I summarise these problems, and estimate the likely direction of bias for each paper.38 Acknowledging these individual biases allows us to discuss how cost-effective these papers suggest tropical disease research is, in a way that does not take the reported figures at face value.
To recap, because I can’t fully assess their method, I am uncertain as to whether the ‘Donor investment choices for Global health’ over- or under-estimates the effect. They slightly underestimate the disease burden by only assessing 3 WHO regions, but this factor is not enough to overcome the broader uncertainty about this model. The estimate of $17-$107/DALY is therefore highly uncertain but not clearly biased.
The lower estimates of the Making Markets paper appear to overestimate the effectiveness of research, because they assume that without an AMC, a vaccine will never be developed, and this seems unlikely. We should then focus on the higher estimates, which suppose that vaccine development is brought forward ($80-$90/DALY). However these estimates also seem to be highly uncertain. The risks of rising development costs, combined with Farlow’s criticism of probity and methodology lead me to suspect that the estimates of $80-$90 are overestimates of the impact of AMCs, though it is unclear how the impact of AMCs is related to the impact of other types of funding.
The Tuberculosis paper is likely to overestimate the effectiveness of vaccines because it assumes again that a vaccine would never be developed without public funding, and this is improbable. Moreover, it may underestimate the costs of developing a vaccine. Its estimates of $5-$235/DALY are therefore likely to overestimate the effectiveness of vaccine research, though perhaps not significantly.
The GPP paper appears to overestimate the effect of vaccines on health, and therefore it is likely to overestimate the effectiveness of vaccine research. Its relatively abstract modelling is also likely to miss out certain important features, increasing the uncertainty of the estimate. Therefore, its estimates of $45-$91/DALY for the ‘Big Three’ is likely to overestimate cost-effectiveness.
Our new estimate, using GiveWell methods, is also highly uncertain, but appears likely to underestimate the effectiveness of research, since it uses an excessively long lag, and possibly an excessive DALY discount rate. Therefore, its estimates of $44-52/DALY for the ‘Big Three’ likely underestimates cost effectiveness.
The above section has analysed only the relative merits of the papers: section 6 discusses the general characteristics of all papers, and notes some additional potential biases pointing in both directions, which further increase the uncertainty associated with all of these estimates.
One should note that much of the above analysis is somewhat contradictory. For instance, I saw the GPP figure of $45-$91/DALY as an overestimate of cost-effectiveness, and the new estimate of $44-$52/DALY as an underestimate. Such variations are partly because the papers study different types of intervention, so they fail to all estimate the same thing. It is also partly (and primarily, in this case), an indication that my understanding of the biases of the papers is flawed. These tensions should further increase our uncertainty about what the best estimate is.
However, focusing solely on direct effects, it appears that nearly all of the models are overestimates of the effectiveness of medical research. Section 6 will argue that indirect effects may cause a bias in the other direction, but the overall impact of that section will simply be to increase uncertainty about the estimates.
However, the magnitude of the biases is unknown, and I am not aware of any methods for estimating the magnitude of biases, so I can’t quantify plausible estimates for the true cost-effectiveness of tropical disease research. Without some quantitative method, I am unwilling to give a single estimate, or even a confidence interval, since that would give a highly intuitive analysis too much concreteness.
It therefore remains highly uncertain whether the impact of medical research into tropical diseases is higher or lower than the WHO cost-effectiveness threshold of $100/DALY. It is plausible to me that tropical disease research comfortably beats this threshold, but it is also plausible that it fails to meet either by some way. The evidence I’ve gathered simply isn’t strong enough to say, because each study is highly uncertain and plausibly biased.
Thus far, I’ve focused on looking at average effects, over a range of diseases. However, diseases specific considerations seem very important to the assessment of any particular project. These considerations take three forms: first, disease-specific knowledge will determine whether new products are needed for elimination, or if current control methods, if scaled up, will suffice. Second, disease-specific knowledge will help one to understand the scientific challenge of product development, and therefore the prospects for research progress. Third, the expected burden of disease must be accounted for. It seems that these factors are highly variable: disease burdens vary over orders of magnitude, and it seems plausible that the probability of scientific progress, and need for new products could also cause order-of-magnitude type variations.
Moreover, different funding mechanisms could have very different impacts on research. If Farlow is right, and small AMCs will not affect investment in research very much, their magnitude could well be ten times less than some more established form of funding.
So even if we could find a general cost-effectiveness ratio for tropical disease research, it would not tell us very much about the impact of funding research into a particular disease in a particular way. This impact will be highly dependent on the funding mechanism used, the current funding situation, and the scientific and epidemiological context of the donation. Indeed, if funding- and disease- considerations could both cause order-of-magnitude variations, it seems likely that the effectiveness of particular interventions could vary over orders of magnitude. So if the average impact is $x/DALY, it seems likely that individual projects could be tens or hundreds of times higher or lower than this figure.
Therefore, even if the figures are significant overestimates of cost effectiveness, it seems possible that well-directed funding can have an expected value that compares very favourably with GiveWell’s top recommended charities, even if most funding fails to meet the WHO’s cost-effectiveness guidelines. It is therefore extremely important to identify the diseases and funding types that will yield high returns.
The main aim of this report has been to assess the general returns to funding, rather than finding the most promising options within this area. However, as a supplement to the report, I have compiled appendices which summarise some of the evidence relating to the assessment of which diseases, and which funding mechanisms are most promising. The evidence is incomplete, and does not allow me to draw any firm conclusions, but will hopefully provide a basis for future research.
One of the key decisions is which disease to fund. This includes assessing how likely it is that progress can be made in product development for a particular disease, and assessing what impact new products are likely to have. I have summarised some evidence of the relative need for research in various diseases in Appendix 1.
In Appendix 2, I give a brief summary of the various funding mechanisms available, and some arguments relating to which is most likely to be effective. The funding mechanisms can be broadly split into ‘push’ mechanisms, which involve providing funding to prospective research projects, and ‘pull’ mechanisms, in which prizes are offered to people who develop products. There are arguments relating to the general funding of each.
Before concluding I should summarise some concerns that relate to all of the studies that I examined.They may be the result of publication bias, causing overestimates. They fail to account for epidemiological effects, which may cause them to underestimate effectiveness. More generally, many of the studies were old or too theoretical. These considerations make bias in either direction more plausible, and increase the uncertainty we should have about the estimates.
First, we should note again the inherent difficulty of evaluating the impact of research, especially before the fact. It is often difficult to assess which papers have been the most important even ex post, because it is unclear which pieces of research developed crucial preliminary ideas. It is even more difficult to do ex ante, because it is inherently unclear what research into the unknown will discover.
Second, all of the studies carry out their research on quite a general level, and do not account for the scientific challenges related to product development for each of the diseases. Since, as argued above, the magnitude of these scientific challenges may be a significant determinant of effectiveness, this increases the uncertainty relating to these estimates.
Third, we should also note some doubts about probity. None of the studies cited above have been published in peer reviewed journals. Nor have I checked their calculations or sources. Therefore there remains the possibility of poor scientific practice, or miscalculation. Moreover, it should be noted how closely their results are grouped in the range of $10-$100/DALY. At first glance, this might seem encouraging: there is significant agreement between the studies on the impact of research funding, despite their diverse methodologies. However, the studies’ methods are so diverse that we would expect them to reach quite different results. The small range of results, which present vaccine development as extremely cost-effective, but not implausibly so, are in the ideal range for anyone wishing to encourage more funding for vaccine research, and this is exactly what some of the organisations who funded the study are engaged in. We should therefore be careful to ensure that assumptions have not been selected in order to include this specific result. The criticisms of Farlow et al lend credence to this possibility for the Making Markets paper, in particular.39 We would expect this bias towards effectiveness to lead to some overestimation of effectiveness.
A fourth limitation common to all studies is that they miss out secondary effects. These include interactions between diseases (for instance, Schistosomiasis possibly increasing the prevalence of HIV)40, as well as non-health benefits such as increased productivity, improved access to schooling, and economic growth. As noted above, it is highly contentious whether these indirect benefits exist, and what magnitude they have, but it is possible that this causes all studies to somewhat underestimate the impact of medical research.
Fifth, several of these studies are several years old, and it is unclear to what extent their findings will extrapolate to a situation with a different disease burden, research stage, and funding situation.
These additional uncertainties and biases further reduce our confidence in the figures discussed above.
Whilst there remains a lot of uncertainty with regards to this topic, I argued in section 5.4 that it is possible that some of the best funding options will be more effective than GiveWell’s current top recommendations, even if the average funding option is not very cost-effective. Moreover, if we can find such opportunities, they could well be large funding opportunities: there may be millions of dollars worth of excellent donation opportunities that provide a higher expected value than GiveWell charities.
This is an indication that it may be worthwhile to invest further resources, both in the form of further research and, eventually, funding. However, even if such donation opportunities do exist, it may be very costly to find them. If the costs of finding such opportunities are too high, then the expected average cost-effectiveness of the funding opportunities effectively falls.
So whether we should pursue further research depends on how likely we are to find these best funding opportunities, and how good those opportunities are likely to be. This is in itself an open research question, and the answer that follows is very rough.
We need estimates for three quantities: the cost in dollars of research to find the best interventions (call this R), the basic cost-effectiveness of these best interventions in DALYs/$(call this C), and the size in dollars of the funding opportunity for these interventions (call this N). Then the average cost-effectiveness of funding the best interventions, including the research needed to find those interventions, is given by:
(N + R)(N/C)
Which is the usual form of total cost in dollars divided by total benefit in DALYs
On a relatively pessimistic set of assumptions, we assume that the search space of possible diseases and funding mechanisms could be investigated relatively thoroughly with around four years of work from a researcher with relevant experience. Therefore, research costs might be around $200,000. If they identified a relatively small, say $1m, funding opportunity, that had relatively low cost-effectiveness of $49/DALY, then 1,000,000/49 = 20408 DALYs would be averted at a cost of $1,080,000, giving a cost-effectiveness of 1,200,000/20408 $/DALY = 59 $/DALY. So in this case, funding more research would be worse than the $50/DALY benchmark.
However, with more optimistic assumptions, say N =$10m, R=$80,000, and cost-effectiveness of $40/DALY, the total average cost-effectiveness is (10,000,000 + 80,000)/(10,000,000/40) = 40.32 $/DALY.
Therefore, even plausible changes in values of the variables can cause the average cost-effectiveness to cross the $50/DALY benchmark. I am highly uncertain about what figures are appropriate for all of the variables, so it remains an open question whether this topic is worth pursuing further. Before and as further research is carried out, we need better estimates of these figures.
However, on the assumption that it is, these are the recommendations that I would make:
The main type of research that is needed is applied and scientific: in order to find the most promising opportunities in this area, we need to look at disease- and funding-mechanism- specific issues, which require scientific expertise and an applied focus.
However, there is also room for more reliable assessments of the field in general: that is, a more complete and acceptable version of the models discussed above. This will be useful not only in assessing whether the best options are likely to compete with GiveWell’s current top charities, but also in ascertaining that the field remains promising enough to justify the investment in scientific advice (that is, trying to determine plausible values of N, R, and C to input into the model above, or improving on this model in other ways).
With regards to this sort of research, my experience accords with GiveWell’s analysis that there is not enough marginal and ex ante analysis of funding.41 The model used in the GPP report seems very promising at analysing the marginal ex ante effect of funding on vaccine creation, and is the only method analysed that can capture marginal rather than average impact. The GPP paper is, however, unusually poor at assessing what the impact of a vaccine would be once developed: it makes the simple assumption that it removes the entire disease burden. However, other models, though poor at assessing the impact of funding on vaccine funding, use more sophisticated epidemiological models, to provide more convincing estimates of the impact of a vaccine once developed. There therefore appears to be a possibility of developing a model which uses GPP’s analysis to estimate the impact of funding on vaccine development, but uses more sophisticated epidemiological analysis to estimate the impact of a vaccine once it is developed. This model could overcome many of the estimation problems that are present in current reports. However, I was not able to use the model in this paper because it would require a significant amount of time to gather and model data, and the primary focus of this project is to review the literature. I hope to pursue such a model in the future, however.
There are two broad options with regards to funding medical research.
The first is to try to assess current organisations working in this area, and find one that is particularly effective, and may be worth funding. Doing this is beyond the scope of this report, but I list some organisations worth assessing in Appendix 3.
The second is for a new or existing organisation to attempt to gain expertise in which research opportunities are most promising, and look into making grants itself directly. This appears to be OpenPhil’s strategy: they are recruiting and working with scientific advisors to carry out reviews of potential disease areas, with a view to making grants to promising opportunities.
Given that the targeting of funding is so important, it appears that the second strategy is likely to have higher returns, since it is possible that an organisation (such as OpenPhil) that is effectiveness-focused may make better funding decisions than a conventional charity, provided it has access to sufficient scientific expertise. OpenPhil’s current strategy therefore seems highly promising.42
Finally, funding of medical research need not be the only option here. Indeed, advocacy work may be particularly useful in this area, since there are already large amounts of money spent on this area, and we could attempt to influence them in order to make them more effective. Further, as Appendix 3 notes, there are some pull funding mechanisms, such as priority approval, that are relatively low-cost for governments to pursue (see the appendix for details). Assessing the potential efficacy of such advocacy could be considered a subsidiary research priority. One means of assessing this would be to looks at the formation of current funding bodies.
However, due to uncertainty about the effectiveness of this area, and the likely high variation in cost-effectiveness between different disease targets, it seems that neither of these funding strategies is likely to be particularly fruitful until further research has identified the most promising targets for funding.
This report has examined existing literature in an attempt to estimate the average cost-effectiveness of research into tropical diseases.
However, the substantial challenges to estimating this have made any firm conclusion about average cost-effectiveness, let alone marginal cost-effectiveness, impossible. There is some hope that this could be alleviated somewhat with further research into the cost-effectiveness of medical research. More importantly, it has indicated that the cost-effectiveness is likely to vary widely, according to disease-, technology-, and funding-types. Research into certain diseases is hence likely to be a great deal more cost-effective than research into others, and research into specific NTDs may well be highly cost-effective. However, there is a great deal of uncertainty surrounding all of this, and additional research funding in some areas may not have any effect (this may be particularly accurate for NTDs for which highly cost-effective interventions already exist).
Further discussion of research for specific diseases, of different funding types, and the practical costs involved can be found in the appendicies. However, for medical research overall, it remains unclear exactly how cost-effective it is and whether or not it compares favourably to the interventions carried out by our top-recommended charities.