Stay up to date with Giving What We Can's blog. Featuring the latest news about our community, how to donate effectively, charity evaluation, community events and more!

Blog

Giving What We Can's blog, featuring the latest news from our community.

Donate

Pledge

footerNavigation

About us

Contact us

Our team

Our values

Our members

Our events

Transparency

Our mistakes

Our impact

Donate to Giving What We Can

Careers

Leave feedback or make a complaint

About

Our best charities

Take a giving pledge

Get involved

Make a bequest

Videos, books, podcasts

FAQs

Learn more

Press kit

Media coverage

Case studies: People who pledge to give

Media

Footer Navigation

Make your donation go further this Giving Season

topNavigation

Our research team’s 2026 recommendations

How we choose which charities to recommend

Other places to donate

Create a fundraiser

Best charities

The 10% Pledge

The Trial Pledge

Pledging vs. donating

View all pledgers

Share your pledge to inspire others

Giving pledges

About our research

Browse our guides and resources

Connect with our community

Read our blog

Help others discover effective giving

Research and resources

Who we are & what we do

Our structure

Mistakes

Top Navigation

Giving What We Can

A CONVERSATION WITH HOLDEN KARNOFSKY, CO-FOUNDER OF GIVEWELL-PART III

Part III: Funding Randomised Control Trials

Recently, Giving What We Can staff sat down with Holden Karnofsky, the co-founder of GiveWell. The goal was to try and critically explore GiveWell's methodology, research approach, and recommendations. The 

Rob: While you think there's a lot of value in learning about giving opportunities, you haven't recommended any charities that are explicitly trying to do this, like the randomised controlled trials of JPAL or IPA. Probably the reason you haven't done that is it's harder to get evidence of their impact - connect the dots between the reports they write and spending by governments or aid agencies. Is this a mistake?

Owen: If you think there is some kind of market efficiency in charity giving, isn't getting more information out there about the best charitable intervention going to have be massively good?

Holden: I think that's a reasonable argument. Randomised controlled trials are a good example. We've had tons of ideas about RCTs, but we dropped most of them, because the softer the data is, the more it helps to be an expert in the area before you have faith in your ideas. We've just never come to an idea about RCTs that's worth our time trying to write.

Owen: We're not suggesting that you actually do RCTs, but why hasn't GiveWell looked into a charity that does RCTs in global health?

Holden: The reason we recommend what they do is that we think they are the best giving opportunities we can find and we have a good view of how they will go. We've tried to understand what the general operations support of JPAL and IPA do and 

, but we couldn't get any good idea of what they would do with more funding from that.

Perhaps you would say that more funding would produce more RCTs, but keep in mind the timeline - good RCTs take years from start to finish. The one's we've learned from are the ones in global health where there are a lot of them and places like Cochrane Reviews have pulled them together, and we haven't learned as much from RCTs looking more into social science, and we don't think they're worth recommending.

By directing more money toward producing RCTs, it seems we'd just be increasing the supply of RCTs, which is not the same as GiveWell forming hypotheses and seeing how the charity will proceed, given more money.

Rob: You've said that when you've found it difficult to determine what JPAL or IPA would fund with extra money, and they probably don't even know what their marginal product is because of the timescales involved. But is it a sensible requirement to know what they will do? If these organisations have been sensible in the past, might they not make good choices in future?

Holden: I think I've misunderstood the question. There are two potential goals - form hypotheses and follow impact of donations and just increasing RCT supply. I agree that donating to IPA or JPAL will increase the supply of RCTs, but I disagree that they'll help with our hypotheses. Only a minority of RCTs actually change the way we think about aid.

Rob: When you're looking at JPAL or IPA, what are you comparing them to? Isn't it so hard to find the impact of any of that, and therefore we'll never know what it's most appropriate to compare JPAL and IPA to, especially when trying to figure out how much we should adjust our estimates downward based on considerations not included in the explicit cost-effectiveness estimations?

Holden: This is another are where there are thousands of possible comparisons and we should be thinking about that rather than looking for one particular line of comparison. There's a general comparison we can make to the track record of quantitative social science and the track record of research in general influencing policy. Also, in general, we should just expect that opportunities to do good will be closer to the average than they initially look, especially when we don't know much about them.

Carolina Flores

Is an end to malaria in sight?

A new vaccine against malaria using parasites from irradiated mosquitoes has just passed the first phase of clinical tests with 100% success, surpassing large difficulties in production and improving greatly on any previous attempts. If further trials confirm this result and obstacles to distribution and application are removed, an end to malaria may be in sight.

At the start of August, Science magazine reported on the impressive results of PfSPZ in its first safety trial. This vaccine was developed by Stephen Hoffman's team in Sanaria, a company based in Rockville, Maryland, using parasites from irradiated mosquitoes. The six subjects given five doses intravenously were 100% protected from later bites by infected mosquitoes, whereas five of six unvaccinated controls developed malaria, as did three of nine people given only four doses of the vaccine [1]. Previous vaccines had never come close to this level of efficacy.

The next phase is the implementation of larger trials in regions where malaria is rampant, which will test for protection against different strains and on different age groups. One is already set to start in the Ifacara Institue in Tanzania. Hoffman hopes to have a vaccine licensed within four years.

If that goal is achieved, the benefits to developing countries will be enormous. Every year, malaria kills between one and three million people, and infects as many as half a billion people [2]. The economy slows down and a huge burden is placed on the health system.

An effective, widely distributed vaccine would highly reduce the incidence of malaria and therefore make a crucial contribution to these countries' development, as acknowledged by the World Health Organization (WHO), which set as a target the creation of a vaccine with 80% efficacy by 2025 [3]. Is the PfSPZ going to be the vaccine to achieve this?

A different approach to vaccination against malaria

Results from the 1970's which showed that immunization with over 1000 bites by irradiated sporozoites confers about 90% protection against infection, lasting for 10 months [4]. Hoffman decided to try to create a vaccine of equivalent effect.

This led him to using the whole Plasmodium falsiparum (Pf) - the parasite which, out of the four strands that cause malaria, accounts for almost all the fatal cases - in the sporozoite (SPZ) stage of its life cycle. This approach was radically different from the mainstream line of employing just a handful of proteins from the parasite. Initially, many researchers were sceptical of the possibility of mass-producing sporozoites (between ten and a hundred thousand per vaccine) within the required health and safety standards.

Against the odds, Hoffman's team managed to construct a successful production process. The researchers raised mosquitoes in industrial quantities, fed them with infected blood, and irradiated them to weaken the parasites, so that they can infect but cannot cause the disease. After that, the parasites were harvested from the mosquitoes' salivary glands, purified and cryopreserved to obtain the vaccine.

Despite the initial success, optimism must for now remain cautious. First of all, unlike most widely used vaccines, which are given orally or by injection just under the skin, the PfSPZ is intravenous. This makes it harder to apply, particularly to children. As 70% of malaria deaths are of children under five, this is a serious issue.

Under-the-skin vaccine trials have not been successful: in 2011, only two out of eighty participants were immunised [5]. Hoffman claims that the small volume of the vaccine (0.5 millilitres) and the tiny syringe required will make it widely applicable, especially taking into account that the company is looking into improving the intravenous delivery system; but other researchers believe that it won't be possible to use on children.

Secondly, the vaccine needs to be kept frozen in vapour nitrogen phase, which is an obstacle to distribution. However, it should be able to piggyback on veterinary infrastructure, and to be transported with veterinary medicine and semen for insemination of livestock.

Though it is still hard to predict whether this vaccine will be successful, it is undeniable that it represents a big step in malaria research, and that it gives new reasons for optimism about reducing the incidence of the disease.

http://dx.doi.org/10.1126/science.1241800

http://www.againstmalaria.com/Faq_malaria.aspx

http://www.who.int/vaccine_research/diseases/soa_parasitic/en/index4.html

[4] Hoffman, S. L. et al. J. Infect. Dis. 185, 1155-1164 (2002).

[5] Epstein, J. E. et al. Science 334, 475-480 (2011).

http://www.nature.com/news/zapped-malaria-parasite-raises-vaccine-hopes-1.13536

Peter Hurford

Smart Development Goals: A promising opportunity to influence aid spending via post-MDGs? 

Giving What We Can recently concluded its investigation of the Copenhagen Consensus Centre, discussed previously in our post “

Copenhagen Consensus: Making a Bet on Catching a Big Fish

 (CCC), known for organising research forums on global prioritisation, aims to influence the next round of 

 via research and public advocacy, beginning now and continuing until the post-MDG process completes in 2015. The CCC is of particular interest to us as it represents the primary example of an organisation engaged in global prioritisation research. We are by nature highly sympathetic to this research, which we have used several times. In addition, we have done 

 suggesting high levels of cost effectiveness for further funding of research of this kind.

The Millennium Development Goals (MDGs) are set by the United Nations every fifteen years and provide guidance for spending of international aid and development assistance. These goals have included cutting extreme poverty and global hunger by half. They also specify metrics by which to assess whether goals are met.

CCC’s plan is to influence the next round of MDGs by (a) involving UN ambassadors in full-day sessions exploring the results of their cost-benefit analyses, working with them to set individual priorities, (b) doing the same with the NGO community and journalists, and (c) reaching out to the public via a website and articles in major publications. CCC’s research is already fully funded, but greater funding is needed for publicity and outreach.

CCC’s case for impact is that (1) changes in the MDGs can cause large changes in the distribution of international aid, (2) the CCC can recommend goals that predictably deliver better value for money, and (3) CCC will have a good chance of influencing the MDGs. If these claims are true, then CCC can potentially move billions of dollars to higher-impact causes.

Detailed analysis suggests that the CCC is able to make a reasonable case in support of its core claims. Analysis of aid spending data, alongside testimony from NGOs and governments, shows that the MDGs are highly likely to influence a significant share of the total aid budget. It is also plausible that there is sufficient variance in the cost-effectiveness of the MDGs that rearranging them would be impactful. Finally, by focusing on cost-effectiveness and evidence-based prioritisation to a greater extent than other groups, CCC has a credible claim to be able to select better goals than other parties involved in the process.

While we are very impressed by the work of the CCC, we have a number of lingering concerns that keeps us from recommending this project alongside our top-ranked charities, such as the Against Malaria Foundation. In particular, we’re concerned that the CCC is spending too much time on controversial arguments challenging the cost-effectiveness of climate change mitigation, that the MDGs might already reach a decent outcome without their work, and that CCC experts make overly confident claims about their modelling and the future.

. Given the pressing demands of its current research work, our investigation has been constrained by the limited time available to the CCC in speaking to us, addressing our concerns, and preparing documents. Nonetheless, we believe the information contained in this report will be useful to both members and the general public.

While we commend the work of the CCC, we do not feel that we can yet recommend this project alongside our top recommended charities, such as Against Malaria Foundation or Project Healthy Children. However, we are taking a ‘wait and see’ attitude. This project is still in the process of getting off the ground. We do not yet know what CCC’s recommendations will be, and have limited information about what messages they will focus on in their sessions. We cannot say with very much confidence to what extent climate change is likely to figure as a focus for advocacy, nor to what extent the CCC’s recommendations will diverge significantly from those otherwise favoured by the MDGs process. We’ll continue to monitor CCC’s efforts to influence the MDGs, and are prepared to reconsider if our doubts turn out to be unfounded.

A CONVERSATION WITH HOLDEN KARNOFSKY, CO-FOUNDER OF GIVEWELL-PART II

Recently, Giving What We Can staff sat down with Holden Karnofsky, the co-founder of GiveWell. The goal was to try and critically explore GiveWell's methodology, research approach, and recommendations. The 

Giving What We Can's aim was to act as lawyers, not judges, to help GiveWell smooth over any potential flaws and become more accurate. In playing the 'devil's advocate', Giving What We Can might come across as more negative and skeptical than is their actual view of GiveWell.

Ultimately, these conversations are valuable because GiveWell relies on people to challenge them but, often enough, there are not enough feedback loops in the world of philanthropy. Giving What We Can's recommendations will also be improved, since we rely heavily on GiveWell's research. Likewise, it should benefit the thousands of people who donate based on GiveWell's recommendations and the hundreds of people who have taken Giving What We Can's 

You can find the first part of the interview 

Rob: As GiveWell is moving into different kinds of interventions (migration, catastrophic risk, etc.), do you worry that you might bring over too much skepticism because of the work you’ve done in the past on completely different areas?

Holden: There is a risk on both sides and we’re aware of it. And I think the bigger risk is on the other side -- not bringing in enough skepticism. For example, global health has a really good track record -- it’s quantifiable, measurable, and accountable. The track record of people trying to do things that rely on specific views on how very low probability events will affect the very far future has a really bad track record.

Donors are afraid that GiveWell is good at understanding interventions with lots of evidence and accountability, but that when GiveWell moves into evaluating politics or asteroid impacts, one can’t know nearly as much, and GiveWell will underestimate their own uncertainty because they’re still thinking about global health.

I’m excited about politics and existential risk reduction. I probably could give to more of these “blue sky” things when we have better recommendations, but I’m concerned that I’m just being crazy and overconfident.

Rob: Do you think there is a 50% or higher chance that GiveWell will find something twice as high impact as malaria within the next, say, five years?

Holden: I do, though again I’m worried about being crazy. I don’t think we can make a really quantified or irrefutable case, so we expect people to use their own judgement calls. GiveWell will move to just putting up information about different areas and explaining what Holden thinks and where he gives his money, but we’ll tell people they’ll have to decide for themselves because it will come down to very intangible judgements.

I personally think this highly measurable stuff is just not as high impact as this “blue sky” stuff, as long as the “blue sky” stuff is done right. I assume it’s easily to do badly and therefore gets a bad reputation, but it could be done well and be at least an order of magnitude better than global health interventions. But that’s just my guess, and I don’t know if Elie Hassenfeld would agree with me.

Rob: My impression is that Elie probably doesn’t. But I basically share that outlook. Therefore, I’m personally holding on to my money rather than donating it, because I expect that soon we’ll find a much more high impact place to put it. I’d be curious to know if you are holding your money as well, and whether you think others should do the same, at least in the short-term.

Holden: I’m not holding on to my money. I’m donating at a regular pace. We wrote about this in “

”. I think giving itself is a mechanism for learning.

Let’s think about this two ways. First, you’re a big donor with your own foundation and your own staff. At that point, giving regularly allows you to continue to gain access to the things you are giving to and continually re-examine your hypotheses in a way you wouldn’t be able to do otherwise. But I agree that foundations shouldn’t be giving out nearly as much now as they ought to when they finally understand enough.

A small donor won’t have this level of access. But they can give to GiveWell’s top charities because it allows GiveWell to gain that level of access. See our post that “

Giving to GiveWell’s Recommended Charities Helps GiveWell

”, because people will engage with GiveWell much more if GiveWell can demonstrate more money moved, and this will greatly improve our ability to find higher impact interventions.

Rob: I agree that’s true, but my guess is that consideration is outweighed by the chance of us finding something many times better. Though there’s value in learning, there’s probably more value in holding on to your money. Aren’t we kind of biased toward giving people advice that they should give more, because it’s better for our organisations.

Holden: There are issues about psychology when making recommendations about giving -- for example, people who save might not always follow through. I’m not concerned about this for myself, but I can see it being an issue for some people. GiveWell takes a strategy of always saying what we think and not adjusting our advice for this psychology.

I disagree with your reasoning because if the whole effective altruist community were thinking your way, GiveWell’s input would be zero, and we couldn’t find opportunities better than the Against Malaria Foundation. And most of our donors are considering this and won’t just continue to give regardless.

Rob: I definitely think that some people should be giving. But given that you’re moving potentially $10M a year, I don’t think that adding an extra $100K is worth the cost in terms of not being able to later fully fund even better interventions.

Holden: I think it is worth the cost. Of course, it’s not quite linear -- the first few million is definitely more important. But when you’re recruiting or talking to a charity, the bigger the amount of money GiveWell moves, the better.

And there’s still a concern that if you think this and restrict your money, others will follow, and GiveWell’s money moved will go down.

Paul: One response to these issues of psychology and having a credible impact down the line so charities will engage with you is to encourage people to donate to donor advised funds with a governance structure that charities can understand. If GiveWell moved a few million dollars less directly to top charities, but had that few million dollars in a donor advised fund for “blue sky” projects, it’s not clear to me that this would decrease your ability to evaluate these “blue sky” projects.

Holden: I think that’s a reasonable observation, but it gets into questions of psychology. If GiveWell could replace all the donations moved to top charities with donations to a donor advised fund, and emphasized the size of our assets rather than the size of our money moved, we would do this. But if we could only do this for half our donations, things get more complicated.

Also, many people don’t understand what donor advised funds are. It would need to belong to an organization in order to be useful and this is difficult to explain to people and it’s not very credible. Keeping it simple is better.

Paul: How has the $1M for GiveWell Labs worked in gaining access to the sort of charities you talk to?

Holden: The $1M didn’t matter much at all because we started working closely with Good Ventures. Good Ventures is accomplishing what you’re saying because they give to show they are serious, but they’re giving a small fraction of what they’ll eventually be giving. And this is credible because people know about the size of their endowment.

As long as our relationship with Good Ventures stays strong, we’ll be able to gain access. But being able to say that the “effective altruism” movement is a real movement and being able to say that the movement is recruiting lots of people and moving millions of dollars from individuals is very helpful to GiveWell.

Should We Tell People That Giving Makes Them Happier?

It seems strange to even ask. Most people would point to the fact that they’re altruistic and want to 

. Others are concerned with inequality and justice. Another group points to the concept of “paying it forward” or repaying a debt to society. Other explanations cite various religious or social reasons.

Not too many people cite the fact that giving makes them happier. Even if people agree this is true, I don’t often hear it as people’s main reason. Instead, it’s more like a beneficial side effect. In fact, it seems pretty odd to me to hear someone boldly proclaim that they give only because it makes them happier, even if it might be true.

, should we be promoting that publicly and loudly? Perhaps it might make a great opportunity to tap into groups who wouldn’t consider giving otherwise or have misconceptions that giving would make them miserable? I’m a bit worried about how it might affect people’s incentives.

In this essay, I follow the evidence provided in the Harvard Business School working paper “Feeling Good About Giving: The Benefits (and Costs) of Self-Interested Charitable Behavior” by Lalin Anik, Lara B. Aknin, Michael I. Norton, and Elizabeth W. Dunn. Overall, in light of potential incentive effects, I think caution and further investigation is warranted when promoting the happiness side of giving.

Giving What We Can has published its own review of research on happiness and giving and find a pretty strong connection. And it’s true -- lots of evidence confirms the connection and even indicates that it’s a causal relationship rather than a misleading correlation. In fact, it goes in both directions -- giving makes people happier and happier people are more likely to give[1].

Neurological studies of people found that people experienced pleasure when they saw money go to charity, even when it wasn’t their own, but experienced even more pleasure when they gave to charity directly[2], a conclusion that has been backed up with revealed preference tests in the lab[3, 4].

This connection has also been backed up in numerous experimental studies. Asking people to commit random acts of kindness can significantly increase self-reported levels of happiness compared to a control group[5]. Further research found that the amount people spent on gifts for others and donations to charity correlates with their self-reported happiness, while the amount they spent on bills, expenses, and gifts for themselves did not[6]. Additionally, people given money and randomly assigned to spend money on others were happier than those randomly assigned to spend the same amount of money on themselves[7].

People generally believe that spending on themselves will make them much happier than spending on others[6], which, given that this isn’t the case, means there is plenty of room for changing people’s minds. However, any social scientist or avid reader of Freakonomics knows that altering incentives can create unintended effects. So is there a potential harm in getting people to do more giving via advertising self-interested motive?

The classic example is that of the childcare center that had problems with parents who were late to pick up their children. They reasoned that if they charged fines, parents would stop being late, because they would have an economic incentive not to. They found instead, however, that introducing a fine actually created even more tardiness[8], presumably because what once was seen as rude and bad faith now could be made up for with a small economic cost. More surprisingly, the amount of lateness did not return to pre-fine levels even after the owners stopped the policy[9].

Other studies have found similar effects. A study of 3-5 year old nursery students who all initially seemed intrinsically interested in various activities were randomly put into three groups. One group made a pre-arranged deal to do a one of the activities in which they seemed interested in exchange for a reward, another group was surprised with a reward after doing the activity in question, and the third group was not rewarded at all. Those who were given an award upfront ended up significantly less intrinsically interested in the task than the other groups after the study was finished[10]. A similar study found that students who were interested in solving puzzles stopped solving those puzzles after a period ended where they were paid to solve puzzles[11].

In general, money and reminders of money tend to make people less pro-social[12]. This has also been found to some degree specifically in the world of charity. In a randomized field experiment, donors were encouraged to donate to disaster relief in the US and were randomly either enticed with an offer of donation matching or not. The study found that while people donated more often with the promise of donation matching, their contributions after the donation matching dropped below the control group, ending with a negative net effect overall[13].

Another study found that when gifts were sent out to donors, larger gifts resulted in a larger response rate of returned donations, but yielded a smaller average donation[14], though I suppose this could just be because more people who usually would give nothing were giving a small amount, bringing the average down. More importantly, this study found no net decrease in future donations after gifts were no longer sent out; instead, donations returned to their normal levels[14].

And certainly it’s worth noting some times when appeals to self-interest are successful. I couldn’t find any studies where this was the case. However, there is one anecdotal example: as Nick Cooney points out in “Self-Interest Can Make the World a Better Place -- For Animals, At Least”, reduction in people eating factory farmed meat is coming almost entirely from people motivated not by concern for animal cruelty, but concern for their own health. Could advocating self-interested donations be the same as advocating health-motivated vegetarianism?

It’s not very good to just let things be unclear if they don’t have to be, and I think we can resolve this issue with more scientific study. For example, one could randomly select one group to receive information about giving and happiness, another group to receive other standard arguments for giving, and a control group to receive no arguments or information about giving at all, and track their donation habits in a longitudinal study. This study would have it’s complications for sure, but could help see if information about giving and happiness backfires or not.

Or perhaps one could perform a field experiment. You could set up a booth asking people to donate to your cause and randomly include information about giving and happiness or not in your pitch and see how this affects immediate and long-term contributions. Doing this would have added advantages of being much quicker to run and not leading to people donating only because they think they’re being observed.

[1]: Anik, Lalin, Lara B. Aknin, Michael I. Norton, Elizabeth W. Dunn. 2009. 

“Feeling Good about Giving: The Benefits (and Costs) of Self-Interested Charitable Behavior”

. Harvard Business School Working Paper 10-012. [2]: Harbaugh, William T. 2007. 

“Neural Responses to Taxation and Voluntary Giving Reveal Motives for Charitable Donations.”

 Science 316: 1622-1625. [3]: Andreoni, James, William T. Harbaugh, and Lise Vesterlund. 2007. 

. New Palgrave Dictionary of Economics. [4]: Mayr, Ulrich, William T. Harbaugh , and Dharol Tankersley. 2008. 

“Neuroeconomics of Charitable Giving and Philanthropy”

. In Glimcher, Paul W., Ernest Fehr, Colin Camerer, and Russel Alan Poldrack (eds.) 2009. 

Neuroeconomics: Decision Making and the Brain

. Academic Press: London. [5]: Lyubomirsky, Sonja, Kennon M. Sheldon, and David Schkade. 2005. 

“Pursuing Happiness: The Architecture of Sustainable Change.”

 Review of General Psychology 9 (2): 111–131. [6]: Akin, Lara B., et. al. 2010. 

“Pro-social Spending And Well-Being: Cross-Cultural Evidence for a Psychological Universal.”

 National Bureau of Economic Research Working Paper #16415. [7]: Dunn, Elizabeth W., Lara B. Aknin, and Michael I. Norton. 2008. 

“Spending Money on Others Promotes Happiness.”

 Science 319: 1687-1688. [8]: Gneezy, Uri and Aldo Rustichini. 2000a. 

 Journal of Legal Studies 29: 1-18. [9]: Gneezy, Uri and Aldo Rustichini. 2000b. 

 Quarterly Journal of Economics 115: 791-810. [10]: Lepper, Mark R., David Greene, and Richard E. Nisbett. 1973. 

“Undermining Children's Intrinsic Interest with Extrinsic Reward: A Test of the ‘Overjustification’ Hypothesis.”

 Journal of Personality and Social Psychology 28(1): 129-137. [11]: Deci, Edward L. 1971. 

“Effects of Externally Mediated Rewards on Intrinsic Motivation.”

 Journal of Personality and Social Psychology 18(1): 105-115. [12]: Vohs, Kathleen D., Nicole L. Mead, and Miranda R. Goode. 2006. 

“The Psychological Consequences of Money”

. Science 17 (314): 1154-1156. [13]: Meier, Stephan. 2007. 

“Do Subsidies Increase Charitable Giving in the Long Run? Matching Donations in a Field Experiment”

. Federal Reserve Bank of Boston Working Paper #06-18. [14]: Falk, Armin. 2005. 

A Conversation with Holden Karnofsky, co-founder of GiveWell-Part I

A Conversation with Holden Karnofsky, co-founder of GiveWell

Ultimately, these conversations are valuable because GiveWell relies on people to challenge them but, often enough, there are not enough feedback loops in the world of philanthropy. Giving What We Can's recommendations will also be improved, since we rely heavily on GiveWell's research. Likewise, it should benefit the thousands of people who donate based on GiveWell's recommendations and the hundreds of people who have taken Giving What We Can's 

Rob Wiblin: It seems that GiveWell has a pessimistic view of charities, assuming the worst of them unless there is strong evidence otherwise. Is that an accurate representation of GiveWell?

Holden Karnofsky: No, and this is where a lot of confusion comes from. We should distinguish between a "pessimistic attitude" and a "regressive attitude". A pessimist attitude would say that charities are not doing good unless otherwise proven. A regressive attitude is that when you see a claim that someone is an outlier having massively more positive impact (or negative impact) than the average charity, they are probably closer to the average than they claim to be. This "regressive" attitude, not the "pessimistic" one, is the attitude GiveWell has.

We at GiveWell are looking for the best opportunities. Most charities are doing good work. But when we run across something that looks really good, the most likely explanation is a combination of exaggeration, error in estimation, or missing key pieces of information. However, as you gain more evidence and better knowledge in the field, this becomes much less of an issue.

In most areas of life you assume that things are "too good to be true" unless otherwise proven. You don't just jump at any "get rich quick" scheme -- when you see something you don't understand, you don't take immediate action. Rather, you are unsure and investigate it further. There might be reasons why this analogy isn't perfect for charity, but the general principle applies to many parts of life.

And throughout our history at GiveWell, this approach has been confirmed as something that works - it pushes our biases in the right direction relative to our initial estimate. It's not about being skeptical that charities are doing good.

Rob: Maybe a key disagreement between GiveWell and Giving What We Can is that GiveWell thinks we should shift our estimates downward in light of low evidence more than Giving What We Can thinks we should. What empirical evidence do you have to back up this claim? And how do you quantify how much we should regress?

Why We Can't Take Expected Value Estimates Literally (Even When They're Unbiased)

 and I published a spreadsheet there that gives a sense of how GiveWell thinks regression should be done. There also was a follow up called 

Maximizing Cost-Effectiveness via Critical Inquiry

The basic idea is that when you have a "back of the envelope calculation" (where the 95% confidence interval includes the estimate that the charity is no more cost-effective than average), it's hard for your estimate to get to or above the two standard deviations from average range, no matter what your explicit estimate was -- even if you initially estimated that your charity had an impact that was thousands of standards of deviations better than average.

Owen Cotton-Barratt: GiveWell seems to imply that if average is within the 95% confidence interval, the estimate will always turn out to be only two standard deviations above average at best. Is this always the case?

Holden: This is not always the case. It's only for estimates that are sloppy and involve a lot of guesswork. I wouldn't want to make this claim about things we understand much better, and that's why GiveWell places such a high priority on understanding things better.

Consider the concept of "broad market efficiency". Imagine you're looking at a field of research you don't understand very well and lots of people understand it better than you. If you find something really great, it's more likely the experts are ignoring it for a good reason.

But if you're the expert, can see things from all angles, and you know all the relevant evidence, and you've checked everything out, it's a completely different story. And some really good work has been done, so GiveWell isn't saying that if you think something's a good idea you're wrong. And this is all consistent with what the math says.

GiveWell has never once gone looking to "knock down" an estimate. Our ideal situation is that we find massively excellent cost-effective interventions that are far better than anything else you could get. But we have found in our history that we've revised our estimates downwards. For example, very early in our history, we heard that the Fred Hollows Foundation was curing blindness for $50 and we hoped it was true. But when we looked in depth into the numbers, we found that the number was not well grounded.

When VillageReach was our top charity, we estimated them at under $1000 per life saved. But with further investigation to learn more, we kept finding that estimate going up, and ultimately we had to say that we weren't even sure if VillageReach was having an impact. And that was really tough for GiveWell because we were going against our first "big bet".

Thirdly, there is deworming. We were suspicious about the figure that deworming saves a year of healthy life (DALY) for every $3.41 spent. This number never looked right based on what we knew. When we looked into the spreadsheets for the number, we found five spreadsheet errors that involved overestimates and underestimates in cost-effectiveness. When those errors were all corrected, the "$3.41" estimate was off by a factor of 100. 

And we published all this work for the public

Fourthly, a major foundation told us that Mothers2Mothers was having a huge impact, serving 20% of all eligible mothers in Sub-Saharan Africa. But when we looked into this, we found the charts didn't make any sense. Furthermore, if their claim were true, we should be able to see trends in country-level figures, and we did not. The country-level data flat out contradicted this claim, and 

Blog

A CONVERSATION WITH HOLDEN KARNOFSKY, CO-FOUNDER OF GIVEWELL-PART III

Is an end to malaria in sight?

Smart Development Goals: A promising opportunity to influence aid spending via post-MDGs?

A CONVERSATION WITH HOLDEN KARNOFSKY, CO-FOUNDER OF GIVEWELL-PART II

Should We Tell People That Giving Makes Them Happier?

A Conversation with Holden Karnofsky, co-founder of GiveWell-Part I

New Updates to Education Research

Project Healthy Children - Nurturing the next generation in Africa

Don’t Call Us a Charitable-Industrial Complex

Measuring What Matters: Two key indicators for the Successor Development Goals