Giving What We Can's blog, featuring the latest news from our community.

Stay up to date with Giving What We Can's blog. Featuring the latest news about our community, how to donate effectively, charity evaluation, community events and more!

Blog

footerNavigation

About us

Contact us

Our team

Our members

Our events

Our mistakes

Donate to Giving What We Can

Transparency

About

Our best charities

Take a giving pledge

Get involved

Make a bequest

Videos, books, podcasts

FAQs

Sitemap

Learn more

Press kit

Media coverage

Case studies: People who pledge to give

Media

Footer Navigation

topNavigation

Recommended charities

High-impact causes

Why give to funds

How much to donate

Why donate publicly

Recommendations

Comparing charities

Choosing a cause

Inclusion criteria

Trusted evaluators

Our approach

Videos, books, podcasts and essays

Myths and misconceptions

How rich am I?

Can money buy happiness?

Resources

Research

Team

The Pledge

Trial Pledge

Further Pledge

Bequests

Why pledge

Pledges

Newsletter

Podcast

Videos

Sharing our ideas

Events

Giving games

Charity elections

Pledge drive

Careers

Community

Donate now

Best charities

Manage my donations

Tax deductibility

Donate

My pledge

My donations

My income

My impact

Pledge

My giving

Top Navigation

Eliz Kilich

A promising study on the long-term effects of deworming

Currently Giving What We Can supports two charities working on deworming – the well-established 

, which primarily targets Schistosomiasis. and the promising but less established 

, which aims to enable countries to implement deworming programs relating to Soil Transmitted Helminths.

In October 2014 GiveWell suggested that a new unpublished 

, by Croke (2014, 'The long run effects of early childhood deworming on literacy and numeracy: Evidence from Uganda'), provided promising data on the long-term effects of deworming, indicating that it represented a major update in this field of research. The cluster randomised trial compared the literary and numeracy test scores of children 7-8 years after mass deworming (22%-66% of the children in the parishes) for soil-transmitted helminths of pre-school children in Eastern Uganda parishes to the scores of a control population which had not undergone mass deworming during this time period. The author found improved test scores in those treated with deworming, and concluded that this strengthens the evidence for cognitive benefits for mass deworming in areas of high worm prevalence. Whilst appreciating the strengths of the study, as highlighted by Jake Marcus in 

, I analysed some of the limitations of the study.

Cluster vs. Individual Randomised control trials

One of the main criticisms of Croke’s study lies in the study design adopted - a cluster randomised trial. A cluster randomised trial is used when one cannot randomise individual participants, so instead randomises groups. For instance, a cluster randomised trial might look at the effects of different teaching styles (A & B) on school classes. This is appropriate because school teachers are not individual tutors but can only provide for the collective when giving a lesson. Since the single intervention affects a large group and it cannot be specifically directed at individuals, the important outcome is the effect on the whole class.

The outcome being measured in the Croke study is collective improvement in cognitive ability, which may seem reasonable. However the use of cluster randomisation is less intuitive here because children are individually being dewormed. Therefore randomising individuals would be a better study design.

On the other hand, cluster randomisation could account for any spillover effect - deworming having additional benefit to children in close proximity to those dewormed. Individually randomised trials cannot take into account spillover effect, so they could underestimate the effect of deworming. Further, randomising individuals would have been substantially more expensive.

Nonetheless, by being unable to follow up participants directly as individuals and instead relying on population-based data, the study is subject to a greater propensity for confounding factors to influence the results.The propensity for confounding in cluster randomised trials is based on baseline differences and on the hidden effects of unmeasured changes to the groups during the study. For instance, the two groups may differ substantially in baseline characteristics which could affect outcome. In the example above, if children in class A are more clever than those in class B, finding that class A improved after a different teaching style is confounded by the children being more amenable to learning.

Croke tries to control for potential confounders by showing that there is no difference in socioeconomic status amongst groups. But there is a flaw in this method. The socioeconomic data (which is collected independently from the study and only includes 22 of the 48 parishes in the original study) does not apply to the original sample (intended for deworming or control) at the beginning of the study. So it only shows that socioeconomic differences were not present at the end of the study (2010) when the cognitive ability studies were performed. Whilst it is plausible that there were not socioeconomic differences at the beginning of the study, this has not been directly shown in his paper.

When considering the data Croke acknowledges the limits of a cluster randomisation and so applies analysis to show that his data is “robust” to obvious potential confounding factors. Croke attempts to show “robustness” by analysing the data where certain factors were controlled, such as restricted water access, along with other measures. This involves checking if the results still stand despite standardising a potential confounding variable between the two groups.

For instance, if one was looking at death rates between two places in the country they may find that in place A this is much greater than place B. However this could be because of the confounding factor that the majority of people living in A are much older than B. Therefore one can standardise the data to compare all those in the same age range categories to test if that result still stands despite the differences in average age being controlled in post data analysis.

In Croke’s study, one way in which robustness of data is illustrated is by showing that the youngest and oldest children in the test classes would have smaller differences in test scores than the intermediate age groups, as they would not have been in the population eligible for deworming 7 years earlier. Whilst intuitive that those at the extremes of age would have had reduced number of deworming protocols, and so may have less differences in cognitive ability, this has actually not been evidenced. The graph provided shows no confidence intervals and did not indicate any statistics used to compare the subgroups, so cannot be assessed appropriately.Even if those statistics were used, the study was not powered to detect differences in this subgroup population. In addition, standardising the data cannot account for confounders that are not tested for and may be less obvious.

Further to these limitations, the study assumes that the populations surveyed during deworming and 7-8 years later are comparable,the latter containing the population that was originally dewormed. However, 

 revealed that a total of 3.1 million out of 23 million persons (13 percent) were enumerated outside their district of birth and hence classified as internal migrants.3 This data suggests that the population sampled in the parishes randomised for deworming or control may not correspond to the populations from which the cognitive function data were collected.

Another issue is that the endpoints for the study – differences in standard error of Maths and English tests - are limited measures of cognitive ability. Further, there is no reasonable suggestion as to why deworming would greater effect Maths tests than English tests, as the results indicate. It is also unclear what these differences actually correspond to in absolute terms, on a practical level for the populations analysed. In addition, other studies have shown increased school attendance, which this study fails to show. The reason for this inconsistency is unclear.

Finally, crucial to the analysis is an absence of evidence for combination deworming - there is no data on whether Schistosomiasis deworming occurred in the interim between Soil Transmitted Helminth deworming and cognitive testing of the populations, or on what other public health interventions occurred.

In conclusion, the study has significant limitations. But this should be seen in the context of the wider difficulties of researching the effects of mass deworming. Cognitive benefits are difficult to measure, especially over significant periods of time. This is made even more difficult when looking at substantial benefits to groups (such as spillover effects), as studying group data introduces various confounding factors. Functional outcomes and the ability of improved cognitive abilities to dramatically affect the prospects of a child in this context are difficult to demonstrate, as abundant difficult to measure factors contribute. These difficulties explain why controversy in the field of deworming is rife. This study has significant limitations with data subject to confounders and therefore it may not be as “promising” as we hoped. Further blog posts regarding how we might re-evaluate the evidence will be posted soon.

Some questions on the cognitive benefits of deworming

Andreas Mogensen

If we find that a deworming programme in one region improves school performance, how much have we learned? Should we be confident that a programme carried out by a different organization in another country will have similar effects? Will the programme be equally effective if it’s handed over to the government and scaled up?

 is a Post-Doctoral Research Fellow at NYU’s Development Research Institute whose work addresses questions like these. She founded 

 to synthesize data from different impact evaluations and improve the assessment of development interventions. Her paper 

“How much can we generalize from impact evaluations?”

described as a must-read for development economists

. I recently got in touch with Eva to learn more about her research.

Could you tell us about your work as a development economist? How does your academic research relate to your work with AidGrade?

Right now I'm focusing on leveraging AidGrade's data for research because there is a lot more that can be said with it. But I am interested in other topics in development as well and have several other studies in progress. I'm lucky because my academic work only allows me to set up the structure for and manage AidGrade on the side, and the staff are so great that they can take it from there.

Your paper assesses to what extent we can generalize from evaluations of programs implemented by one organization in a particular location to the efficacy of similar programs implemented by other organizations elsewhere. Why is this issue important? Does it matter for individual donors?

How much one can extrapolate across settings is critical to understanding what we can take away from any one study. Without this kind of work, we could make fantastically wrong predictions. For example, a similar issue came up in medicine, where many studies were traditionally done on men but there has been increasing acknowledgement that effects of some drugs might be different in women or children. The issue is likely to be even more important in development economics, because there are so many more things that could interact with the intervention and determine its effects. So one reason this work is important is because it teases apart some of the factors associated with the effects of a program and helps to improve our ability to predict.

Apart from the prediction issue, knowing the distribution of effects is also important. If we were equally unsure of the effects of all development programs, perhaps we could get away with only looking at the average effect for a particular intervention, but if we know some have spanned a broader range than others that could make a difference as to which we want to support. A former professor and inspiration, Adrian Wood, liked to say that first we should just do no harm; one can imagine perhaps we should pay more attention to the lower end of the range of possible effect sizes because if you do end up doing an intervention with poor results you might sow distrust, and research in psychology shows people dislike losses more than they like gains. Alternatively, if there are poverty traps, perhaps we should weight the large, positive effect sizes more highly. Either way, it is important to consider the full range of possible outcomes.

Finally, and less relevant to individual donors but more relevant for researchers, this kind of work can help tell us where we should allocate more attention -- where are the big unknowns and unexplained variation.

How do you assess the generalizability of impact evaluations?

AidGrade gathered data from about 600 impact evaluations of 20 different kinds of development programs. These were academic studies, and you can think of our data as essentially comprising the results tables along with characteristics of each paper (e.g. whether it was a randomized controlled trial, whether it was randomized by cluster, characteristics of the sample being studied, etc.). We also coded what kind of outcome the results captured (e.g. attendance rates in percentage points, height in cm, etc.).

For the main results, I run leave-one-out meta-analyses of all the papers within each intervention-outcome combination. For example, there might be six papers estimating the effect of school meals (intervention) on attendance rates (outcome). I take one of those estimates out, generate the meta-analysis result from the other five, and see how well that meta-analysis result predicts the one left out, cycling through all possible permutations. It's a bit more complicated than that, but that's the quick summary.

You find that a positive, significant result in one study is a poor predictor of finding a similar result if the same intervention is studied again: there’s only a 32% chance that the next results with be positive and significant. How worried should we be in light of this finding?

It does suggest some caution. One should also consider the sample size of the studies. Studies are very frequently underpowered, meaning that there is just too small a sample to be likely to observe a significant effect even if the program really had a positive effect. In general, especially with regards to how we commonly talk about programs or see them discussed in the press, I think we don't know quite as much as we think we know. Academics are better at being careful about this.

You find that government-implemented programs generally fare worse than programs implemented by academics and NGOs. Why do you think we find this pattern of results? What lessons can we take away?

There are a few possible explanations. The first thing that comes to mind for most people is that government-implemented and NGO/academic-implemented programs are qualitatively different – the government-implemented programs, in particular, tend to be much larger. There could be "equilibrium effects" at a large scale that reduce the observed impact. For example, perhaps microfinance programs help some business owners but at the expense of others -- if one looked at a small-scale program, it would appear to help, but it might not when scaled up. Alternatively, it could simply be harder to implement a program well at scale. Another possibility is that NGO/academic-implemented programs are simply better-managed or devote more effort or other resources to each beneficiary.

In my paper, I show that while it is true that larger programs tend to fare worse, government-implemented programs still perform worse even controlling for sample size, so size is not the full story and it appears to be a combination of factors. More work is needed to distinguish between the remaining possibilities.

Based on your findings, are there any changes you think we should make in how we go about trying to identify the best charities to support with our donations? For example, should we significantly increase our confidence in an organization like GiveDirectly, where we have an evaluation that looks directly at their program?

I think that is a correct inference to draw. There is definitely still reason to sometimes support unstudied charities, but the full range of possible results and our uncertainty about them should enter into that calculation. We should also be cautious as to whether the evaluation of a charity was cherry-picked in any way.

What steps can be taken to improve the generalizability of impact evaluations? Are development economists following these steps or is generalizability a neglected concern?

Great question. First, there are things that people could do to help measure the local generalizability of their results -- for example, include subgroup analyses in their pre-analysis plans. This means that they commit to examining differences in results for different subsets of their own sample. While this would only give us within-sample variation, I found that variance in results within a paper is associated with variance in results across the papers that look at the same intervention-outcome, and it might be relevant locally (such as if the program were to be expanded), so it would seem a step in the right direction. There should also be better coordination across people working on different papers so that they use more of the same outcome measures. Paul Gertler and co-authors, among others, have been setting up impact evaluations that ask the same questions in many different settings. This is very valuable work.

More generally, better integration of theory in development work would help. People like to say that impact evaluations are testing some causal chain, but if you delve into the papers, how they think the intervention actually works is often left unstated. It's a knotty problem, but the holy grail would be to uncover some stable parameters explaining at least some of the variation in an intervention's effects. Failing that, we would need many more iterations of experiments.

“Without this kind of work, we could make fantastically wrong predictions”: An Interview with Eva Vivalt

Michelle Hutchinson

Giving What We Can is maintaining its list of recommended charities this year: we continue to recommend Against Malaria Foundation, Deworm the World, Project Healthy Children, and Schistosomiasis Control Initiative. We believe these organizations remain the 

 to which to give and we hope you’ll support them with your donations this holiday giving season!

Last year, GiveWell raised concerns about AMF's room for additional funding. We're pleased by the progress AMF have made in distributing funds to support net distributions. Beginning the year with approximately $12.5M, AMF has committed $10.3M to fund distributions in Malawi and the DRC, whilst raising around $2.65 in donations. They are currently engaged in a number of negotiations, though current funding levels would allow AMF to take on a maximum of three out of the distributions currently in discussion. AMF are also planning to partner with the Liverpool School of Tropic medicine to assist in studies designed to assess the extent of insecticide resistance in areas such as the DRC. We continue to recommend AMF as a top charity and we’re happy to see that GiveWell has placed AMF back on its list of top charities in the most recent update of their recommendations.

GiveWell’s recommendations this year also includes a second tier of stand-out charities: Development Media International, Living Goods, GAIN, and ICCIDD. We’re looking forward to learning more about these promising organizations in future. DMI has previously been on our radar: they are an exciting organization currently trialling a 5-year mass-media campaign designed to reduce childhood mortality and morbidity in Burkino Faso, with final results expected in the second half of 2015. Living Goods provides health counselling and sells health-products door-to-door: they are also engaged in an RCT of their program, with published results expected soon. Like Project Healthy Children, GAIN and ICCIDD work on micronutrient fortification: these organizations are focused on reducing iodine deficiency via salt iodization.

In terms of membership growth, we’ve had a fantastic year. We had 394 members at the beginning of 2014. There are now 690 Giving What We Can members in all! That means 296 people have signed the Pledge this year, compared to just 119 in 2013. We’re very happy that so many more people have been inspired to take the Pledge and we’re thankful to all our members, new and old. We can’t wait for the new year!

Our recommended charities for the year

Rob Wiblin

 is our sister organisation - we are cross posting this 

As Head of Research, you will become part of our founding team to lead our research efforts into how to select the most high-impact careers. Our research is what drives the advice we provide via our website and career coaching.

Specifically, you become responsible for:

Deciding what questions to investigate and personally coming up with answers to these questions, or overseeing the delegation of this research to others.

Helping to translate these findings into practical products that help people make better decisions - books, web applications, workshops and others.

Fitting the research into the overall strategy for the organisation.

We are a small team, so you'll also have the opportunity to be involved in recruitment, fundraising, publishing our findings in the media and whatever else most interests you.

This role combines great potential for impact with great potential for growth and learning. We believe 80,000 Hours has the potential to transform careers advice, and you would be playing a central role in our strategy. As one of the key team members of a rapidly growing small organisation, you'll have a huge amount of responsibility, autonomy and room to grow.

If you work for 80,000 Hours, you’ll be part of a focused team of intelligent, enthusiastic, driven colleagues who want to change the world, located in our offices within the beautiful University of Oxford, just an hour from London .

What does the research involve in more detail?

Check out our existing pages for examples. Some big questions we've focused on so far are:

what factors should you look for when comparing two careers? e.g. how to prioritise building skills against your immediate impact?

what steps should you take to make your next choice?

which career paths are most promising in general?

We set about answering these with a mixture of expert interviews, literature searches and quantitative estimates. All our research findings are applied in our one-on-one coaching to test their usefulness and get further feedback.

We're looking for someone who is highly enthusiastic about 80,000 Hours and effective altruism, who could see themselves becoming obsessed with the question 'in which career can you make the most difference?'.

You'll need a combination of research skills honed on messy real-world problems, and exceptionally clear written communication.

Some examples of experience that would be especially useful include: journalism, graduate study (especially in social science), policy research, and philosophy study - though we're interested in anyone who can demonstrate the skills.

The ideal candidate could also grow into a major leadership role within the organisation, so evidence you can run your own thing and lead people is a major plus.

Candidates will be selected on the following traits, and should aim to demonstrate them in their application:

Applications close Midnight GMT, Friday 12th December.

See other job openings at the Centre for Effective Altruism.

80,000 Hours is seeking a Head of Research

Francesca Day

I joined Giving What We Can because I am a Christian.

 The Christian call to charity can sound almost clichéd at times, but the fundamental message is radical. Jesus tells us “love your neighbour as yourself” (Mark 12:31). This sounds obvious and easy – but Jesus makes it clear in the Parable of the Good Samaritan that our neighbour is not just people who live near us or people who we like, but every human being in the world. We are called to value each and every human being in the world as much as we value ourselves.

Applying this principle to charitable giving, we see that putting the odd pound in a collection bucket just isn’t going to cut it. If we really love those living with poverty, illness or oppression as we love ourselves, we will do everything we can to help them. We will make real sacrifices to help them.

Charitable giving is a focus of Jesus’ teachings. He tells us “Sell your possessions and give to the needy. Provide yourselves with money bags that do not grow old, with a treasure in the heavens that does not fail, where no thief approaches and no moth destroys”. It is clear that Jesus had some fairly serious charitable giving in mind.

By sacrificing worldly goods to help those whose need is so much greater than our own, we build treasure in heaven, and we make the Earth a better place. Jesus is talking about big sacrifices here – He’s saying we should go as far as to sell our things to raise money to give to the poor! When we consider that we’re called to value them as much as we value ourselves, this idea doesn’t seem so silly.

Do I live up to these ideals? Absolutely not. I strive to model my life on Jesus’ teachings, and taking the Giving What We Can pledge is a step towards that goal. I have pledged to give 10% of my income to whichever charity I think most effectively helps those living in extreme poverty. 10% feels like a lot, but it shouldn’t. It feels like a lot because I haven’t succeeded in truly loving my neighbour as myself. Taking a public pledge to give helps ensure that my actions will live up to my ideals.

Why the focus on effectiveness? Now I’ve pledged to give this money, I have to choose where to give it. How should I choose? I should choose in a way that 

values every single human being as my neighbour

. Not just people who live near me, not just people who speak my language, and not just people suffering in ways that I might one day suffer. I should value the child in Ethopia dying of a disease I haven’t heard of as much as the lady down the road battling cancer.

With this in mind, I’ve decided I want to 

give to a charity that can help as many people as possible with my donation

. How can I work out which charity that is? It’s hard – there are several organisations researching this questions, but no definitive answers. Jesus does not give us lists of rules to follow – He tells us to love our God and love our neighbour and we have to figure the rest out ourselves. We have to figure out how and where we should give to best love our neighbour.

Jesus tells us “be as shrewd as serpents and as innocent as doves” (Mark 10:16). We are called to give wisely and to give generously. We are called to love our neighbours by giving freely of all the gifts God has given us – our worldly possessions, our intelligence and our humility. For me, being a member of Giving What We Can is part of this commitment.

Christianity and Giving

Charlie Satow

A bed net is distributed in the forest and no one is there to track its delivery and use. Does it contribute to reducing the incidence of malaria?

This question gets to the heart of a key issue in development policy and one that is extremely important to Giving What We Can: data. And with the much-heralded ‘data revolution’ demanded by the UN High-Level Panel on the post-2015 development agenda, it seems to be rising up the list of priorities for governments and multilateral institutions. In this post I want to take a look at this ‘data revolution’ and why data are important for tackling global poverty.

But first, the ‘so what?’ question. Why is data important? The first reason comes from an efficiency argument central to Giving What We Can’s mission: you need to gather data 

to judge whether an intervention is having the effects it was supposed to have

, and thus whether it is a success. What we really mean when we talk of investing in data collection is investing in effective school systems, in well-run hospitals or in successful nutrition programmes. That much is familiar. But this can also 

contribute to the democratic political process

, by equipping people with the information they require to hold their leaders accountable.

Claire Melamed, of the Overseas Development Institute, makes a further, ‘equity’ argument for gathering data[1]: 

we tend to have least information about the people who are most marginalised

 (e.g. slum dwellers[2], women, the disabled), which leads them to receive less attention in terms of policy responses**.If we had more information about these groups we would provide better services for them**. These are three compelling reasons for caring about data or evidence.

What are the current problems associated with data collection in developing countries?

The problem is that the current state of data collection in many developing countries is extremely poor - so much so that Morten Jerven has written an 

 about the poverty of GDP statistics in Africa. And here we are not talking about minor mistakes: for example, when Nigeria’s GDP was re-based, the new estimate showed a 90% increase[3]. And to give another example, the High-Level Panel’s briefing document on the data revolution states that every year around 50 million births go unregistered.[4]

Given this situation, some sort of ‘revolution’ seems appropriate. But the term ‘data revolution,’ officially comprising ‘two main objectives’ and ‘four components’[4], is a slippery one. As Claire Melamed discusses on a 

 on the subject, it holds different meanings for different people.

bringing ‘big data’ into the mainstream of measurement and policy

. This could mean using mobile phone data to track refugee movements [5], or satellite imagery of trees to measure rates of deforestation [6].

strengthening capacity in traditional government statistics agencies

: training more statisticians, getting computer systems to work, etc. Perhaps, then, the ‘data revolution’ is simply an aspirational term, 

pushing policymakers to prioritise “more data, better data, disaggregated data, frequent data,”

 as Amanda Glassman of the Center for Global Development expresses it in the same podcast. It has certainly succeeded in drawing political attention.

For Glassman, one of the key issues is addressing the fundamental problem of incentives underlying the poor data. Administrative, government-collected data remain low-quality in many countries partly because of donor actions: they have demanded accurate household surveys to be able to evaluate their programmes, and to track indicators of their interest (e.g. the Millennium Development Goals). This has produced a system of little coherence and with certain types of information prioritised over others, for example at the national level at the expense of more data disaggregated into population groups.

Faced with this situation, one concrete proposal for 

donors is to engage in funding compacts with recipient-country governments to achieve certain standards on data

, following the payment by results strategy discussed in an 

, and with civil society oversight [7]. This could perhaps be nested within what the High-Level Panel calls a 

‘Global Partnership for Development Data’

, which would also help set common global standards for data reporting, reducing fragmentation and double-collecting.

Visible, user-friendly and practical data

What else can be done here and now to improve the data we collect? Melamed identifies some such steps, for example 

, particularly by publicly-funded bodies such as the UN, and also 

Blog

Some questions on the cognitive benefits of deworming

“Without this kind of work, we could make fantastically wrong predictions”: An Interview with Eva Vivalt

Our recommended charities for the year

80,000 Hours is seeking a Head of Research

Christianity and Giving

The data revolution: what's behind the barricades?

A Brief Update on Ebola - Are There Any Effective Interventions in Sight?

The Cost of Violence II: What can we do?

Problems are in the eye of the beholder

The Cost of Violence I: What should our priorities be?