Stay up to date with Giving What We Can's blog. Featuring the latest news about our community, how to donate effectively, charity evaluation, community events and more!

Blog

Giving What We Can's blog, featuring the latest news from our community.

Donate

Pledge

footerNavigation

About us

Contact us

Our team

Our values

Our members

Our events

Transparency

Our mistakes

Our impact

Donate to Giving What We Can

Careers

Leave feedback or make a complaint

About

Our best charities

Take a giving pledge

Get involved

Make a bequest

Videos, books, podcasts

FAQs

Learn more

Press kit

Media coverage

Case studies: People who pledge to give

Media

Footer Navigation

Make your donation go further this Giving Season

topNavigation

Our research team’s 2026 recommendations

How we choose which charities to recommend

Other places to donate

Create a fundraiser

Best charities

The 10% Pledge

The Trial Pledge

Pledging vs. donating

View all pledgers

Share your pledge to inspire others

Giving pledges

About our research

Browse our guides and resources

Connect with our community

Read our blog

Help others discover effective giving

Research and resources

Who we are & what we do

Our structure

Mistakes

Top Navigation

Stephanie Crampin

Video interview with Evidence-Action

On Tuesday 22nd October 2012, Giving What We can researchers Rob Wiblin (CEO of the Centre for Effective Altruism) and Chloe Kane (Giving What We Can researcher) took part in a Skype conversation with Alix Zwane (Executive Director of Evidence Action: an organisation pioneering 

, as well as Deworm the World, another of our recommended charities), and Karen Levy (Evidence Action’s Regional Director). You can see the highlights of the conversation below.

Dispensers for Safe Water is a charity that Chloe and the research team investigated as part of a larger project, the report of which can be read 

. They found that the new programme looked very promising, and of a comparable level of effectiveness to Against Malaria Foundation, and our other top charities. We took part in this Skype call to find out more about what Dispensers for Safe Water do and to answer questions such as how much funding they need, what they measure and why when recording their success, and more general information about why the dispensers programme is favourable to other WASH-related interventions such as promoting handwashing.

We were impressed by what we heard, and learned a lot, including the following key facts: Dispensers for Safe Water has a goal of "reaching twenty five million people by the end of 2018. The intention of the programme is to become sustainable from carbon revenue by this time.” This is because they are "taking advantage of the increased cost-effectiveness that comes from operating at scale. So every additional resource has more and more leverage, towards that goal." They track that the dispensers working, and full of chlorine. While lacking specific “health evidence on dispensers, they have health evidence on chlorine, and are currently measuring adoption. The two of those things together give meaningful information” which is used to calculate the programme’s effectiveness, and which has so far provided very promising results.

Evidence Action is a new organisation, and expects to have an 

 and donations system by early November 2013. Giving What We Can will report further on their work, and expects GiveWell to do so as well, so look out for more details!

You can download a full transcript of the interview 

Christopher Hogg

Is private soap squeaky clean? A look at the potential effectiveness of corporate handwashing promotion

Giving What We Can’s mission statement, staring at me from the wall of our office as I write this, is to “move money to the world’s most effective charities”. Working out what these are though can be tricky, as us at the research team have found out over the past week in delving into the literature of schemes involving 

Handwashing promotion programs (teaching people the importance of using soap properly) compare very favourably to our recommended interventions (in terms of $/

There is not much data relating to the success of WASH schemes, and the methodology is weak, hence at present we will not recommend charities to the same extent as our 

, soap manufacturers are in a position, unique to this sector, to fund handwashing campaigns in a manner that benefits both themselves (in profits) and society (in increasing health).

Washing your hands can make you both happy and healthy.

In most instances diarrhoea is caused by consuming bacterial contaminants, often through fingers or food. Washing hands 

 at important periods of the day (such as after defecation and before eating) removes these contaminants from the surface of the skin, cutting the risk of diarrhoeal disease by 40-50% (Curtis and Cairncross, 2003).

However, rates of handwashing with soap (HWWS) are low. This is not caused by a lack of (clean) water, or indeed a lack of soap. Rather, the soap available is used to wash clothes and not hands (Aunger et al, 2010). Therefore what is required prevent the disease are education schemes that encourage behavioural and attitude changes towards soap.

Tip-toeing into the Pool of Private Public Partnerships...

Traditionally donor funded NGO programmes, such as Charity Water and Water Aid, provide these schemes, often in combination with infrastructural improvements. The (too) few studies that exist on standalone schemes have been shown in 

I wish to stress that private companies which have both a financial and PR incentive, can implement schemes which have the same, if not better results. Unilever is one of these companies, whose Lifebuoy soap brand has the potential to be well publicised and used across the world.

It worked in the past. Could similar adverts eradicate diarrhea in the developing world?

The Lifebuoy soap brand began in Victorian England and is known in North America for its adverts which 

. Its focus is now on the developing world where it is one of Unilever’s top selling products, bought from markets in Ghana to 

, the brand states its main aim: “to change the hygiene behaviour of one billion consumers across Asia, Africa and Latin America, by promoting the benefits of handwashing with soap at key occasions”. At present, no NGO lists this as a single aim. Though we are cautious, given that too often big businesses say one thing, but do another (for example 

Coca Cola’ climate change campaign in contradiction to its relation

), there are two reasons to be optimistic about their strategies in relation to the wider WASH sector.

First, their focus on cost-effectiveness. Giving What We Can’s take on the charity sector is to treat donations in a similar way to business investment, a 

 but which we wholeheartedly defend. Just like its advertising campaigns, the promotion schemes are designed to get the largest impact for the money the company invests in. It makes sense in a competitive market. When Unilever saw that by working with governments, local NGOs and schools to “train the trainers”, it saved the company funds but delivered equal results in Indonesia, it sought to extend the scheme to other countries.

Second, Lifebuoy appears to collect good evidence on the impact of its schemes. This kind of self evaluation is what we want to see 

. The Lifebuoy brand has conducted its own Randomised Control Trial in Mumbai, keeps sales data on its soaps through consumer surveys, and are even developing three dimensional sensors in some of their soaps to monitor their use, a method which, due to its unobtrusive nature, is perhaps more accurate than the current norm of participant observation (Vindigni et al, 2011).

It will be good to see more transparency in Unilever’s data. Their website does not show, or may not have, these evidence reports mentioned above. If Giving What We Can are going to support schemes such as this, being able to see the data and analysing it for ourselves is crucial. Perhaps this is where the private sector collides with the public sector, as companies are notoriously secretive about releasing knowledge which may provide a competitor a route into the market.

We would also like to see the company investing more money on these schemes. Unilever funds are diverted away from the Lifebuoy schemes to other brands, which add less health benefits to the world. There is a growing market for soap in Africa and Asia, and potential for lots of money to be made, although the risks at present seem to be too great for the soap companies to invest.

Money for (new) soap? Large companies must be persuaded that there is large enough demand for soap in developing countries such as India to make investment worthwhile.

An advocacy organisation could therefore be needed who can show and persuade private companies to divert funds to the developing world. Currently I do not know of any group which does this, but please 

 or add a comment below. There is an urgent need for more hygiene promotion schemes, and with the funds, knowledge and incentive of private companies, these can be successfully run without donations. Hopefully I have shown you that this change can be realistically achieved surprisingly soon. Part of the solution of moving money to the world’s most effective charities may well be putting more power to the world’s most effective companies.

Aunger et al, 2010: “Three kinds of psychological determinants for hand-washing behaviour in Kenya” Social Science & Medicine 70 (pp 383-391).

Curtis and Cairncross, 2003: Effect of washing hands with soap on diarrhea risk in the community: a systematic review. Lancet Infect Dis 3(5): 275–81.

DCP2 Report. Cairncross, Valdmanis (2006) “Water supply, sanitation, and hygiene promotion” In: Disease Control Priorities in Developing Countries (2nd Edition). New York: Oxford University Press. 771-792

DfID, 2013: “Water, Sanitation and Hygiene Evidence paper” Department for International Development May 2013.

Vindigni et al, 2011: “Systematic review: handwashing behaviour in low- to middle-income countries: outcome measures and behaviour maintenance” Tropical Medicine and International Health 16(4) pp 466-477

Michelle Hutchinson

Six Month Retrospective Progress Review

Giving What We Can aims to review its progress and plans every six months, and to publish the conclusions on the blog. In addition to the usual review of the previous 6 months of and plan for the following 6 months, in this case one of the trustees of the Centre for Effective Altruism (Nick Beckstead) produced a review of Giving What We Can as a project. This blog post and the following one contain the headline conclusions of these and links to them.

The first six months of the year have been an important period of learning for Giving What We Can, and have indicated that we should shift our focus more towards individual discussions of charity effectiveness and donation, away from mass methods of communication. GWWC has continued to grow steadily, and expected money moved to effective charities seems to be well worth the money invested.

Giving What We Can’s mission is moving money to the world’s most effective charities. Each of our members commits to donating at least 10% of their income to the most effective poverty relief charities. Therefore, the key metric we use to measure our progress is the number of members we have. In the first half of this year we gained slightly fewer new members than we did in the preceding 6 months, but about the same number as we had in the 6 months before that; this may indicate that the December giving season has a noticeable effect on our membership numbers. This was steady growth for the period, but meant that we ended it on 329 members, which was slightly below our goal.

We set an ambitious target for this period because it included the second half of a press push we had organised in December, which we were hoping would increase member growth. Having updated the website with an improved look and a blog with meant readers could be kept up to date with new research developments, our aim was to publicise the organisation and website as widely as possible to get members. The media campaign was a big success in terms of coverage, with articles on Giving What We Can in the Sunday Times and the New Statesman and on Sky, BBC Breakfast and NPR, among others. This coverage did not lead to as many new members as we had hoped.

Our best guess for why membership growth following the media push wasn’t higher is that taking a pledge to donate 10% of your income is a big step for someone to take simply from reading a media article. This would indicate that we ought to take a more personal approach, and that it would be good to have a more gradual way for people to get involved. The most important lesson we drew from this was that we needed to learn more about the process which leads to people joining Giving What We Can.

In order to learn more about that process we interviewed members, and did a survey with interview follow-up of people who weren’t previously involved with Giving What We Can. Two findings were particularly interesting. One was that people who heard about Giving What We Can through the media had typically not needed much persuading to join, indicating that media pieces (at least of the fairly brief form we got) didn’t tend to have much persuasive force. The other was that people who weren’t interested in joining Giving What We Can became so after having an in-depth conversation about it. These indicate that in order to make more impact on people’s behaviour, we should focus more on having in-depth and ongoing conversations with people. For more details, our full six month review is 

Over this period, we worked both on first order research into finding effective charities, and on more methodological questions, such as what the best ways are to find great giving opportunities, whether it’s better for people to donate now or invest to give later and what the appropriate age-weighting for DALYs is.

With regard to methodological questions, we produced research papers on the topics we set out to, but they generated less interest than we had expected. Investigating individual charities was reasonably successful. Our individual charity research resulted in finding a charity which we consider to be an opportunity for very high leverage, Project Healthy Children. We also wrote reports on a few other charities which seemed promising, though we did not end up recommending them. However, this research required a large time input, and in order to find a charity which is in expectation more effective than AMF (for example) and for which our estimate is equally reliable and exact would take far more resources, since charities in areas which are effective and have low uncertainty are more likely already to have been found. Overall, we think that at the moment it would be more effective for Giving What We Can to focus on driving donations to charities we know to be cost-effective than on trying to find better charities. Therefore, over the next six months we will be concentrating on outreach. In order to achieve that, we will produce blog content which aims to interest people in the idea of cost-effective giving and to drive traffic to the website.

performance review of Giving What We Can for 2011 and 2012

 estimated the return from gaining one member as being $17,000 to the most effective charities, counter-factually adjusted and time discounted. There was a very large range on this figure. Over 2012, the year in which Giving What We Can took on staff, the cost per member was calculated to be $260. The review contains the conclusions from the 2011 giving survey (the results from the 2012 survey weren’t all in at the time the review was written). Of the over 60% of members who had filled in the survey, 90% had kept their pledge. Approximately 70% of members donated to recommended charities. One question we were particularly interested in was whether members who had joined as students continued to give when they entered employment, since many of our original members were students who now have jobs. Of people in that category, just over 70% responded to the survey, and of those nearly all had kept their pledge. This is a very good sign. Although this obviously does not mean they will definitely continue giving for their whole career, it seems less likely that a person will cease giving once they have begun than that they would fail to start.

The review contained various useful lessons. One was the importance of increased data centralisation – of having one system which captures all our relevant data. We are putting this into practice by making increased use of our CRM. With the arrival of a full-time Director of Community, Ben Clifford, we are able to have one person collecting and analysing all our member data. Another recommendation made by the review was to have a real-time donation mechanism, rather than an annual survey. This would make it easier for people to let us know about their donations, and thus for us to track our impact. We aim to do this in our next iteration of the website, in which we plan to introduce ‘personal dashboards’, which will allow people to keep a record of their giving, and tell them how much they have achieved with that giving.

The first 6 months of 2013 were an important learning time for Giving What We Can. The most important lesson was that we should try out a way of encouraging people to gradually increase their level of giving, and provide as much support and personal discussion as possible – it is not enough to simply present the information to as many people as possible. The next blog post will describe how we plan to put it into practice. The conclusion of the performance review was that the expected amount raised for extremely cost-effective charities by Giving What We Can appears to be well worth the resources invested in it.

Carolina Flores

G8: KEEPING PROMISES? Health

Health has been the focus of a large number of G8 commitments, and the achievement is rated as good. In fact, there has been a lot of progress over the past decades. From 1990 to 2011, there have been:

a 30% fall in malaria deaths, with 1.1 million deaths averted,

24% fewer people became infected with HIV/AIDS in 2011 than in 2001. In low and middle income countries, the figure is as high as 50%,

since 1988 the number of cases of polio has decreased by over 99%, and polio-endemic countries have declined from 125 to 3,

between 2000 and 2011, the percentage of households in sub-Saharan Africa owning at least one bednet rose from 3 to 53%,

in the same period, global coverage in measles vaccination rose from 72% to 84% and estimated measles deaths decreased by 71%.

How do these advances connect to G8 intervention?

The G8 supports health system strengthening through a variety of programmes. The biggest commitment - to provide 60 billion in funding to fight infectious diseases and improve health systems between 2008 and 2012 - was 80% delivered by 2011, so it is on track to be fulfilled. In 2005, the G8 pledged to mobilise support for the 

Global Fund to fight AIDS, Tuberculosis, and Malaria

. So far, these countries have provided almost ¾ of its funding - $13.5 billion. The Fund has provided:

anti-Tuberculosis treatment for 9.7 million people,

G8 countries have also provided support for:

Global Alliance for Vaccines and Immunisation

 (GAVI), with a pledge of $4.3 billion. Since 2000, GAVI has helped 72 low-income countries immunise 370 million children and avert more than 5.5 million deaths,

. In April 2013, $4 billion was pledged to the Polio Eradication and Endgame Strategic Plan 2013-2014.

the development of new vaccines, microbicides and drugs for AIDS, malaria, tuberculosis and virus research. G8 funding has increased by 22% since 2005, support for Product Development Partnerships has grown 77%.

maternal and child health, with $5 billion additional funding pledged in 2010 and support for initiatives such as the 

the development of treatment for neglected tropical diseases, with an increase in funding of 44% between 2007 and 2011, and a commitment to the control and elimination of ten priority neglected tropical diseases was also made last year.

But many important challenges remain. The central problem seems to be the lack of a qualified health workforce in developing countries. The improvement in this area has been negligible. Of the 57 countries below the World Health Organisation threshold of 2.3 health workers per 1000 people, 19 countries have seen an improvement, and 18 countries have had a decrease in density of health workers (there isn't data from the remaining 20). This is a critical situation: without health workers in place, new vaccines or treatments cannot reach many of the people who need them.

Also, we must not forget that AIDS and tuberculosis continue to be big problems. 7 million people in need of anti-retrovirals still lack access to these; and there were still 8.7 million cases of tuberculosis in 2011, with 1.4 million deaths. With the latter disease, there is the added risk of multi-drug-resistance TB.

For these reasons, Oxfam considers the report to be overly generous in this area.

G8: KEEPING PROMISES? Economic Development 

The G8 rate their work in economic development as good, claiming that there has been positive progress in supporting trade and development, including infrastructure and the investment climate, particularly in Africa. Most of the support has been in the form of:

facilitating dialogue with regional economic communities and governments.

As a highlight, over $20 billion have been donated to 

, a world programme aimed at fostering the economic growth of developing countries through increasing their participation in international trade.

On the other hand, progress towards the target of halving remittance costs (from 10% to 5%)- thereby increasing the amount of money entering developing countries in relation to the money sent - by 2014 has been slow, and it seems unlikely that it will be reached.

G8: KEEPING PROMISES?-Aid and Aid Effectiveness

The G8 rated their intervention in Aid and Aid Effectiveness to be satisfactory. Official Development Aid (ODA) from the G8 to low income countries has increased 30% from 2004 to 2012. But,perhaps due to the recession, ODA in 2012 was lower than in 2011. Another negative point is that the commitment to focus aid on low income countries does not seem to have been achieved, as only roughly 1/3 of aid has gone to these countries.

As a strong positive, the G8 are carrying through funding the Highly Indebted Poor Countries Initiative. Their pledge of $1 billion to the debt relief trust fund has been far surpassed: the total donated is now $1.9 billion. In addition, in 2005, the G8 agreed a proposal for debt relief that would cancel these countries' remaining debts, with an agreement to offset their debt and interest on it dollar for dollar up to their burden shares. As of the end of 2012, all had paid the agreed amounts.

Aid effectiveness has become progressively important in recent years. In particular, the G8 are taking part in the Global Partnership for Aid Effectiveness and Development Cooperation, set in 2011. This partnership prioritises the use of monitoring frameworks with the aim of achieving higher levels of transparency in interventions. In conclusion, though good work has been done on debt relief, it seems that the G8 should direct their aid more towards low income countries, and that more work needs to be done on aid effectiveness.

G8: Keeping promises? A look at the Lough Erne Accountability Report

_ This is the first part of a week long series looking at this summer's G8 summit. The G8 Summit was hosted by the UK government on 17-18 June at Lough Erne. One of the important products of this summit was the Lough Erne Accountability Report, a document detailing the progress the G8 have made towards their goals in nine key areas for world development, with a focus on the poorest countries: aid and aid effectiveness, economic development, health, water and sanitation, food security, education, governance,peace and security, and environment and energy._

The heads of government of the countries which form the G8 - the USA, the UK, Japan,Canada, France, Germany, Italy, and Russia - meet every year to discuss an agendarelevant to global development. All eight are in the top twelve countries with the highest gross domestic product, and the sum of their GDPs is about 60% of the world's total. For many, this makes them a symbol of concentration of capital and global injustice, and they have often been accused of not doing enough for international development. Because of their power and economic strength, these countries have huge potential to act upon the world's biggest problems - if they do choose to act, and do so effectively. In this context, accountability mechanisms are most welcome. They allow us to check the progress the G8 are making towards their own compromises, and to know the size of their contribution to important humanitarian goals. In short, they give us material on which to answer the question 'Are the G8 keeping their promises?'.

This is the second comprehensive accountability report on development goal G8 have produced since 2010. It covers the 56 development commitments present in that report, plus those agreed for inclusion in subsequent summits. In the majority of cases, countries are rated collectively, as most commitments have been made by the G8 as a group. Both quantitative and qualitative measures are employed. Whenever possible, the report uses the same data sources as previously (for example, the OECD or the United Nations), to maintain consistency.

In the past two decades, there has been great progress in eliminating global poverty: between 2000 and 2010, the percentage of the world's population living in global poverty fell from 34.1% to 20.6%. This doesn't mean that the G8 are doing enough. Massive problems remain - after all, over one in five people still live in extreme poverty. It is extremely unlikely that the first Millennium Development Goal - the eradication of global poverty by 2015 - will be reached. The progress in all the areas covered in the report is rated as satisfactory or good. In general, there has been an increase in funding, with more money pledged and delivered. Highlights include a 30% increase in official development aid between 2004 and 2012, the inclusion of considerations on aid efficiency, the commitment to donate $60 billion to fighting infection diseases and improving health systems between 2008 and 2012 (which was, in 2011, on track).

However, it is clear that progress has been insufficient in many areas. Some of the funding appears to have been directed into interventions of reduced effect, and some key areas lack funding. For example, only about a third of official development aid goes into low income countries, there are big funding shortfalls in education, and 1.1 billion people still lack access to basic sanitation. There clearly is scope for improvement in aid efficiency.

As an additional issue, the accuracy of the report has been questioned. The Overseas Development Issue (ODI) claims that the report:

lacks benchmarks for individual countries, hiding the significant differences in performance,

uses divergent reporting standards by different donors,

fails to recognise incompatibilities between different G8 policies.

We will look at each area of the report in detail, starting with 'Aid and Effectiveness'.

Clara Marquardt

‘Real World Evaluations’ – randomising on a national scale

Giving What We Can promotes effective altruism - giving informed by rigorous evidence of programmes' and projects' effectiveness. An increasing demand for evidence necessitates a corresponding growth in the supply of such evidence by NGOs & social enterprises. Clara Marquardt from the Cambridge GWWC chapter interned over the summer with 

, an organization aiming to conduct "the most robust evaluation that has ever been conducted of a mass media intervention in a developing country". Clara took the opportunity to discuss the movement towards impact measurement and the challenges associated with real world randomized controlled trials with Will Snell, DMI's Director of Public Engagement & Development.

Clara: Could you give us a brief overview of DMI's objectives and approach?

Will: I would describe DMI as an organisation that uses radio and television to improve health outcomes, especially maternal and child health outcomes, in developing countries by changing behaviours. We are bringing together two different disciplines. One is the creative and logistical expertise in managing large-scale multi-issue media campaigns focused on for example the prevention and treatment of diarrhoea or safe child birth; the second is the scientific rigour of measuring the health impact of those campaigns.

We have developed a statistical model with the 

London School of Hygiene and Tropical Medicine (LSHTM)

 that allows us to predict and then measure not just behaviour change outcomes of a campaign but also health impacts in terms of numbers of lives saved. Historically there has never been any such attempt to measure or model the impact of media campaigns on mortality. This is because the impact on all-cause mortality is very hard to identify statistically given the many possible causes of deaths and the limited impact of any campaign.

The model predicts that DMI's campaigns will reduce under-five mortality by 16-23% within each African country at a cost that is as low as that of any other available intervention…between 2 and 10 USD per DALY.

Clara: You have already mentioned a defining characteristic of DMI - a strong focus on 'science' and impact measurement. How did this come about?

Will: This is a story about Roy Heads, DMI's founder and chief executive who had previously set up and run the BBC World Service Trust's health division and in doing so had worked on a large number of extremely successful mass media health campaigns.He set up DMI with the objective of injecting a more scientific approach because he felt that the impact of these campaigns while significant was not being measured sufficiently robustly.

Amongst other things this was leading to a lack of prioritization of such campaigns by funders and governments…the aim was to prove their impact and thereby persuade donors to make this a public health priority alongside complementary supply side initiatives. The scientific evidence base could also be used to maximize a campaign's mortality impact by informing the choice of health issues to campaign on.

Clara: Is DMI then starting to see this strategy pay off? Are donors and funders responsive to the provision of rigorous evidence? Which type of funder is proving the most receptive?

Will: DMI is starting to see the benefits and an increasing interest among funders. At the moment we have a track record, an evidence base based on behaviour change outcomes. The first time that we are directly measuring mortality outcomes is in the current Randomised Controlled Trial. While that is on going (endline results will be published in 2015) we don't have hard evidence on health outcomes other than a couple of campaigns where have been able to measure trachoma prevalence - in Ethiopia for instance. A lot of funders are responding to what we are doing. By definition the most science driven funders are waiting for the results of the Randomised Controlled Trial - this is gold standard evidence for them.

I would say that the major push for impact measurement is coming from certain bilateral donors who are extremely evidence and research focused, and also from some more public health focused organisations. Both groups of donors are driven partly by value for money considerations… I agree that the development of new financing mechanisms such as Development Impact Bonds (DIB) and the movement towards impact measurement are mutually reinforcing. The development of a DIB market is dependent on a certain amount of accurate data and evidence being available but once DIBs become more widespread this will motivate NGOs and other organisations to become more robust in terms of impact measurement in order to secure continued funding.

Clara: You have already mentioned DMI's on-going Randomized Controlled Trial (RCT) - an evaluation quite unique in its scope and scale. How would you describe the project?

Will: We were in discussions with the Welcome Trust when developing the DMI-LHSTM model to predict how many lives we save and they wanted us to provide some real world evidence that this could work. They funded us to set up a RCT to provide gold standard evidence that observed mortality reductions are attributable to our campaigns rather than other factors.

This is an evaluation design that has rarely if ever been used in the field of media - it is very difficult to randomize media interventions. We looked at nearly every country in Africa and Burkina Faso was the only country where this design could work because radio is very fragmented and localized. While there is a national network very few people listen to it, partly because it is in French, partly because it is under resourced. Everyone listens to local FM community radio stations and TV is more or less non-existent in rural areas. Given the presence of 'media dark areas' we were able to select 14 zones across the country each measuring roughly 50 km radius around a small town and randomly selected 7 to be intervention zones and 7 to be control zones.

Map of Burkina Faso showing intervention areas in blue and control areas in red

In the intervention zones we are broadcasting a 2.5 years child health campaign - a mixture of radio spots (one minute adverts broadcast 10 times a day) and dramas (interactive 5-10 minute health dramas broadcast every evening) (see 

Clara: From a technical perspective, you have already explained how you managed to implement treatment randomisation on a national scale. Are there any other areas where the Burkina RCT did diverge from the idealized 'textbook' RCT methodology? In other words, what are the challenges of actually realizing this rigorous evaluation design?

Will: Sure, the most obvious way in which we have diverged from an idealized design is in the number of clusters. Having 7 intervention and control clusters reduces our statistical power to attribute impact.There are obvious practical reasons why we couldn't go beyond that: There are a limited number of radio stations in Burkina Faso with a sufficient listenership. Also, this is a 12 million USD trial - would we have doubled in size we would have more or less doubled in costs. That notwithstanding we have a 80% statistical power (a one in five chance that we will have a significant impact on mortality but won't be able to prove it statistically).

The other area in which we have faced scientific challenges has been around controlling for other health interventions. There are a number of nationwide interventions including vaccination campaigns that should theoretically affect control and intervention clusters equally. But there have been some health campaigns where there has been a possibility of them broadcasting or operating in some areas but not in others. We have been working closely with the Ministry of Health to try and avoid that happening during the broadcasting period of our trial. That has been possible in most cases so we think that contamination from other projects is minimal… but will never be completely eliminated.

Clara: You have referred to two potential constraints on the implementation of RCTs: Their high cost and their long duration which makes it harder to minimize potential contamination. Based on such considerations what is your take on the World Bank's assessment that "experimental designs…can only be applied in a very small proportion of impact evaluations in the real world…RCTs can probably only be applied in less than 5% of impact evaluations" [1]? Seeing as DMI has already shown that randomization, one of the key challenges, is realisable on a national scale do you see the Burkina RCT as a confirmation that RCTs can be become more widespread?

Will: We knew this from the start - we would not be expecting to run RCTs on a regular basis. We will look at opportunities for future RCTs because it is clearly a level of rigor that other designs will not achieve … but it remains difficult to run RCTs around media in most contexts.

For campaigns in other countries we will probably be looking at quasi-experimental evaluation designs such as non-randomized controls and time series analysis which permits a much greater degree of flexibility in terms of campaign design.

Clara: Would you nevertheless agree that there are ways of reducing the cost and burden of conducting RCTs? Would you in this sense see your strategic collaboration with LSHTM as a model for future RCTs? How about the call for 'piggybacking' on existing data gathering efforts in place of conducting independent and costly baseline and endline surveys?

Will: I do think the bringing together of DMIs and LSHTM's technical expertise is a viable model. Our future research will no doubt be based on similar partnerships. Any partnership brings its own complexities but I think that the benefits to this kind of partnership outweigh these. From the NGO, implementer perspective it makes sense to partner with an academic body with the independence to provide credible results and the necessary technical Measurement and Evaluation expertise. From the academics' perspective, the NGO will hopefully have the expertise in managing and implementing the intervention in question.

I think that there are increasing possibilities of using secondary data sources, especially when a project is in some way interfaced with the health system and outcomes are measured in terms of increased health system usage (which is measured by Ministries of Health) - appointments for children with malaria or sales of Oral Rehydration Solution (ORS) sachets for instance. It has been difficult for us in Burkina Faso partly because there are few reliable data sets and partly because many of the behaviours we are promoting are household behaviours such as breastfeeding that the government is not recording. RCTs demand a level of rigour at every stage of the process that limits the number of appropriate existing data sets.

Clara: Are there any challenges associated with DMI's movement between RCT and quasi-experimental evidence? In other words, is there a risk that once DMI is committed to the RCT standard of evidence funders will discount other evidence?

Will: There certainly is an issue around potential funders waiting for the RCT results despite the fact that our evidence base is already stronger than that associated with most other (mass) media campaigns. In terms of moving from RCT to non-RCT evidence in the future we can break it down into two parts: One is the attribution of impact and the second is the conversion of behaviour change outcomes into lives saved impacts.

The RCT's aim is to as far as possible show that firstly, there is a direct link between the radio campaign and lives saved and secondly, that our model accurately predicts lives saved based on the intermediary behavioural change variable. Assuming that this is the case we will then be able, with reasonable rigor, to say that in future campaigns we don't need to measure mortality which is much more costly for a numbers of reasons including the larger sample sizes required. Instead it will be sufficient to measure behaviour changes and use the 

 (an independent tool funded by the Bill & Melinda Gates foundation that predicts how many lives could be saved in different countries if the coverage of key interventions such as breastfeeding and insecticide treated bed nets was increased from their current level) to translate behaviour changes into lives saved.

The remaining challenge then is to attribute our behaviour change impact to the campaigns. This will require us to use the most sophisticated quasi-experimental designs that strike the right balance between robustness and technical viability - we are working on developing such kinds of evaluations. One example would be dose response analysis: When we are conducting Knowledge, Attitude and Practice surveys (the standard term for surveys that assess people's belief and behaviour changes) we also ask people whether and how much they have been exposed to the campaign. We can then break down the behavioural data by campaign exposure. If people with low campaign exposure changed their behaviour less than people with high exposure that provides us with the additional evidence that the campaign itself was having the impact.

Clara: There has always been a concern over the external validity and generalizability of RCT results. Specifically, a concern over the lack of understanding of the process of change behind observed results and the contextual, mediating factors that enabled the results to come about. In DMI's case these could include supply side health interventions or specific features of the radio stations' broadcasting formats. What is your understanding of these factors in relation to the Burkina RCT? To what extent can you move beyond the statement that "a very specific mass media campaign using context specific stories reduced mortality in the specific socio-political environment of Burkina Faso, Africa by x%"?

Will: The longer and more complex your theory of change the harder it is to generalize the results of an RCT which reveals significant impacts of an intervention. Our theory of change is relatively straightforward with effectively only one intermediary variable and outcome - behaviour.

In order to better understand the link between media messages and behavioural changes we are conducting qualitative research to compliment the quantitative research. We do this partly to inform our message production in terms of understanding barriers to behaviour change and to pre-test spots for accessibility and impact but also to understand responses of our audiences using feedback research with individuals and focus groups. That gives us some idea - anecdotal - to what extent people are changing their behaviours and whether if they are changing them that results from direct exposure to the campaign or from having heard the messages being discussed in social forums, from neighbours and friends.

Clara: Everyone within DMI is very aware of the limitations of the trial's results but how do you communicate this to the public? How will DMI communicate the results so as to maximize their (policy) impact while acknowledging their limitations and contextual background?

Will: If the results are positive of which we are confident - but then you can never guarantee this - the challenge will be to be optimistic but realistic about their generalizability and contexts. We are optimistic about this. While Burkina Faso is an easy country to work in in many ways it is also a very challenging environment in terms of healthcare provision and media penetration. Programs that work in context of Burkina Faso's very limited health care service are likely to work in other countries.

The opposite danger is that the results foster a sense that any media campaign can have a similar impact. We will be doing our utmost to emphasize that much of the success of the trial was due to the way in which it was implemented. Promoting not just the use of media to change health behaviours but a specific approach as to how to do that. We have a developed an approach which we call Saturation+ that comprises three aspects: Saturation, reaching the largest possible proportion of our target audience; science, using the models we discussed to maximize health impacts and to measure health impacts robustly; and stories, investing in both the qualitative research to identify behavioural barriers and a creative process linking local scriptwriters with international experts and expertise.

Clara: A recent publication on Development Impact Bond has called for them to be "open by design", to use a "share standard of reporting and conducting trials", and to "create organizations and funds to…systematize the evidence"[2]. To what extent are these observations relevant to RCTs? In what ways will DMI contribute to the building of a body of evidence on how to conduct real world RCTs? Is there a potential conflict between DMI's social motivation and purpose which is served by growth of evidence-based approaches and the need to protect DMI's expertise and distinctive features?

Will: The DMI-LHSTM model has an intellectual property aspect to it, but all the model does is to model health impacts and thereby maximize campaign impact via message selection. The most effective way to maximize health impacts lies in the approach to running campaigns - our communication strategy will therefore focus heavily on our Saturation+ approach that we will be sharing widely.

Our focus is on sharing the results and the information that helps people run effective media health campaigns rather than on sharing insights on how to run effective RCTs. This goes back to our view that there most likely won't be too be many future RCTs in this area. Where forums such as the International Initiative for Impact Evaluation (3iE) or the Poverty Action Lab (J-Pal) are however interested in aspects of our implementation process we will share this information.

[1] "Conducting Quality Impact Evaluations under Budget, Time and Data Constraints", The World Bank Independent Evaluation Group, 2006

[2] "Development Impact Bond Working Group Report", Centre for Global Development and Social Finance, 2013