Jeffrey Sachs vs. Michael Clemens: the Millenium Villages and Evaluating Impact Assessments – By Magnus Taylor
Jeffrey Sachs and Angelina Jolie: advocates for the Millenium Villages project.
Michael Clemens likes evaluating impact. In fact, it might be more accurate to say that he lives impact evaluation; such is the intensity of his knowledge – theoretical and specific – on this particular avenue of development economics. Clemens’ recent and probably most high-profile work has focused on how impact assessment has been carried out on the Millenium Villages project. He addressed a joint RAS/ODI audience in London on Wednesday 3rd July to talk about the Millenium Villages project and impact assessments
The Millenium Villages project is a joint venture of The Earth Institute at Colombia University (home of Jeffrey Sachs) and the United Nations Development Programme (UNDP). The project is most closely associated with Sachs, author of such rock star development books as The End of Poverty (not uncontroversial in itself – see William Easterly’s review) and on occasion to be spotted consorting with actual rock stars such as Bono and actress Angelina Jolie.
So, projects conceived under Sachs’ brand of Big Plan solutions can be an easy target for development naysayers. However, Clemens isn’t really one of these, he professes that his critical assessment of aspects of the Millenium Villages Project reveals “an incredibly sad story” where “no one wins” – the implication being that Clemens isn’t making criticisms of the project because he wants to, but rather because, as ODI Director Alison Evans pointed out, “this matters – this is a lot of money.”
What is Clemens saying?
The Millenium Villages (MV) projects constitute 14 different locations in rural Africa where poverty is being fought at village level towards achievement of the Millenium Development Goals (MDGs). The project’s own literature states that it fights “˜the root causes of extreme poverty, taking a holistic, community-led approach to sustainable development.’ Clemens contention is that impact assessment has been carried out with such a lack of rigour or basic independence of those conducting assessments, that any claims for the success of the programme are at best dubious, and at worst false.
Impact assessment in the development “˜industry’ is nothing new. According to Clemens “˜randomisation’ is all the rage right now, but randomisation – essentially comparing a group of people who did receive the benefits of a particular development programme with a group who didn’t – does not mean rigour. In the case of the MV programme, no “˜control’ villages that were totally without the benefits of such developmental inputs were studied. There seems to be some contention over whether “˜control’ villages would be possible, or ethical, with the MV project originally asserting that they were not.
In addition, a major contention of Clemens is that a comparison with the base-line of development in the Millenium Villages ie before the projects began, fails to take in to account that many African countries have experienced high levels of economic growth in the last 2 decades, which has been the major cause of improvement in development indicators, notably child mortality. Despite some claims, later withdrawn by the Millenium Villages project, it was found that child mortality in the Villages was declining at about the same rate as it was in the broader districts/regions and countries.
The riposte of Sachs and fellow Millenium Villages advocate Prabhjot Singh stated that “with communities, there are no true controls – life changes everywhere, in the MVs and outside them.” Which when it was published in October 2011 was described by David Mackenzie – Lead Economist, Development Research Group, World Bank – as “˜baffling.’ For a more detailed piece on the academic back-and-forth as it took place in October last year, there’s a good piece on the development blog View from the Cave.
So, why is this seemingly technical discussion between development geeks interesting/relevant to people with a wider interest in contemporary Africa? And by extension, why did the RAS and ODI invite Michael Clemens to talk about his views on MV and impact evaluation? As mentioned before, for projects that cost vast amounts of money, and in a climate where there isn’t a never ending pot of aid money to spend, it matters how we use it. Therefore it matters that we assess the effectiveness of projects properly. Would aid donors be better off spending their money and political capital on something that is proven to work eg bed nets to stop transmission of malaria by mosquito?
Impact assessment should also force us to assess what impact development assistance is having in relation to the broader economic changes that are at present constantly championed by optimistic commentators.
Clemens’ criticisms of impact evaluation and the MV projects have seemingly created some tension and bad-feeling between a number of economists with a stake in either camp. As was evident at the RAS/ODI meeting, much of this comes down to whose figures you find most credible, which for the non-specialist, can become somewhat bewildering.
To read Michael Clemens’ presentation in full click here
To watch a video recording of the event click here
Magnus Taylor is Managing Editor, African Arguments Online
Magnus, thanks for your interest in this subject. I’m extremely grateful to RAS for co-hosting and for the time and thoughts of everyone who watched and participated.
I want to clarify one thing that is incorrect in the post above. There is no doubt or controversy about “whose figures are more credible”. The Millennium Villages Project has been forced to admit, by our work, that its figures were false. The editors of the journal determined this, and they eventually admitted it themselves.
The principal result of their flagship research product on child mortality (the central outcome in their own evaluation protocol) was discovered by myself and Gabriel Demombynes to be false. Upon learning of this, the world’s leading medical journal, The Lancet, forced the project to retract this finding. While they had been announcing that child mortality fell much more rapidly at the project sites than nationwide, it was revealed—only through our efforts—that the fall in child mortality at these sites simply matched the overall nationwide fall occurring at the same time. Since then, Jeffrey Sachs admitted that we were correct and that the numbers published by the project had been wrong. The project’s head of monitoring and evaluation lost his job over this incident.
In the presentation I was trying to be diplomatic and kind, so I did not place too much emphasis on these facts, which have irreparably damaged the credibility of this project. But apparently I was so diplomatic that the truth was not clearly presented. So here I am spelling it out in starker terms, for clarity.
Thanks again for the writeup and for your engagement in the debate!
There are two processes at play here. That the MV are not delivering as they claim to be and the value of impact evaluations themselves. Careful analysis of the intervention can tell usnalotnabout it’s effectiveness, often enough to make a decision. In some cases though IEs can be useful, but we should not forget that they can be rather expensive themselves.
As with the intervention we need to be careful about the research methods we choose.
The presence of Jeffrey Sachs and Angelina Jolie for this Good Work really inspire others to come forward and help the poor and eradicate poverty from our society.