10 Jun Cómo usar Outcome Harvesting para evaluar el Impacto
Have a look to this interesting debate in the Outcome Mapping Learning Community:
Dear Julius and colleagues,
Thank you for challenging us to respond to your inquiries. Certainly I have not used OM nor OH in the evaluations I conduct that have to respond to the five mentioned criteria: Relevance, Efficiency, Effectiveness, Impact, Sustainability.
However both OM and OH have always called my attention and I have taken different courses to be able to use the methodology. When? When I see the opportunity….Again….When do I see the opportunity? With a client that is sufficiently open to new ideas and metholodogies, and less constraint by the «Development International Principles….rules» (whatever you call it) (OECD standards, etc.).
In my experience, private foundations may be more open to these methodologies…..OM is particularly useful when planning a project. I have used it with an important US private foundation that works with institutions beneficiaries in third countries that are not familiar with cooperation development industry concepts and methods (nor for instance with Logical Frameworks). And OM was really helpful at the planning stage. It helped people to get into the idea of What change do we really want?….., What alliances do we need then….?
I know I did not respond to you on the sense you expected, but this is the reality I face. However, if OM and OH is getting more used with «less constraint clients», it will eventually get more known and hopefully accepted, valued and spread…..
Ana García Femenía
«Thank you Julius for continuing to push us on sharing our views of using Outcome Mapping for evaluation. The International Year of Evaluation is half over and there is no time to waste. And thanks to Ana Maria Garcia Femenia for sharing your concerns about the viability of using OM or OH to assess any of the five OECD-DAC criteria. I would like to take up Julius’s challenge and comment on using OM for evaluating impact. There are two reasons for doing so.
First, I believe it would be an enormous contribution during the International Year of Evaluation for the OMLC to address the issue of impact assessment.
Second, I confess that I have become increasingly uncomfortable with my long-standing position on impact. My view has been inspired by that in the Outcome Mapping manual (see especially the section Why Not Impact?, pages 5-10) and, above all, has been based on my own experience in development work during virtually my entire adult life.
Basically my argument has been that when you are dealing with complex challenges – i.e., most of the human development and social change that I know of -, it is more sensible to focus on outcomes. Those results, I have maintained, are the closest you can come to understanding what an intervention has contributed to impact as defined in the OM manual: «changes in state – for example, policy relevance, poverty alleviation, or reduced conflict» (page 2) that affect social and environmental well-being. This is, of course, a position held by a number of colleagues around the world.
My rejection of impact evaluation was first challenged in 2006 with the publication of When Will We Ever Learn? Improving lives through impact evaluation. The ensuing turmoil around the need or not for scientific proof of cause and effect as the gold standard for assessing what a development or social change intervention has achieved, confused the issue for me, and I know for many others as well. Therefore, I suggest that the first task in exploring how Outcome Mapping and Outcome Harvesting may be used for impact evaluation is to agree on what we mean by «impact».
This is not straightforward because out of the almost ten years of confusion, controversy and discussion have emerged four definitions of «impact».
1 – The most common or popular definition continues to be that «impact» is the result of an action, that is, impact is any effect of an intervention. This definition does not, however, clearly distinguish between the different types of results, namely, outputs, outcomes and impact.
2 – The Abdul Latif Jameel Poverty Action Lab (J-PAL) and the International Initiative for Impact Evaluation (3ie) define impact as the difference between what an intervention achieved (the factual), and what would have been achieved without intervention (the counterfactual). This establishes the essential criteria that the effect is the difference between the results achieved and those that would have been achieved anyway.
3 – The Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD-DAC) maintains that «impact» is the positive and negative, primary and secondary long-term effect produced by an intervention, directly or indirectly, intended or unintended. This definition does not address the issue of whether or not the effect would have happened in any case.
Incidentally, for an up-to-date, fairly concise (35 page) review of the second and third definitions, see Impact Evaluation: A Guide for Commissioners and Managers, by Elliot Stern for the Big Lottery Fund, Bond, Comic Relief and the Department for International Development, published last month.
I am persuaded that the evaluation of impact of an intervention must be about the long-term effects that would not have occurred anyway. But there is still another factor to take into account in the light of the recently declared 17 universally applicable Sustainability Development Goals (SDGs).
4 – A definition of impact must incorporate the economic, political, social and environmental requirements for sustainable development. According to the UN’s SDGs this would include: «poverty eradication, changing unsustainable and promoting sustainable patterns of consumption and production and protecting and managing the natural resource base of economic and social development».
In sum, taking what I consider the essential features of these four definitions, my working definition of impact is:
Impact is the long-term, sustainable changes produced by an intervention – that would not have occurred anyway – in the conditions of people’s lives and the state of the environment and which reduce poverty, improve human well-being and protect and conserve natural resources.
Does this definition ring true to others? Can you see Outcome Mapping contributing to assessing impact so defined? Do you have an evaluation experience to share in doing so?
For my part and speaking for Outcome Harvesting, I believe that indeed it is an approach that can be used to assess sustainable development impact as I have defined it. In fact, I am currently exploring with a colleague undertaking an evaluation in which we will use OH and other bottom-up tools to assess the different dimensions of the sustainable impact of an intervention in an African country. We will gladly share that experience with you as soon as possible».
Ricardo Wilson-Grau Consultoria em Gestao Empresarial Ltda
Evaluation | Outcome Harvesting
Rua Marechal Marques Porto 2/402, Tijuca, Rio de Janeiro, CEP 20270-260, Brasil
Telephone: 55 21 2284 6889; Skype: ricardowilsongrau
Direct via VOIP, dialing locally from USA or Canada: 1 347 404 5379[/list][/list]