How does an international team of scientists generate and deliver research results that
will significantly, sustainably, and measurably benefit large numbers of poor people, many
of them small farmers?
That mouthful of a question goes to the heart of CIAT’s work. But it
also has serious riders: How do you preserve the natural resources—forests, soils,
water, biodiversity—that underpin human livelihoods? How do you gauge success when
the bottom line isn’t purely economic? How do you attribute credit for success when
multiple influences are at work? To ensure adoption of new technologies, who should be
involved in the research and its evaluation?
These are a few key issues surrounding the measurement of research impact.
In recent years, though, many donors and research centers, including CIAT, have been
rethinking the very notion of development impact. It’s proving to be a slippery
concept, even for veteran donors that have operated aid programs for decades.
The winding road to impact
The experience of Canada’s International Development Research Centre (IDRC)
provides a useful example of how donors and other agencies are grappling with impact
assessment. Some years ago IDRC undertook a major evaluation of 64 completed research
projects that center staff had rated as successful.
"Disappointingly, at the conclusion of these studies, ‘impact’ continued
to elude us," says Terry Smutylo, director of IDRC’s evaluation unit. "The
reports focused heavily on outputs, providing little data on reach and even less on
impact." Although the external evaluators were armed with an analytic framework to
guide their inquiries, they didn’t have at their disposal the necessary data
collection instruments to zero in on impact.
IDRC’s two decades of experience with research evaluation led it to experiment
with a different approach, called outcome/impact mapping. The assessment of impact unfolds
in parallel with the R&D process itself, not as an afterthought. The expected impact
of research and strategies for achieving it are stated at the outset. Thus, an assessment
of the project conducted after research results have had ample time to be adopted and
benefit people can be based on clear criteria and identifiable milestones.
The method steers away from trying to prove hard causal links between research and its
ultimate "impact" on people’s day-to-day well-being. Rather, it maps out a
plausible route between research and development, a journey that may have many twists and
turns. The method shows how a research project’s outputs, like improved technology
and training, contribute to "intermediate outcomes," especially positive
responses by direct partners in research. These may be producer organizations, community
development groups, government agencies. or NGOs. The intermediate outcomes, referred to
as the research project’s "reach," constitute a kind of soft or preliminary
impact. The mapping then continues with analysis of how those outcomes, in turn, might
influence the broader community of intended beneficiaries.
Research partners and other stakeholders, including end users of research results, are
expected to participate fully in the mapping work, as they also stand to benefit from
lessons learned. The idea is to create an open learning milieu for all stakeholders. Among
other things, they provide valuable information on the complex local environment in which
innovations flourish or wilt.
The attribution gap
A complicating aspect of impact assessment is that multiple actors and factors
invariably mold the development process. For example, the very act of building research
partnerships to maximize the eventual benefits of research makes it difficult to say in
the end who exactly contributed what. This is one element of what has been called the
"attribution gap."
Another source of uncertainty is the decision-making behavior of end users and their
relations with other gatekeepers of technological change. "Innovation in agriculture
happens through the interaction of farmers, veterinary doctors, district extension
managers, farm advisers, technical specialists, and many more," writes Thomas Kuby, a
senior professional with the German Agency for Technical Cooperation (GTZ). From analysis
of those interactions, he says, we now understand innovation to be "a social
process."
In classic linear models of innovation—moving from research, to diffusion, to
adoption, to development impact—social dynamics have not been given much attention.
Yet, they are a potent ingredient of impact, with the annoying tendency to twist the
straight arrows of flow charts into pretzels. Thus, the playing field on which research
aims to catalyze or "push" human development and alleviate poverty is rather
chaotic.
However, as Kuby declares, "to say that impact assessment is difficult does not
mean that it’s impossible." He and his colleagues, like their counterparts at
IDRC, are trying to straddle the attribution gap that emerges from the "maze of
complexities" lying between a specific research output and wider development trends.
GTZ’s strategy is based on a variation of its now classic goal-oriented planning
model for projects and programs called ZOPP. The "impact chain" has seven
components: inputs, activities, outputs, use of the outputs, direct benefits, indirect
benefits, and highly aggregated development changes. Evaluation is divided into two
realistic, manageable tasks. Since the attribution gap lies between the assessment of
direct and indirect benefits, this is where the division of labor is made.
Kuby likens GTZ’s method to building a bridge over a river. Spans are started on
either side and eventually joined in the middle. To avoid the risk of collapse, the weight
of each arc is supported independently until the very end.
GTZ staff working on the project being assessed deal with the first five components.
It’s their job to monitor and account for everything that happens up to and including
the achievement of direct, measurable benefits from the agency’s work. Beyond the
attribution gap—the other side of the river, so to speak—assessment is assigned
to evaluators outside the project or program. They examine spin-offs and overall
development impact from a broad perspective.
Reminiscent of IDRC’s approach, the final step is to argue for a
"plausible" link between these two spans of the impact bridge. In assessing
project contributions to wider development benefits, contends Kuby, "the false ideal
of ‘scientific proof’" must be abandoned. "In the political arena
where the funding decisions are made, plausibility lies at the core of credibility. People
know that development is difficult and complex. Whilst they expect accountability, they
will, in the long run, believe plausible arguments more than bombastic ‘proofs’.
Stories without facts will not do."
Getting from R to D
At CIAT the impact assessment team and other Center projects share the task of tracing
plausible paths from research outputs to development outcomes. The impact team consists of
six full-time equivalent professionals, with strong emphasis on economics. Since the group
is small, it can’t evaluate every CIAT research project or output. So, it focuses on
key research projects for which there is a clear need for impact data.
In doing so, the impact assessment group estimates either the likely benefits of
potential future interventions (ex-ante studies) or the actual benefits of past
interventions (ex-post studies). For example, an economist recently analyzed the potential
economic impact of vertical tillage and sustainable agropastoral cropping systems. The
work focused on a major agroecosystem of South America: some 320 million hectares of
savannas, mostly in Brazil, whose acid soils suffer from low nutrient levels.
Innovations covered by the study include the use of fertilizers and lime to build soil
fertility, methods to break up compacted soil, and suitable cropping patterns. Soil maps
from the Food and Agriculture Organization (FAO) of the United Nations and the UN
Educational, Scientific and Cultural Organization (UNESCO) were integrated into a
geographic information system (GIS). This made it possible to visualize how the estimated
economic value added from applying various technologies would differ from one savanna area
to another, or from one country to another.
The economic benefits of adopting vertical tillage in the savannas over 25 years were
estimated at US$52.2 million for Bolivia, $4.4 billion for Brazil, $30.5 million for
Colombia, and $116.4 million for Venezuela. The study concluded that the CIAT innovations
in question are economically viable options for sustainable agricultural production in the
savannas.
The utility of such a cost-benefit analysis is clear. It can help national planners
make informed decisions about agricultural investments that may affect millions of people.
And it can arm CIAT with the information it needs for setting future research priorities.
On the ex-post side, the impact assessment unit recently documented the use of improved
rice, bean, cassava, and forage germplasm in 28 countries of the tropics. Part of a global
study of international agricultural research, the CIAT component looked at germplasm to
which the Center made a key contribution.
Results indicate that both release and adoption of CIAT-related varieties have gone up
over the years. The findings contradict the view that the impact of the Green Revolution
has been winding down. True, in many cases the yield gain from new varieties is lower than
in earlier decades. But many farmers prefer the newer varieties, because they resist
disease and other stresses better, providing an added measure of food and income security.
The study’s germplasm-adoption figures vary widely, depending on the crop and
country. In some instances, though, the national area planted to new varieties with CIAT
content, particularly rice and beans, has been extraordinarily high—between 82 and 95
percent. Such seed-based technology clearly can promote more secure rural livelihoods and
affordable staple-food prices for both the urban and rural poor. The link between our crop
research and the positive impact on developing countries is, we believe, a plausible one,
backed up by a large body of evidence.
Seeds and beyond
But CIAT recognizes that poverty is complex and variable and must be attacked on many
fronts at once. Contributions from both social and natural scientists are essential.
Better germplasm, based on state-of-the-art science, is just one element in CIAT’s
overall strategy.
"By themselves, new seeds don’t necessarily help people manage their natural
resources better, find new markets, or improve their community development capacity,"
says economist Douglas Pachico, CIAT’s director for strategic planning and impact
assessment. "So, we’re now developing other kinds of research outputs as well.
But there are a lot of unknowns in trying to measure their impact." In the case of
evaluating research on natural resource management, for example, "we haven’t
reduced these complex problems to cookbook solutions."
The multiple research products Pachico is talking about include agronomic practices and
advice, decision support tools, organizational methods for empowering rural communities,
as well as new germplasm. "We increasingly see our CIAT outputs as different types of
information," he says. "It may be about managing soils on farm, or ways in which
a community can structure itself to make better decisions, or about new crops and market
opportunities."
Economic analysis of impact
Different interventions, though, demand different methods of impact assessment. In the
past CIAT has relied heavily on traditional economic analysis, especially calculating the
dollar value of increased production resulting from new agricultural technology.
This approach, says Pachico, has strengths and weaknesses. On the positive side,
economic analysis is highly systematic. It has well-developed statistical tools for
adjusting monetary data to account for factors like inflation and the purchasing power of
different currencies. Results can therefore be compared across regions and time periods.
And because key conclusions are often expressed in common measures like percentages and
dollar values, they can be easily grasped by nonspecialists.
In addition, says Pachico, "there are real links between wealth generation and
well-being, even if the correlation isn’t perfect. Wealth doesn’t buy happiness,
but it does create options to fulfill needs." Lower food prices have a similar
effect, especially on the poor, who spend a large proportion of their income on food. An
advantage of economic analysis is that it can clearly and convincingly correlate the
results of agricultural research with income and price trends.
However, economic analysis runs into difficulties when the benefits to be measured
don’t have a clear market price. Political empowerment of a community, new skills
gained by farmers who participate in research, gradual improvements in soil fertility, and
enhanced biodiversity are hard to express in dollars. While economists are continually
working on methods such as "shadow pricing" for quantifying such benefits, these
don’t fully capture social and environmental values.
Economic analysis will continue to play a vital role in impact assessment at CIAT. But
as we diversify our research outputs, it must be combined with other disciplines and
perspectives to properly assess how equitably benefits are being distributed. Besides
measuring overall economic surplus generated by innovation, we need to know what kinds of
people were affected and what the changes mean to their well-being.
To document impact in this way requires an understanding of how people perceive
benefits. Current thinking thus puts great emphasis on involving the rural poor not only
in research but also in assessing outcomes. Over the past decade, CIAT has promoted and
experimented extensively with farmer participatory research. We’re now exploiting
that experience to improve our impact assessment methods.
Mapping impact
At the same time, CIAT is paying close attention to the experience and thinking of
agencies like IDRC and GTZ, as described earlier. Our goal is to be able to map outcomes
and impact in a way that brings noneconomic factors into the equation.
One method now under development defines a framework for assessing the benefits of
natural resource management research that CIAT is conducting or applying in five reference
sites. These "living laboratories"—in Honduras, Peru, Nicaragua, and
Colombia—represent three types of agroecosystems typically populated by the rural
poor: hillsides, forest margins, and savannas.
Our impact assessment scheme is based on a "research-to-development"
framework. In a nutshell it allows us to track the life cycle of specific CIAT research
outputs—from their introduction to partner organizations, to their adaptation,
adoption, and application by community members, through to observable changes in
people’s welfare and in the health of the natural environment.
Several kinds of CIAT research outputs or interventions can be mapped with this
approach. One is organizational models, such as procedures for setting up a participatory
research project or for strengthening local networks. Another is technology and
information; environmentally friendly cropping methods for hillside farms is a good
example. The third main type of CIAT output is decision-support tools—for instance,
methods to help farmer groups exploit new market opportunities.
As with most impact assessment methods, the idea is to find out "what really
happened," or what’s likely to happen. To do this you need to know the
community’s initial state of development and, after the intervention, its modified
state. For each step along the way, researchers and their partners agree on concrete
indicators of progress.
Here, the concept of "capital"—the human, social, technological,
natural, financial, and physical resources available to individuals and
communities—is a useful tool for framing the "before" and "after"
states.
For each CIAT output, impact assessors construct a "causal uptake path" that
leads to the accumulation of various kinds of capital. The resulting flowchart first
identifies the process being influenced—local coordination and organization,
community decision making, or economic production—as well as the specific partners
involved. Next, intermediate outcomes are recorded. How did the partner organizations
react to the intervention? Was there unbridled cooperation? Widespread apathy? Careful
negotiation?
CIAT’s research-to-development framework is just one approach to answering an
increasingly tough question: Are we succeeding in our science-based mission to improve the
livelihoods of the rural poor in developing countries? Undoubtedly, other assessment tools
will be needed, since the nature of the solutions we propose is so varied.
As a producer of international public goods, CIAT deals in multiple forms of
development capital. Some grow rather slowly, and some defy measurement in dollars, pesos,
or any other monetary unit. Our investors, then, cannot flip through the business pages of
a newspaper to see whether our "stock" is up or down. Indicators of success and
impact must be based on many hard-won facts, careful analyses, and interpretations.
The rest of this report presents examples of CIAT’s contributions to development
impact—including past work as well as recent efforts to extend the gains. We hope our
interpretations of the available facts have a ring of plausibility.
|
