A framework for climate services evaluation and its application to the Caribbean Agrometeorological Initiative

Published: 14th June 2018 17:05Last Updated: 14th June 2018 17:05
Fig. 2. Map of the Caribbean with CAMI countries highlighted.

Fig. 2 from the paper: Map of the Caribbean with CAMI countries highlighted.

Introduction

Novel approaches to project evaluations are needed to document the outcomes and lessons to be learned from the numerous and diverse investments international donor organizations, national governments, and regional institutions are making in climate services.

This paper describes an elaborated logic model to structure the evaluation of a climate services program, which we demonstrate in a case study of the Caribbean Agrometeorological Initiative (CAMI). Moving beyond the “loading dock” model of scientific information application, this logic model helps evaluators to address all elements of the provision of climate services – including the quality of weather and climate forecasts and agronomic advisories, the distribution of that information, the uptake of that information, and actions taken by farmers.

Our logic model links the provision of information on weather, climate, and agriculture with decision making, and ultimately with improved social and economic outcomes. While such a logic model necessarily simplifies the full context of any climate services program, it also makes project evaluation much more tractable and generalizable across contexts. Furthermore, this simple logic model can serve to deconstruct conventional thinking about climate services by explicitly addressing the social and process dimensions of climate services that are sometimes neglected in project design, implementation, and evaluation. CAMI partner countries are developing climate outlook bulletins to communicate a three-month seasonal forecast. Despite these high quality seasonal forecasts, we note shortcomings regarding the dissemination of that information, its uptake by farmers, or the ability or willingness of farmers to act on that information.

This paper was originally published in Climate Services (Volume 6, Pages 65-76) In April 2017: http://www.sciencedirect.com/science/article/pii/S2405880716300565#f0010

It is based on an earlier USAID report "The Caribbean Agrometeorological Initiative (CAMI): An Evaluation of Climate Services".

*Download the full text from the right-hand column. Excepts providing details of the methods of and key lessons from the paper are provided below.

Methods and Tools

We offer a more fully elaborated logic model to structure the evaluation of a climate services program, which we demonstrate in a case study of the Caribbean Agrometeorological Initiative (CAMI). Through use of this logic model, we are able to take a full lifecycle approach to the evaluation of the CAMI program, assessing not only the quality of weather and climate forecasts and agronomic advisories, but also the distribution of that information, the uptake of that information, and actions taken by farmers. Our climate services logic model helps identify weak links in the chain of climate services. While we illustrate the practical implications of this logic model by discussing the CAMI evaluation below, the purpose of this paper is to present the logic model itself as a theoretical development worthy of replication in other contexts. The authors believe that this logic model can serve to deconstruct conventional thinking about climate services by explicitly addressing the social and process dimensions of climate services that are sometimes neglected in project design, implementation, and evaluation. The logic model itself should prove useful beyond the Caribbean region and the agriculture sector.

We tested this logic model by evaluating CAMI, a three-year, ten nation, European Union sponsored project that sought to “increase and sustain agricultural productivity at the farm level in the Caribbean region through improved dissemination and application of weather and climate information using an integrated and coordinated approach” (CAMI, 2010; see Vogel et al., 2014 for the full evaluation). CAMI was funded by the EU as an African, Caribbean, and Pacific Group of States’ Science and Technology Programme initiative, and was administered by the Caribbean Institute for Meteorology and Hydrology (CIMH). The initiative involved the countries of Antigua and Barbuda, Barbados, Belize, Dominica, Grenada, Guyana, Jamaica, Saint Lucia, Saint Vincent and the Grenadines, and Trinidad and Tobago (see Fig. 2 for a map of the region). CAMI began in February 2010 and ended in early 2013.


Fig. 1 from the paper: Agricultural climate services logic model. (For large image right click to open in new window or click here: https://ars.els-cdn.com/content/image/1-s2.0-S2405880716300565-gr1_lrg.jpg)

Lessons Learnt

The provision of climate services might not lead to the desired outcome of increased agricultural productivity for a variety of reasons. These constraining factors include low-quality or inadequate information, poor information distribution, inability of farmers to understand the information, and farmer unwillingness or inability to act on that information. In other words, when evaluating CAMI, the context of application involves more than simply the quality and/or quantity of the information produced by CIMH, national meteorological agencies, or national agricultural agencies. Continuing, targeted investments will be needed in order to sustain the climate service capacity built by CAMI. Specifically, we identify the following needs at the conclusion of CAMI:

Fortify cross-agency relationships. In many countries, CAMI was the first opportunity for meteorological and agricultural service staff to work collaboratively. Meteorological services staff must continue to collaborate with agricultural services staff for CAMI-initiated efforts to succeed in the future. Moving forward, CAMI partners should look for opportunities to collaborate with their agricultural services counterparts; this will help build the agronomic capabilities with meteorological services and meteorological capabilities in agricultural services.

Place additional emphasis on agricultural interventions. While some participants identified this as a challenge, it should remain a goal of CAMI partners to clearly articulate crop impacts and agricultural interventions of the meteorological and climate data they provide.

Track information distribution. Currently, CAMI partners do not have a sense of how many farmers they are reaching through their primary climate service – outlook bulletins. In the future CAMI partners should aim to track the distribution of outlook bulletins to better understand their reach. Options include tracking the number of “clicks” or downloads from websites, tracking the number of hard copies distributed, monitoring attendance at forums, and working with extension agents to track information sharing.

Use interactive information-sharing methods. CAMI partners should focus on those information distribution methods that allow interaction with end-users. These methods could include one-on-one contact between extension agents and farmers, forums, outreach to effective farmer organizations, and call-in radio programs. In particular, outreach to informal networks has the potential to spread climate services due to farmers’ reliance on peers for guidance. These methods provide opportunities to ensure that information is conveyed clearly, and allows end-users to provide valuable feedback.

Expand the role of agricultural extension agents. Agricultural extension agents have great potential to communicate climate information with farmers. However, many agricultural extension officers could benefit from additional training on understanding and communicating climate data and agricultural impacts. CAMI partners are already seeking funding to conduct training sessions with agricultural extension agents to increase their capabilities with regard to climate information.

Seek feedback from end-users. CAMI partners should actively seek feedback from farmers on outlook bulletins. This will help ensure that key messages are clearly conveyed, and that their climate services have the information farmers need most. Options for actively seeking feedback include soliciting feedback at farmers forums, tracking questions on radio programs, setting up automated web-based surveys, having agricultural extension officers actively distribute surveys, or sharing websites, email addresses, or telephone numbers where users can provide feedback.

Continue to refine outlook bulletins. CAMI partner countries are still working to determine what information is most valuable for farmers. CAMI partners should continue to refine the content of their outlook bulletins based on changing needs – guided by feedback from end-users.

Develop metrics to measure success. CAMI has not yet defined how it is measuring the primary goal of “increased agricultural productivity.” This goal can be measured through several metrics. CAMI partners should develop a collective set of metrics and begin taking stock of their progress.

Think long-term. Sustainability of CAMI will be a challenge. CAMI is still in the process of scaling-up its climate service and already must seek new funding sources. CAMI partners should seek more stable, longer-term funding if possible.

CAMI is an example of the tremendous amount of ongoing experimentation in the climate information arena under the rubric of ‘climate services.’ Novel approaches to project evaluations are needed to document the outcomes and lessons to be learned from the investments international donor organizations are making. Much of the experimentation in climate services focuses on agriculture, given its traditional role in economic development, but we believe that our logic model could serve as a guide in non-agriculture contexts as well.

Further resources

  • Suggested Citation

    Vogel, J., Letson, D., and Herrick, C. (2017) A framework for climate services evaluation and its application to the Caribbean Agrometeorological Initiative. Climate Services. Volume 6, pp. 65-76. DOI: https://doi.org/10.1016/j.cliser.2017.07.003