Using wellbeing indicators and climate information to assess adaptation effectiveness

Submitted by Julia Barrott | published 19th Feb 2016 | last updated 13th May 2019

Introduction

Adaptation is increasingly heralded as the means to securing development in the face of climate change. But how can we be sure that it is effective?

One option is to use a range of ‘wellbeing indicators’ alongside climate information to monitor and evaluate adaptation over the short and long term. Wellbeing indicators overlap to a large extent with standard development indicators. But they are influenced by variations and trends in climate and, if used alone, can paint a misleading picture of adaptation effectiveness. By combining them with climate information — data on how key climate variables and hazards change over time — evaluators can ‘calibrate’ wellbeing indicators to provide a more accurate assessment of adaptation activities.

This IIED Briefing* presents approaches for measuring the effectiveness of adaptation in the long- and short-term using climate information and wellbeing indicators. It discusses potential explanations for varying trends in wellbeing indicators in the face of climate change, and the use of counterfactuals, control groups, event comparisons and statistical models for assessing adaptation progress.

*download available from the right-hand column and via links provided under Further Resources.

Policy Pointers

Familiar indicators of human wellbeing and development performance can be used to evaluate adaptation effectiveness, if they are interpreted in the context of climate information. 

Over long timescales, planners can evaluate adaptation by asking whether wellbeing is stable or improving in the face of intensifying climate hazards. 

Over shorter timescales, adaptation can be evaluated through comparisons with ‘no-intervention’ counterfactuals.

Evaluating long-term adaptation

Development planners need to evaluate the extent and success of adaptation over long timescales at national, sectoral and local levels. They can do this by analysing long-term trends in wellbeing indicators — in the context of climate information, which shows whether hazards are getting worse, not changing at all or becoming less severe or less frequent. Three scenarios illustrate how climate information can alter the interpretation of trends in wellbeing indicators (see Table 1 from the report, provided below).

Scenario 1: Wellbeing indicators show clear improvement. The most likely interpretation of this scenario is that adaptation is occurring and is enabling people to improve their wellbeing despite climate change. But climate information is required to confirm this interpretation, as it is possible that hazards are not intensifying or they are reducing in frequency or severity.

Scenario 2: Wellbeing indicators show no significant change. This might be because adaptation has made little or no difference to people’s wellbeing. But, if hazards are intensifying, adaptation is likely to have stabilised a development situation that would otherwise have deteriorated. Unless development planners understand how climate hazards are evolving, they will not be able to interpret such a scenario: they might mistakenly conclude that adaptation is not effective, and remove support that is preventing a decline in wellbeing.

Scenario 3: Wellbeing indicators show a decline. A crude conclusion here is that adaptation has failed. But it is possible that adaptation has prevented an even greater decline in wellbeing. Planners must assess whether climate hazards have intensified, and whether they have done so to such an extent that adaptation activities are inadequate. In other words, has adaptation actually failed or is it delivering ‘invisible’ benefits? The answer to this question will enable planners to make informed decisions on abandoning or augmenting their adaptation activities.

Implicit in all these scenarios is the idea of a ‘counterfactual’ — a hypothetical scenario or narrative in which the same hazards occurred but there were no adaptation activities. By comparing these counterfactuals with reality, planners can better understand the true impact of adaptation.

In the counterfactuals for scenarios 1 and 2, climate change prevents development from improving wellbeing, or results in a decline in wellbeing. If hazards are intensifying but wellbeing is stable or improving, planners can conclude that adaptation is happening without the need to construct a counterfactual.

Scenario 3 is more complicated, as it involves comparing different degrees of decline in wellbeing. Here, it is important to build a counterfactual that describes how much wellbeing would have declined without adaptation. 

Evaluating adaptation in the short term

Monitoring wellbeing indicators over the long term may provide government planners with a good assessment of adaptation effectiveness. But it is not always feasible or appropriate for evaluators of individual projects and programmes, which often last only a few years. Nonetheless, wellbeing indicators can still be defined at the impact level and monitored over short time periods (a few years at most).

How this is done will depend on the availability of information, resources and expertise, as well as on the nature of the relationship between the relevant wellbeing indicators and climate variables. All methods still use climate information to put wellbeing indicators into context. Where they differ is in their approach to developing counterfactuals, and in their use of quantitative data.

Three options for assessing the impact of adaptation in the short term are outlined below.

  1. Control groups. Changes in wellbeing indicators can be compared between beneficiaries of an intervention and a control group (those who have not benefited) through a randomised control trial (RCT) or ‘difference in difference’ (DiD) approach. RCTs require large sample sizes and will probably only work for a small proportion of projects or programmes. Conversely, DiD studies use small but representative groups of beneficiaries and non-beneficiaries and can be used to compare wellbeing indicators before and after the intervention. In either case, it is important to use climate information to confirm that both groups have been exposed to the same climate hazards. (Of course, the comparison groups should be as similar as possible in other respects too, to help rule out any other reasons for difference in wellbeing indicators.)
  2. Event comparisons. Two or more similar hazards might occur shortly before, during, or soon after an adaptation intervention. If this is the case, it might be possible to compare the effects of these hazards and use participatory surveys and beneficiary narratives to investigate the reasons for any differences. Questions to ask here include: Have losses or other adverse impacts been reduced over time? Is the intervention responsible for these reductions (in whole or in part)? If so, to what extent did it contribute to them? Climate information is again important to ensure that the hazards being compared are sufficiently alike. For example, daily pressure, rainfall and maximum wind speed data can be used to characterise storms and identify events of similar magnitude occurring in the same district. But it is also important to ensure that non-climatic factors that affect people’s vulnerability to the hazards in question are sufficiently similar. For example, price rises in key commodities might mean that people have less to invest in reconstruction after a disaster. These price rises might be driven by changes in the global economy, national policies, or climate disasters in exporting countries.
  3. Statistical models. Some wellbeing indicators are historically strongly correlated with climate variables. For example, annual growth in gross domestic product in sub-Saharan Africa is closely related to rainfall over certain periods.2 Famine in northern Nigeria has been associated with deficits in rainfall above a certain magnitude.3 And increases in mortality in urban centres across the world occur when temperature and humidity indices rise above certain thresholds.4 Such historical relationships can be used to model variations in wellbeing indicators, over time periods that include— and extend beyond — an adaptation intervention. The difference between these modelled changes to wellbeing and the actual changes measured on the ground can then be attributed to the adaptation intervention — assuming that other explanations can be discounted.

Of course, approaches based on statistical modelling require both wellbeing indicator data and relevant climate data to be available over long periods (preferably several decades or more). And they are only applicable where clear relationships exist between relevant variables.

All three of the ‘short-term’ approaches — control groups, event comparisons and statistical models — can also be used to monitor the effectiveness of adaptation outside individual project and programme contexts, over both the short and long term. For example, they might be used to assess the effectiveness of adaptation that arises from changes in policies and governance; or they might be used to assess the cumulative impact of multiple adaptation projects or programmes.

Further resources

  • The International Institute for Environment and Development (IIED) promotes sustainable development, linking local priorities to global challenges. We support some of the world’s most vulnerable people to strengthen their voice in decision making.

    Garama 3C Ltd is a small UK-based consultancy speacialising in climate change and international development, with a focus on adaptation, mainstreaming, M&E and the delivery of related training in the UK and globally.

    This research was funded by UK aid from the UK Government.

     

    Suggested citation:

    Brooks, N. (2014) Using wellbeing indicators and climate information to assess adaptation effectiveness. IIED Briefing. London.