When an indicator lies: A 20% increase that wasn't really an increase. Three questions to ask before you trust a result | Associate Writer

Indicators are central to monitoring and evaluation. We use them to track progress, compare results against baselines and targets, and assess whether programmes are achieving what they intended.

But an indicator can be correctly measured and still lead us to the wrong conclusion.

Imagine a livelihoods programme where participants’ average monthly income increases from 10,000 in local currency at baseline to 12,000 at final evaluation. Income increased by 20%. The data are correct, the calculation is correct, and the target may even have been achieved.

Now add one piece of information: inflation was 25% over the same period.

Suddenly the result tells a different story. Participants are earning more in nominal terms, but prices have increased even faster. Their real purchasing power has declined.

Nothing was wrong with the indicator. The problem was moving too quickly from “income increased” to “participants are economically better off.” The indicator was right. The conclusion was wrong.

Inflation is only one example. Depending on what we’re measuring, exchange-rate movements, changes in market prices, seasonality, policy changes, changes in reporting or service coverage, or external shocks can all affect what an indicator means.

None of this is new to M&E. We’ve long recognized that indicators need context and that different sources of evidence can tell different parts of the story. What I want to offer here is a simple way to put that principle into practice: three questions to ask before turning an indicator into a conclusion.

Three questions before drawing a conclusion

1. Is the measure trustworthy?
2. Has something changed that affects what the result means?
3. What other evidence helps us understand the change?

Think of it as a 10-minute interpretation check before finalizing a result. Often it can be done with information we already have.

1. Is the measure trustworthy?

Start with the indicator itself.

Was it defined the same way at baseline and final evaluation? Were the data collected using comparable methods and reference periods? Were the populations comparable? Was the calculation correct?

Two figures can both be technically correct without being directly comparable. Income collected through household self-reporting at baseline and administrative records at endline, for example, may differ because of how income was captured rather than because income changed. The same problem can arise when reference periods or sample composition shift between rounds.

In our example, suppose none of these issues applies. The indicator was defined consistently, the methods were comparable, and the calculation was correct. Participants’ average monthly income really did increase by 20%.

The indicator is trustworthy. But that doesn’t yet tell us what the increase means.

2. Has something changed that affects what the result means?

Now look around the indicator.

In our example, something important changed: prices increased by 25%.

That contextual information changes the interpretation. The indicator tells us that nominal income increased. It doesn’t, on its own, tell us whether participants can afford more than they could before.

Other factors could matter too. Exchange-rate movements may affect comparisons when income is converted into another currency. Market prices can alter household expenses and business costs. Seasonality can make income collected at different times of the year hard to compare directly.

The relevant factor will depend on the indicator. The question is simply: what changed around this indicator that could change what it means?

3. What other evidence helps us understand the change?

Once we identify something that could affect the interpretation, we look for evidence.

For our income example, inflation or local price data can tell us what happened to purchasing power. Participant feedback adds another layer: are households finding it easier or harder to pay for food, transportation, and other essential expenses?

This doesn’t necessarily require another survey. The information may already exist in secondary data, administrative records, programme monitoring, field observations, or qualitative data.

Instead of reporting:

“Participants’ income increased by 20%, indicating an improvement in their economic situation.”

we might report:

“Participants’ nominal income increased by 20%. However, inflation was 25% over the same period, indicating a decline in real purchasing power. Participant feedback also points to increasing pressure from food and transportation costs.”

The indicator didn’t change. Our interpretation did.

Indicator plus

I think of this simple practice as Indicator Plus:

Indicator + context + complementary evidence + a learning question

It’s not intended as a new M&E framework. It’s simply a reminder not to let the indicator become the conclusion.

The learning question pushes us to ask what the result means instead of simply reporting it. In our example, it might be: has the increase in income improved participants’ ability to meet their needs?

The principle isn’t specific to income. An increase in reported GBV cases may be influenced by improved reporting mechanisms. An increase in detected disease cases may reflect expanded testing. Agricultural results may be affected by weather or seasonality.

The contextual factor will change from one indicator to the next. The basic discipline doesn’t: check the measure, check the context, and look at the other evidence before deciding what the result means.

Indicators are essential. They tell us that something changed. But on their own, they can’t always tell us what that change means.

So before celebrating an increase, worrying about a decline, or declaring that a target has been achieved, take ten minutes and ask the three questions.

Sometimes the answers will confirm the story. Sometimes they’ll change it.

Don’t throw away your indicators. Give them company.