Howard White of 3ie discusses some of the problems we often see in how people handle data in this post, “Using the Causal Chain to Make Sense of the Numbers“. The essay makes many excellent points which are relevant both to how programmes are designed and how they are evaluated.
However, I have to take issue with one section:
And different ways of presenting regression models can give a misleading sense of impact. A large reduction in relative risk – a ‘good odds ratio’ – can reflect quite a small change in absolute risk. Three randomised controlled trials have found circumcision reduces the risk of transmission during unprotected sex by around 50 percent. The reduction in risk was from around 3.5 percent to 1.5 percent. Just a 2 percentage point absolute reduction, so 50 men need to be circumcised to avoid one new case of HIV/AIDS.
This is a bit misleading. In assessing effects we are interested both in relative and absolute effects, yes. But White fails to acknowledge here that the absolute risk at the outset (3.5% in the case of the pooled results of the three trials) is a characteristic of the people being researched. And indeed the absolute risk in the populations in the three studies (Kenya, South Africa, Uganda) was different. The 3.5% and 1.5% figures come from pooling the results of the three trials. If study subjects had come from a population where the pre-existing HIV prevalence was higher, and risk factors (including unprotected sex) were higher, then the baseline absolute risk would have been more than 3.5%. If the risk factors had been lower, the baseline risk would have been lower. White’s estimate that 50 men need to be circumcised to avert one infection is not universally valid. In some places it will be many more; in others it will be fewer. This is one of the reasons male circumcision is primarily promoted in higher HIV prevalence settings.
Having said that, “just” a 2% absolute reduction is actually pretty good when compared to other HIV prevention interventions. Especially when you consider that once a man is circumcised, he stays circumcised, so the risk reduction is permanent. Look at it another way: if the intervention being tested led to a 100% risk reduction, then (according to White’s post), that would be “only” a 3.5% reduction in absolute terms. Still doesn’t look very impressive, does it? Except in this case there would be no new infections whatsoever.
The reason the results of these trials (and any trials) are reported as relative risks is because if you want to estimate what the effects of the intervention might be in another population, you have to apply the relative risk reduction to the absolute risk in each and every population. Reporting it any other way is misleading. White is of course correct that absolute risk reduction is what matters when looking at the overall effect of an intervention or policy, but absolute risk reduction is a function not just of the relative risk reduction of that intervention, but of all the other relevant factors.