Value for money conundrum


Aid donors are increasingly paying attention to value for money in their programmes.  While donors have always wanted the biggest “bang for their buck”, in recent years there has been a step change in the approaches and tools for calculating value for money and in the expectations placed on funding candidates (whether they are governments or NGOs) in terms of demonstrating their value for money (VFM).

Good.  But there’s quite a big catch which, as far as I am aware, is not being addressed.  Donors are applying VFM analysis to their own investments but they aren’t always applying it to the “entire” investment being made in a country.

Take the global response to AIDS, for instance.  As of 2011, for the first time in history, less than half of all money spent on HIV and AIDS in low and middle income countries came from international donors – 51% came from national sources.  But the extent to which the 49% can be spent on programmes that are “VFM certified” largely depends on what the 51% is being spent on.

First example.  In one country, most of the international aid money for AIDS comes from the Global Fund to fight AIDS, Tuberculosis and Malaria.  A good chunk of that money is spent on antiretroviral drugs.  But not any old ARVs: just the most expensive ones. The country produces all of its first-line ARVs – the ones that are the cheapest to manufacture, and that bring economies of scale because they are the ones that are needed at the greatest volume.  What the country  needs assistance for is the second and third line drugs – the ones that have to be imported, and that have to be purchased in much smaller quantities.  For the same budget, the Global Fund could, theoretically, be paying for far more treatments, but it isn’t.  But the imported drugs are no less essential – once someone fails on first line treatment, it is essential to move them on to the next regimen.

Second example.  It is well known that the highest VFM proposition for spending on HIV prevention is to invest in comprehensive programmes with highly affected populations like men who have sex with men, sex workers, and people who inject drugs (“key populations”).  Despite some improvements in recent years, programmes with these populations continue to be chronically underfunded.  Even more significantly, over 90% of money for these programmes comes from international aid sources.  National governments don’t like spending money on this area of programming at all.  In this case, international donors are getting the best VFM (providing the programmes are being implemented properly).  In the longer term though, if the trend toward higher shares of domestic funding continues (and there is no reason to suppose it won’t), these programmes look to be at very high risk of losing support.

A straight contest between the first country and one of the countries where donors pay for all of the key populations work would suggest that the latter is a far more suitable investment.  But it is not that simple. In both cases donors are filling a shortfall in a national response – albeit for different reasons.  It may be that when looked at as a whole, the first country’s response is more economically efficient, even if the aid investment doesn’t make it seem so.  Moreover because things like effective HIV prevention and treatment are made up of many interconnected and complementary interventions, and indeed because prevention and treatment are themselves interconnected and complementary, the cost effectiveness of each intervention often depends on other interventions being implemented.

When looking at value for money, major international donors should think of funding as being “fungible” and should be looking much more at the whole national context and environment for a given programme rather than precisely what they are spending money on.  And if they are funding things that simply can’t or won’t be procured by domestic funds, they need to be thinking about what happens if and when they pull their own funding out.


Global Fund and how to invest in impact


Some time after Aidspan first raised its concerns about the different ways the Global Fund treats national and international grant implementers, the Secretariat of the Global Fund has replied.  I don’t want to go into too much detail, so to cut a long story short the issue was that while the Global Fund allows international NGOs and agencies to report some of their overheads as “indirect costs”, national ones are expected to provide detailed reports and receipts on every single expenditure, whether it is related to programmes, procurement or management.  

The Secretariat’s response suggests there won’t be much change to this policy. This is disappointing, but not just because it seems unfair that the smaller, lower capacity organisations should have a more complicated reporting burden.  To me it is disappointing because it suggests that the Global Fund’s attitude to risk and to performance based funding has not changed a great deal.  Risk remains, essentially, the risk that money went astray or was not all spent in the way that was planned.  Having hovered around some of the discussions about the development of the Global Fund’s new model in the past year I had a small amount of hope that there would be a new approach to risk.  One that considers the risk of people not getting treatment, or of people being mistreated by programmes,

I understand why the Global Fund remains jittery about financial management.  But it really is too bad that the focus is still all about monitoring inputs.  If the Fund was truly supportive of performance based funding it would worry less about how money is spent and what it is spent on, and spend more time monitoring the impact of its investments.

The secretariat’s response also suggests to me that there will be no move toward allowing small grass roots sub-grantees to report primarily on their results either.  The risk, which I discussed in this post some time ago, is that community groups will continue to be expected to spend a lot of their time and good will bean-counting and less of it doing the work their communities need them to do. 

Counting all the things


Since it’s all the rage to talk about measurement in aid and development projects these days I thought I’d share an indicator that I just found in a national AIDS strategy:

Outcome: fewer people have sex under conditions that could impair their judgement. % of [people] who had sex when they were drunk or when their partner was drunk reduced from 4% to 2% for women and 5% to 2% for men [over 8 year period].

Some questions:

  • How do we measure if people were drunk (or their partners were drunk) reliably? What is drunk? How far do people have to think back?
  • What tells us that we know what to do to achieve the hoped for changes (4-2% for women; 5-2% for men)?
  • What tells us that we can achieve a bigger change in men than women?

I’m not saying impaired judgement isn’t a factor. I’m not saying tackling alcohol use is a bad idea for health programmes.  This critique can be cut and pasted for a bunch more indicators in the same document.

I’m just wondering why on earth we bother making promises we don’t know how to keep and measure them in ways that lack any credibility.


Relative risk; absolute risk


Howard White of 3ie discusses some of the problems we often see in how people handle data in this post, “Using the Causal Chain to Make Sense of the Numbers“.  The essay makes many excellent points which are relevant both to how programmes are designed and how they are evaluated.

However, I have to take issue with one section:

And different ways of presenting regression models can give a misleading sense of impact. A large reduction in relative risk – a ‘good odds ratio’ – can reflect quite a small change in absolute risk. Three randomised controlled trials have found circumcision reduces the risk of transmission during unprotected sex by around 50 percent. The reduction in risk was from around 3.5 percent to 1.5 percent. Just a 2 percentage point absolute reduction, so 50 men need to be circumcised to avoid one new case of HIV/AIDS.

This is a bit misleading. In assessing effects we are interested both in relative and absolute effects, yes.  But White fails to acknowledge here that the absolute risk at the outset (3.5% in the case of the pooled results of the three trials) is a characteristic of the people being researched.  And indeed the absolute risk in the populations in the three studies (Kenya, South Africa, Uganda) was different. The 3.5% and 1.5% figures come from pooling the results of the three trials.  If study subjects had come from a population where the pre-existing HIV prevalence was higher, and risk factors (including unprotected sex) were higher, then the baseline absolute risk would have been more than 3.5%.  If the risk factors had been lower, the baseline risk would have been lower. White’s estimate that 50 men need to be circumcised to avert one infection is not universally valid. In some places it will be many more; in others it will be fewer. This is one of the reasons male circumcision is primarily promoted in higher HIV prevalence settings.

Having said that, “just” a 2% absolute reduction is actually pretty good when compared to other HIV prevention interventions. Especially when you consider that once a man is circumcised, he stays circumcised, so the risk reduction is permanent. Look at it another way: if the intervention being tested led to a 100% risk reduction, then (according to White’s post), that would be “only” a 3.5% reduction in absolute terms. Still doesn’t look very impressive, does it? Except in this case there would be no new infections whatsoever.

The reason the results of these trials (and any trials) are reported as relative risks is because if you want to estimate what the effects of the intervention might be in another population, you have to apply the relative risk reduction to the absolute risk in each and every population. Reporting it any other way is misleading.  White is of course correct that absolute risk reduction is what matters when looking at the overall effect of an intervention or policy, but absolute risk reduction is a function not just of the relative risk reduction of that intervention, but of all the other relevant factors.

Rethinking Global Health


Michel Sidibe, Executive Director of UNAIDS, made an important speech on Novermber 11th, outlining his vision for a new approach to Global Health.  He is joining a growing chorus calling for the lessons of the global response to AIDS to inform an approach to health that emphasises a more people-centred approach.  Moreover he calls for an approach that convinces Ministers of Finance that health is not a cost, but an investment.  An approach that considers improving health as a question of justice: “AIDS forever changed the focus, role and participation of affected communities as essential partners in global health responses.”

The speech should be welcomed; personally though I agree that the response to AIDS can be an inspiration, we need to be clear that it has not been an unmitigated success.  Learning from the AIDS response also means learning from the mistakes.  In particular, the considerable amount of effort and money that has been poured into not only unproven, but also implausible responses.  And the massaging of egos of national and community leaders just because they talk about AIDS, irrespective of the validity of what they are saying.

What most stands out for me in Sidibe’s speech, however, is this section:

Most critically, it means completely rethinking our approaches to delivery. We can gain real advantages through community-based and task-shifting approaches, and by tapping into non-conventional capacities and alternative delivery mechanisms. We must no longer think of a community health worker as a nice person who helps out. We need to try and make them an integral part of the structures of health. 

He’s right. And yet most health systems thinking is stuck in the dark ages: a focus on doctors, nurses, clinics, drugs and policies, that continues to ignore the social and structural factors such as gender inequalities and human rights violations that can impede health.  I’ve spent some time recently discussing with colleagues how to define and more effectively support the broader version of health systems that Michel Sidibe is talking about. It’s going to take a lot more than adding the word “community” to every health systems document.  There’s work to do.

PEPFAR’s Blueprint


The US government has published its Blueprint for Creating an AIDS-free Generation: despite funding stalling, this World AIDS Day is characterised by the optimism of all the big agencies, the narrative is that the end is in sight.

I’ll get round to reading the whole thing some day, but for now I’ve turned straight to page 26, which is all about what PEPFAR is going to do with Populations at Greatest Risk of HIV. Here are a few notes.

The first thing to note is that “Populations at Greatest Risk” are subdivided into several categories: people with tuberculosis; Key Populations; people living with HIV; women and girls; orphans and vulnerable children; and young people.  Some might say the definition of “greatest risk” is rather broad, particularly given that strategies aimed at supporting women will also, of necessity, target men. While the blueprint does put welcome dollars against some specific initiatives, it does not go as far as stating how it will allocate PEPFAR’s substantial funds to these different groups.

However, the use of the term “key populations” to describe men who have sex with men, sex workers, transgender people, and people who inject drugs, will be seen by many as progress, given that up until recently the preferred term was the more stigmatising “most at risk populations”.

Advocates for key populations are likely to pay particular attention to the language on sex workers, given PEPFAR’s history of very restrictive and highly conditional support to programmes with this group.  First thing to note: PEPFAR is now firmly using the term “sex worker” rather than “prostitute”: another welcome shift in language.  There is no word on whether PEPFAR intends to develop guidance for programming with sex workers (guidance on programming with men who have sex with men and people who inject drugs is already available).  There is good language on the importance of properly involving key populations in developing and delivering programmes; however, PEPFAR still requires subgrantees to adopt a policy explicitly opposing sex work as work, a regulation which effectively rules out the involvement of sex worker led organisations.

Among the key actions PEPFAR plans to take, there are points that led me to raise my eyebrows:

7. Support civil society and faith-based work best able to address the epidemic in key populations through mechanisms such as country small grants.

Without wishing to deny the undoubtable contribution faith-based organisations have made to the response to AIDS, particularly in the area of care and support, I know of few if any examples of effective programming by faith-based organisations with key populations.  And every faith-based project with sex workers I have ever seen has involved ineffective, misguided and sometimes damaging “rehabililtation” programmes.

9. Prioritize engagement in health diplomacy to promote the health and human rights of women, girls, and LGBT populations, and advance gender equality.

The omission of sex workers, and people who use drugs, from this section on human rights, is glaring.  The blueprint does acknowledge the existence of laws and stigma against key populations, but while it emphasises that these laws and stigma affect peoples’ access to services, it stops short of recognising that these laws and stigma are at the very heart of what makes key populations vulnerable to HIV, and all manner of human rights abuses, in the first place.  It’s also worth thinking carefully about PEPFAR’s stated support for science-based approaches and human rights, given that the section on “principles” cites Cambodia’s 100% condom use programme for sex workers as an exemplary strategy. The approach raises significant concerns, in particular in relation to informed consent for HIV testing and STI treatment, and the role that law enforcement officers are given in implementing the policy.

Summary: despite some encouraging improvements (not least the very existence of the Blueprint in the first place), we may still have a long way to go before key populations in general and sex workers in particular receive the support they need from the global response to AIDS.


The Global Fund – reinventing “country ownership”


The appointment of a new Executive Director of the Global Fund to fight AIDS, TB and Malaria on November 15th was expected; and few will be surprised that Mark Dybul, former head of PEPFAR, got the job.  His name has been in the frame ever since the previous Executive Director left.

Perhaps more unexpected was the announcement that the Board of the Global Fund has terminated the employment of the Inspector General. Many activists and advocates have not really forgiven the IG for the heavy criticisms faced by the Global Fund about cases of fraudulent expenditure, uncovered by his office (and published in full on the GF website, as per policy).  The Global Fund has made it clear that the Office of the Inspector General will remain, and has initiated the process to find a new Inspector General.

As the Global Fund begins a new phase – new leadership, second decade, new funding model, stronger grant management – it will be interesting to see how it pursues the task of monitoring and auditing grants.  Certainly, the restructuring of the Fund over the last few months, which has shifted the bulk of staffing into grant management and supervision roles, indicates the intention of paying more attention to what gets funded and how.  While this is welcome, however, I hope that actors in recipient countries will also take greater control and ownership.  Because if you take a look at any one of the many reports published by the OIG, you’ll notice something a little bit disturbing.  Take the Ethiopia report.  Here is one of the 86 recommendations:

Recommendation 1 (Significant priority)

To improve the quality of malaria and tuberculosis diagnosis, the FMOH should ensure:

(a) Microscopes are maintained in proper functioning order.

(b) Running water is available in the health facilities.

(c) Refresher training is provided to laboratory technicians based on a needs assessment.

(d) Internal and external quality assurance of the health facility laboratories is conducted as planned and is duly documented.

And here’s another:

Recommendation 75 (High priority)

The PFSA should ensure:

(a) Appropriate racking and other storage media are provided to the facilities in order to facilitate proper storage of health products. In particular, direct storage on the floor as observed in some facilities may lead to products being damaged by moisture hence pallets should be used. Correct handling equipment reduces the risk of injury to workers and damage to stock. Stock organization on shelves and pallets enhances accessibility which facilitates use of first in first out (FIFO) and first expiry first out (FEFO) issuing systems.

(b) Temperature monitoring in storage areas is introduced in order to maintain optimum storage conditions.

So, make sure that broken microscopes get fixed, make sure there is running water, and have proper storage in health facilities.  I don’t think the OIG was wrong to include these observations in his report. In fact, if I was conducting an audit and saw these problems, I’d write them up too.  Had local supervisors or activists already documented these problems in Ethiopia?  If so, how long would it have taken to fix them?  Did those responsible even have the resources to fix them? I have no idea, but I am pretty sure that if the success of Global Fund grants depends on a crack team of highly paid international auditors ferretting out problems like this – when the Fund supports services in tens of thousands of health facilities and community organisations – that there is something wrong with the model.  

The Global Fund does, of course, provide some support to country-level monitoring and accountability – through the Country Coordinating Mechanisms.  But when you compare the oversight tools used by the CCMs to the sorts of data digested by Global Fund portfolio managers and the OIG, there is no comparison – Fund officials have far more information, and far more resources to digest it, than the CCMs.  Moreover, the data that CCMs are encouraged to monitor is somewhat abstract.  By and large, people that sit on CCMs representing marginalised and underprivileged populations are not trained to interpret spreadsheets and performance ratings.

I’d like to see much more support going to community level monitoring of health services.  It does happen, but not nearly enough.  And there is evidence that it has a positive impact on how services are provided, and on health outcomes.  A fantastic example of where it does happen is in Cameroon, where the organisation Positive Generation supports people with HIV and TB to monitor things like stock outs, provider behaviour, and overpricing of services in 40 treatment facilities in the country.

People who get shoddy services should be able to find out why (i.e., because the microscopes are broken, or the water isn’t running, or the drugs have gone bad); as should people who are refused services or who are subject to human rights violations.  And should be able to raise hell and demand changes.  They need the support of civil society organisations, human rights organisations, UN agencies, and the Global Fund to do this.  I’m hoping that as well as being more diligent in its international grant management, the new Global Fund can find ways of generating greater ownership of the problems in the countries it is funding: not just ownership by governments, but ownership by the people who the Fund was set up to help.