Monitoring the treatment rollout

The new National Strategic Plan on HIV, STIs and TB, 2012-2016 (NSP), like its predecessor, has a substantial M&E section. But despite its ambitious targets, there was poor M&E following the last NSP. At the time, the Department of Health lacked the will and staff to get M&E right.

The National Strategic Plan and Monitoring & Evaluation (M&E)

The new National Strategic Plan on HIV, STIs and TB, 2012-2016 (NSP), like its predecessor, has a substantial M&E section. But despite its ambitious targets, there was poor M&E following the last NSP. At the time, the Department of Health lacked the will and staff to get M&E right.

Hopefully the new NSP will fare better. M&E in the new NSP is better conceived than the old one. There is a more manageable set of indicators, a few of which have plausible baseline data and describe feasible methods for measuring success during and at the end of the NSP period.

The indicators are (numbers rounded to avoid false accuracy):

• HIV prevalence in 15 to 24 year olds, estimated at baseline to be 9% (HSRC Household Survey), with the target of reducing this to 4% in 2016.
• HIV prevalence in key populations: there is no baseline data for this in the NSP, even though a 50% reduction is proposed.
• HIV incidence, estimated to be 1% in 2012 (ASSA), with the aim of reducing this by half by 2016.
• HIV mortality, estimated to be over 40% of deaths at baseline (Stats SA) with the aim of reducing it by half by 2016.
• TB incidence, estimated to be almost 1% in 2010 (WHO), with the aim of reducing it by half by 2016.
• TB mortality in HIV-negative people, estimated to be 50 / 100,000 in 2010 (WHO) with the aim of reducing it to 25 / 100,000 in 2016.
• Rate of transmission to infants, estimated to be just under 4% at six weeks after birth in a study published last year by the MRC, with the aim of reducing this to less than 2% at six weeks and less than 5% at 18 months in 2016.
• Patients alive and on ART, for which no data is currently available, but by 2016 the target is to have 94% retention in care by one year after commencing ART, 88% by two years, 82% by three years, 76% by four years and 70% by five years.

Some of these statistics should be treated with caution. The drafters of the NSP recognised this and tasked the SANAC secretariat with leading a process to calculate better baseline data. One indicator not mentioned above is called a stigma index. This is poorly defined and no way of measuring it is described in the NSP. Nevertheless, most of the indicators are reasonable and can be used as the basis for a good M&E programme. The NSP also commits to annual programme reviews, a mid-term evaluation in 2014 and a final evaluation in 2016 that will also provide the baseline values for the next NSP. This is ambitious. If it is to happen, there must be enough people to do this work and clear leadership.

Overall responsibility for M&E has been given to the M&E Unit in the SANAC secretariat. The plan also says that M&E units in provincial AIDS councils and sectors should co-ordinate M&E provincially. Exactly how this will all work is unclear as is the role of Health Department, and who leads the process. But wherever the M&E programme is located, it must be given a substantial budget and quality human resources. The Director General of Health must take ultimate responsibility for making sure M&E is implemented.

The remainder of this article deals with monitoring the treatment rollout.

Getting it right for treatment

How many people started antiretroviral treatment in public clinics and hospitals in South Africa? How many are still in treatment? How many have died? What regimens are people taking? How many have moved to second-line regimens? At what CD4 count does the average person start treatment? How many have suppressed viral load at one year? What’s the average time that people have been on treatment?

These are questions a well-monitored antiretroviral rollout should be able to answer. There should be reports every six months with all this data and more. Unfortunately the monitoring of South Africa’s antiretroviral rollout is not yet able to provide this information consistently and accurately, although there are positive signs that data collection and processing is improving.

Occasionally the Department of Health releases a spreadsheet with numbers of people on treatment. The latest data is for June 2011. There were, according to the Department of Health, over 1.5 million people on treatment in the South African public health system by end of June.

EC	FS	GP	KZN	LP	MP	NW	NC	WC	Total
151,878	79,204	375,798	481,809	109,121	121,516	130,901	17,407	101,812	1,569,446

There are problems with this data:

• The methodology used to collect it is unpublished. Therefore it is difficult to have confidence that the systems used to collect this data are in place. Most facilities get little support on how to track numbers and there is no standardised auditing system. Are health workers consistently recording patient visits, viral loads and deaths? Probably not.
• Some provinces report number of people started treatment and others report number of people on treatment.
• Even if these data were accurate and consistent, they help answer only one of the above questions.

One of the best sources of information on South Africa’s rollout is the International epidemilogic Databases to Evaluate AIDS Southern Africa (IeDEA). This is a pooled database from multiple sites in Southern Africa, including 10 projects in South Africa. This group regularly publishes useful information about the rollout. It can reliably report number of patients, loss-to-follow-up and cohort mortality. But although it is a large cohort, it is still only a fraction of the total antiretroviral patient population in the country.

The Western Cape, assisted by UCT’s Centre for Infectious Disease Epidemiology (CIDER), had the best provincial data collection system. The system allowed the provincial health department to produce a regular detailed report that helped answer many of the above questions. Although the system started off paper-based it is slowly converting to an electronic system that emulates the paper-based one.

Two factors have made the Western Cape system unwieldy in recent years. The province has been trying to integrate the HIV treatment system into its general data systems –a complex task– and it has not invested in enough dedicated staff to keep the system going properly. If the number of people on ART were stable neither of these factors would likely be a problem. But these coupled with the swift scale-up have set the province’s treatment data collection system back.

In addition to the Western Cape, several donor-funded projects operating in the public health system have systems to monitor their subsets of the antiretroviral rollout. An audit showed there were at least 40 such systems, about 15 of which were properly functional. Most of these projects are funded by PEPFAR, which requires detailed treatment statistics to be reported to the United States government.

The National Department has recognised that there are too many systems and that what is needed is one properly working consolidated one. It is encouraging all the various projects to merge their data collection into one national system. For this it has largely adopted the Western Cape system.

This national system recognises that there are three tiers of data collection that need to work together:

• Paper based systems at small clinics.
• Stand-alone electronic data capturing systems at medium sized clinics.
• Networked Electronic Medical Record systems at large facilities.

Clinton Health Foundation officials have been contracted to assist with the rollout of the system.

The government’s move towards a unified, rationally designed treatment monitoring system looks promising after many years of rudderless monitoring and evaluation. There is progress in some provinces, but getting M&E right will require a Herculean effort by government and co-operation from donor funded projects. The response of donor funder projects is mixed. Some deserve praise for the way they are trying to merge their systems into the national one. However, others could do a lot better.

Technological improvements that will help

Several technological improvements and better management of technology will improve M&E systems. The issues discussed here are not exhaustive.

Networked computing is too expensive

Moving from paper to computer is important: computerised algorithms are much more accurate and faster at calculating the number of patients lost-to-follow-up than humans sifting through mounds of paper. When clinics computerise they usually find they have underestimated loss-to-follow-up. One of the barriers to moving to networked computers is the State Information Technology Agency (SITA). Network access must be purchased from SITA. It charges in the region of R20,000 per month for a Diginet line. Few clinics can afford this or need such a sophisticated solution. Moreover it typically takes months to have this installed. Yet a simple wireless connection to the Internet can be setup within days at approximately R500 per month. While it is understandable that a central state agency is responsible for this kind of infrastructure in order to prevent abuse, the exorbitantly priced solution SITA is enforcing is impractical and wasteful.

Free cell phone access to download viral load, CD4 and other results

Many health workers have cell phones with internet access. It should not be too technically challenging to build a system that allows them to download their patients’ blood results from the NHLS (obviously with suitable security). It should also not be difficult to make this downloading free for the nurses, i.e. if the cell phone companies redirected bandwidth charges for certain sites to a health department instead of to the health worker’s account.

Integration of information

South Africa has some very useful health data systems. Home Affairs tracks deaths. The NHLS has a database of viral load and CD4 results. Many pharmacies have implemented sophisticated dispensing systems. It would be ideal if these systems synchronised relevant data with the antiretroviral Electronic Medical Record system. So if a patient died and Home Affairs processed a death certificate, the medical system would be automatically updated; a loss-to-follow-up could be correctly recorded then as a death. Viral loads and CD4 counts could be automatically saved onto patient records. The patient record system could automatically send prescription requests (or vice versa) to pharmacy systems. This would be a complex undertaking but if our treatment monitoring system could get to this level of sophistication it would greatly improve the quality of information on the ART rollout.

Protection of equipment

A key challenge is maintaining computers in peripheral clinics which do not have IT departments. Thefts and breakdowns can take weeks or months to rectify.

It is vital that the National Department of Health continue its effort to get the monitoring and evaluation systems for the ART programme working properly. To do this, donor funded programmes and provincial governments must co-operate and work together, with the aim of creating one national integrated system that records all useful implementation data and reports it regularly.

Thank you to Andrew Boulle for assistance. TAC takes full responsibility for the views in this article or any errors it contains.