06 April 2007
The Comprehensive Performance Assessment regime is 'self-defeating' and 'off-target', research from the University of Oxford has found.
The report, The perils and pitfalls of performance management, describes the Audit Commission's reliance on council self-assessment as 'disturbing', and finds that a number of indicators are open to manipulation.
Researchers Iain McLean, Dirk Haubrich and Roxana Gutiérrez-Romero found that CPA indicators could encourage so-called 'output distortions' – or cheating.
Faced with the choice between closing a school during bad weather or scoring badly against an indicator for school attendance, education directors have an incentive to close the school, Haubrich told Public Finance, even though that means a deterioration in service for the public.
Indicators for the use of leisure centres could also be boosted by closing facilities and so concentrate use in one place, the report found.
The report was published in the April edition of Public Money & Management and came as the commission launched its consultation on a new inspection regime, which could extend the use of self-assessment. But the researchers said that its use so far in the CPA had been 'consistently off-target'.
They compared the 'ability to improve' scores from the 2002 CPA scores to actual reported improvement in 2003/04 and found no correlation.
'In other words, “ability to improve” scores are a very bad indicator for actual improvement,' the report states. The authors attribute this to 'gaming', by councils seeing their 'ability to improve' self-assessments as 'cheap talk', which could boost their overall CPA scores.
Gareth Davies, managing director for local government at the Audit Commission, dismissed the research as 'simplistic, out of date and generally [an] inaccurate view of CPA and performance management in local authorities'.
He said: 'The CPA is widely acknowledged as a key driver of improvement in local services and we are confident that CPA results reflect the quality of services provided by local authorities, not their ability to “play the game”.'
While he conceded there was no correlation between 'ability to improve' scores and actual improvement, he said that element of the CPA was 'never intended to be a predictor of future performance'.
The report also presents evidence of contradictory incentives. While councils could enhance their CPA scores by improving pupil attainment, the same indicators were also used to signal deprivation. Low attainment levels could thus lead to extra funding linked to deprivation.
'Authorities essentially have to choose between focusing on attainment or ignoring it and getting the extra funding… So they are offered a carrot and stick, but if the education director realises they can't do it, at least they get more money, so you take away the bite of the incentive,' said Haubrich.
The report further criticises the ease with which councils could move up a category through 'quick win' activities, such as simply publishing a particular strategy document.
'Improving the ranking therefore becomes possible by simple, sometimes one-off administrative adjustments, without improving the underlying performance,' it states.
PFapr2007
