Troubled Families: the Moneyball moment for payment-by-results?

25 Oct 16

The government’s Troubled Families programme shows how payment by result contracts can go wrong. But it also shows where they can be improved if public bodies become more specific on what they are trying to achieve

The 2003 book Moneyball is the real life story of how the Oakland Athletics baseball team general manager, Billy Beane, and his assistant Paul DePodesta, turned the sport on its head by using a new analytical, data driven approach to building their team. I recently watched the 2011 film, starring Brad Pitt and Jonah Hill, based on the book and there is one line that’s stuck with me. Hill’s Peter Brand (a composite character of Beane's assistants) is explaining why “baseball is in the dark ages” and how teams’ don’t understand what they’re actually buying when they sign a player. “When they look at a player they see athleticism and a great swing, when I look at him I see a set of statistics. The purpose isn’t to buy players, it’s to buy wins.”

The central message is figure out what you really want to achieve. Last week’s Troubled Families row is a case in point and in particular in relation to ‘payment-by-results’ contracts. Payment-by-results (PbR) in principle makes sense, government wants an outcome not a process so will notionally pay on receipt of the outcome. But relying on the provider to collect the data has repeatedly proven problematic. The incentive will always be to make the result fit the contract. At its worst PbR pays almost no attention to results, and becomes by ‘payment-by-activity’. The head of the National Audit Office, Amyas Morse, has previously pointed out the dangers for commissioners in “using this mechanism in circumstances to which it is ill-suited, to the detriment of value for money.” This is certainly the line of attack being used against the Troubled Families programme.

How you judge what constitutes an effective outcome when dealing with complex social problems, let alone contract for it, is one of the most important legacy questions created by this programme. At the Public Accounts Select Committee hearing last week Department for Communities & Local Government permanent secretary Melanie Dawes acknowledged that PbR put “too much focus on the short term”. Her colleague and programme lead Joe Turke similarly suggested that it may not be the “best way” to incentivise what the department wanted. But with the Treasury demanding value for money, and rightly so, it’s difficult to see what alternative approach could have been deployed at the pace required by our political masters.

What’s clear is that PbR is a damaging route to failure if you’re not completely clear on what win you’re trying to buy. Its emphasis on empirical evidence exposes weakness in data. The central criticism of the Troubled Families programme has been the evidence doesn’t show a “significant impact”, either because no such impact was created or we haven’t found a way of collecting and measuring it correctly. One person I spoke to responsible for running a Troubled Families programme in their local authority area said getting access to other agencies data was “painfully slow”.

The issues being dealt with were also far more entrenched than initially thought. If I had a serious criticism of DCLG and the Troubled Families programme it would be that they weren’t specific enough about what they were trying to achieve. In deploying a contractual mechanism that is sensitive to provider context, but blind to nuance in outcomes, for a national programme where the criteria for success was so broad, Troubled Families was always going to be more experiment than programme. In this regard we’ve learnt some valuable lessons about using PbR at scale.

To use PbR effectively commissioners need to be sure they’ve had a Moneyball moment. They need to get past the initial activity and focus on the precise outcome they’re trying to buy, the win. With something as complex and broad as the Troubled Families programme, however, the difficulty is there’s no single definition of a win.

Did you enjoy this article?