More tilting at lamp posts...

Thanks to a heads up from Kyle McKay I see that the MoJ have now published an update on the Payment by Results pilots in Doncaster and Peterborough. You can access it here. (My previous blog posts can be accessed here.)

Proving the PbR pilots have worked is still a long way off, it would seem to me: the MoJ concedes they do need full 12 month post release data in order to do a full blown comparison ("final results will not be available until 2014"). However, this does not prevent them from interpreting the results (in my view) creatively to show how PbR is working in these two areas.

However, I would point out the following:

• They say the "interim re-conviction figures being published in this statistical bulletin are based on periods half the length of those that will be used for the final results".  I say this is not just half this is the first half of a 12 month period. Second halves are also a little harder...
• How typical are Doncaster and Peterborough compared to the rest of the country? All the comparisons made are with national data. Given the news about Doncaster in recent months & years, it is hardly a standard place. Also Peterbrough is a 'new town' and (according to Wikipedia) "Peterborough's population grew by 45.4% between 1971 and 1991" which I think makes it a somewhat unusual place. So are national comparisons really valid?
• They say that "Both PbR prison pilots use a 12 month re-conviction measure which differs from the National Statistics proven re-offending measure. The key difference is that re-convictions only count offences for which the offender was convicted at court, whereas the National Statistics proven re-offending measure also includes out of court disposals (cautions)" and "Additionally, there are a number of other differences between the pilots and the National Statistics proven re-offending measure in terms of which offenders are counted within the cohort". That all seems pretty important to me... does it to you?
• Indeed the whole document seems peppered with so many caveats, footnotes and explanations as to make me wonder just what we are being told.
• They say "Success of the Peterborough pilot will be measured against a control group of similar offenders released from other prisons, with the target met if the frequency of re-conviction events is 10 per cent lower for the Peterborough cohort than for the control group. It is not possible to replicate that comparison for these interim figures". I say: why not? Why is the 'control group' not being monitored in a similar way? It does not make it much of a control group..!
• They say "The national comparisons included with the previous interim figures published on 13 June 2013 included all prisons, not just local prisons. However, because Peterborough is a local prison, using national figures for other local prisons provides a better comparison". Huh? Why were the local prison data not used before? Are they just using whatever data seems to give the 'best' result?
• It would appear that frequency of conviction rates in Doncaster have been coming down since September 2007 whereas nationally during the same period, national figures have been showing a rise since September 2007. The Peteborough pilot began in October 2010. At the very least this shows that national comparisons are dodgy since the trends were going in opposite ways before all the PbR pilots began. It also potentially shows that other significant forces are present in Peterborough that could be creating the positive trends other than the PbR pilots... 
• Similarly the Donaster reconviction rates have been in a downward trend since October 2007, the data appears to suggest: again well before the pilots were begun. 
• (As an aside: the national data shows some very worrying trends: reconviction events per 100 offenders has gone from 66 to 84 between June 2007 and 2012. That is a rise of  over 27%. That is a bit concerning isn't it?)
• But for me the biggest problem with this whole comparison approach is that a potentially huge Hawthorne Effect is not being controlled for. In other words, the mere presence and attention being given to the PbR pilots is what is creating any positive effect, not the pilots themselves. This is not, I repeat NOT, being controlled for. For me this calls into question the whole edifice on which these pilots are based.

Anyone with an ounce of independent thought will understand that, at the very least, these results are not the basis which to build a whole reform of the offender management system. The comparisons are shaky and riven with cautions.

It is time to develop a better experimental framework.