Walter Mitty & secret beauty

Reviews of The Secret Life of Walter Mitty have been mixed. As it happens, I think it is one Ben Stiller's finest movies (he directed it too). As it is (loosely) based on the Thurber short story, it is fair to say that the narrative is pretty uncomplicated. Although the film contains some troubling contradictions (would a highly skilled skateboarder become a negative images manager?), it ends up delivering a thumping good message interspersed with comedic magic (the helicopter pilot) and some quiet moments of Zen (the leopard and the football game).

Go see it. I think you will find it warming and poignant. It should make you laugh and smile lots too.

As to the leadership lesson, it would be too easy to focus on the 'Change Managing Director' who looks like a cross between the heavy baddie in Superman II and Alvin. His style of leadership is so bad as to be chillingly realistic, I worry.

But instead I would like to focus on beauty and the power of what we see that can take our breath away and make us think: what is really going on here? Without giving too much away, the film treats us to some sumptuous cinematography of places that are not widely known for their beauty. In this way, the film cleverly makes us think about how we should always go beyond the obvious caricatures of places (or people) and look instead for their secret lives too.

So in this respect, I want to suggest that good leaders never take anything at 'face value'. Good leaders seek to uncover the beauty beneath and what the real story is all about. Good leadership is about digging, exploring, revealing and finding the beauty that is always there to be found.

What beautiful thing have you noticed today?

_____________

This is the second of my new series of blogs about leadership ideas to be found in the movies of our time. You can read here as why I am doing this. Please subscribe to this blog if you want to read more. Thanks.

The redemption of Mr Banks (and Mr Disney)

I am not sure when exactly I realised Mary Poppins is all about fatherhood. It was sometime since I saw and loved it as a child and long way before Saving Mr Banks came along. It suddenly hit me like a forceful palm to my forehead. It was all about being redeemed as a father, with dear Mary Poppins being the gentlest and sunniest of facilitators.

And since I realised this, I now only ever see the film backwards. In other words, everything is building up to that moment at the end when a kite is made and flown. Saving Mr Banks mirrors this, since of course we know the end from the beginning: Mary Poppins is made and becomes one of the world's most charming and adored films of all time. So I watched for clues as to how this 'kite' is eventually made since so much of the time in the film, it is lying in pieces. Mrs P L Travers seems robustly resistant to letting Mary Poppins fly.

And in the end, not only is Mr Banks saved but so is Walt Disney. I won't reveal how: but redemption is all around in the film.

Which got me wondering, how many leaders can truly redeem both themselves and others too. 

Redemption may be defined as "the action of saving or being saved from sin, error, or evil". In part, this is what leaders should be doing all all the time: taking action to save their people, their organisation and their business from error if not evil.

But how do you become a 'redemptive leader' like Mary Poppins? It cannot just be about magical chalk drawings or being 'practically perfect in every way'. (Although they can help...) Instead I think it boils down to three things:

• Humble non-attachment: the recognition that what is achieved is due to collective effort and is never down to just the one person
• Bright determination: the commitment to making a difference but in a way that is wholesome, upbeat and courteous
• Knowing when to go: ultimately we all redeem ourselves and so while leaders can help this happen, these leaders must also step aside to let the full process occur

What else does it take to be a redemptive leader do you think?

_____________

This is the first of my new series of blogs about leadership ideas to be found in the movies of our time. You can read here as why I am doing this. Please subscribe to this blog if you want to read more. Thanks.

Leadership actually

I was very fortunate to receive a wonderful Christmas present from my wife: an unlimited movie card for the Cineworld cinema in Milton Keynes.

Over the next twelve months I can now go an see as many movies as I wish, and as many times each as I wish to see them! Fantastic! 

Regular readers of this blog will know that I have written several times before about the leadership messages contained within movies. So now you can expect many more of these blogs since I have my unlimited card! Love Actually remains a favourite film of mine (even despite this rather funny and rather scathing review) and contains the line given the PM (Hugh Grant):

Whenever I get gloomy with the state of the world, I think about the arrivals gate at Heathrow Airport. General opinion's starting to make out that we live in a world of hatred and greed, but I don't see that. It seems to me that love is everywhere. Often, it's not particularly dignified or newsworthy, but it's always there - fathers and sons, mothers and daughters, husbands and wives, boyfriends, girlfriends, old friends. When the planes hit the Twin Towers, as far as I know, none of the phone calls from the people on board were messages of hate or revenge - they were all messages of love. If you look for it, I've got a sneaky feeling you'll find that love actually is all around.

I go along with that. But I also think that leadership actually is all around too. Just as government policies come and go, business strategies become popular then fade, companies are turned around or fail or departmental heads just want things to improve (etc)... the common ingredient in all of these is leadership. Nothing much happens in business and organisational change without good leadership. And a heck a lot of things go wrong with poor leadership.

So, I will be dedicating the many film based blogs over the next year to uncovering the good, the bad and the ugly leadership in the films I go to see. I look forward to debating these cinematic takes on leadership with you.

What was the last film you saw and what leadership message did it contain for you?

Seasonal greetings!

Dear reader, here is my Christmas Card to you:

My word and theme for next year is 'Being' (This blog post explains why)

May 2014 be all that you wish and hope for. In this complicated, busy and beautiful world, may you find peace and joy this Christmas. And long may it continue.

Jon

My visit to Number 10

Yesterday I had my first (and probably only ever) opportunity to walk through the doors of 10 Downing Street.

I was there for a meeting of the SME Panel. I have blogged about this before but in summary I have been part of smallish group of SME business people over two and half years advising the Cabinet Office on how to make government procurement more 'SME friendly'.

The essence is this: SMEs can provide far better value services and products to government based on lower overheads, more innovation and better quality. This can lead to some staggering reductions in cost to the taxpayer as well as helping the British economy to grow. It is the proverbial 'no brainer'.

However (and this is a rather large however...) many (most?) government procurement departments at all levels and in many agencies seemed to be wedded to processes that favour larger organisations. And the only winners are these large companies: not the taxpayer, not the growing economy of smaller businesses and not the citizens (as the beneficiaries of these services). Frankly it is a huge scandal and it has been my pleasure to have been a small part in this government's attempt to turn the procurement tanker around in the English channel, as it were.

Anyway yesterday, we were graced by the presence of Lord Young and Nick Hurd MP, Minister for Civil Society. They both listened carefully to all the points made by the panel members present. We also had some useful information given to us by Stephen Allot, the Crown Representative for SMEs.

Here are some of the points that were made:

• We were introduced to the "Small Business / GREAT Ambition" strategy of the government.
• The measures in this include abolishing PQQs for low value tenders (less than €200k), putting all public tenders onto a single site (Contracts Finder), a trip-advisor type feedback mechanism for purchasers to rate suppliers and vice versa... and more. 
• The G-Cloud level playing field has shown just what SMEs can do: "as of the end of October 2013, 56% of of total public sector spend by value through the G-Cloud framework had gone to SME suppliers" (from Stephen Allot's blog)
• One of the suppliers around the table declared that they had just won a contract with a bid of £6m. A large well known IT supplier had also submitted a bid (and lost) of £100m. This points to the vast improvements in taxpayer value that can be achieved by procuring from SMEs
• Many parts of government are pursuing this strategy with verve and alacrity while other parts (sometimes even within the same Department) are still living in the 'dark ages'... paying out to over priced contracts because the procurement processes were not SME friendly
• A system will be introduced to shame government purchasers & large primes who fail to pay their suppliers on time

Nick Hurd challenged the group to come up with a list of actions that can be taken before the next election to institute irreversible change in how government procures, beyond what is already planned. That will be the subject of a future blog...

Being: my word for 2014

Every year, I adopt a word from I CAN, the children's communication charity.

This charity helps children who struggle with communicating to get the support they need. My work and activism depends on my being able to communicate well. So this charity has a special importance for me. And it helps children too. Why wouldn't I shell out £15 to adopt a word...?

Last year I adopted "Childlike" (you can read why here). And in previous years the words have been "Perspective",  "Prosperity" and "Abundant".

This is my word for 2014:

I have ordered an apron too!

But you may be wondering why "Being"....

This has been a difficult year for me: I have been to more funerals this year than ever before. And this isn't just because I am getting older. Many of my friends, family and neighbours who died this year did so under unexpected and very tragic circumstances. I am still grieving. I have no intention of going into details but this year (if it has taught be anything) has made me appreciate the present, the now, the being here now...

Sometimes at this time of year, we risk becoming very acquisitive and thinking about what we have rather than who we are. In all the eulogies I have heard this year, I have heard only talk of the people they were. When we die, we are remembered for who we were not what material possessions we accrued. 

And so by picking this word, I am resolved to being here now, to worry less about what I have and focus on being in the present. To savour each day, each moment... not like it could be my last (even though it could be) but just because it is a moment, a day that will never happen again...

And with this blog post, please accept my seasonal greetings. May I wish all my readers an amazing 2014 which I hope is full of good moments for you and your loved ones.

Leadership priorities for 2014

In issue 378 of Police Professional, there is an article entitled A Challenging Future which features some very useful unpacking of the presentation given by CC Mike Cunningham (Staffordshire Police) to the Excellence in Policing conference, on what kind of police leadership is needed for the future. I recommend the article to you as his slides (now uploaded) only present a smidgeon of what the article reports him as saying.

To summarise, CC Cunningham asked more than 200 personnel at Staffs Police what they thought were the key leadership issues for the future. This helped him identify four priorities:

• Ensure lines of communication with staff and officers are open
• Welcome challenge and understand how the service can respond
• Develop early intervention strategies, spotting problems as they emerge
• Spend time discovering the fears and concerns of staff and officers: a step change in staff/officer engagement

I like this. I always like lists of priorities when they number under ten and I like lists of four even more! And I think these four are useful.

But the question is, as we turn towards the end of this year... what do you think should be the leadership priorities for police (and other public service) leaders in 2014? 

I have posted blogs like this one in the past (see this link for the 2013 list, and these for 2010 and 2012). So please post below, tweet or email if you wish

So to repeat the question:

What do you think should be the leadership priorities for police (and other public service) leaders in 2014?

UPDATE 031213|2312: Thanks to Suzanne Thompson ‏@Suzze05 for these five priorities

1. Open & transparent leadership - visible & approachable
2. Be intuitive & responsive to the 'informal work culture'
3. Be open to genuine innovation & collaboration opportunities to improve practice & add value.
4. Welcome positive change but do not be afraid to openly challenge change based solely on ideology & not evidence based.
5. Listen..no really, really listen, not just pay lip service & go through the motions as staff know the difference

UPDATE 041213|0920: Thanks to Roger Nield ‏@rogernield2703 for these priorities

• Listen to what the public want.
• Do something about it - not just do what's easy or what interests your organisation.
• Tell them what you've done.
• Do this without arrogance, fear or favour and without thought of reward.
• You can boil this down to "Do Good"

UPDATE 081213|1628: Thanks to Annette Hill ‏@familyhrguru for these priorities

UPDATE 121213|2108: Thanks to Sukhvinder Stubbs ‏@Sukhvinder2011 for this observation

What have the Romans ever done for us?

I am just about to zoom off to Aylesbury Vale's annual business consultation meeting at a hotel on the edge of Buckingham. I am optimistic that this won't be like last year when we were talked at for over 80 minutes. Having met with the Deputy Chief Executive the other day, it sounds like it will be much more two way. (I will let you know...)

Meanwhile I have pondering on what AVDC (and perhaps other local councils) can do to help local businesses thrive. This is in all our interests! Here is the list that I have got to so far:

• Buy from us! In other words where the council is purchasing materials or services, go local first!
• Set up your procurement so that it is SME friendly (there is plenty of guidance on that available from the Cabinet Office which, as it happens, I have had a small hand in creating)
• Make business rates transparent: let everyone know what money is collected and how much of it ends up in local council hands
• Use some of the Neighbourhood Homes Bonus and Community Infrastructure Levy to support local economic development
• Make the council 'business friendly' which includes paying great attention to infrastructure development, supporting safe practices at work and promoting the eating establishments that are top notch when it comes to hygiene (see what is happening in Wales) among many other measures
• Make it easy for businesses to hire apprentices and offer work training through connecting people together
• Don't try to compete with local businesses: let your income generation innovation have only a benign impact on the local economy!
• Use local business innovative and entrepreneurial spirit & insights to support the transformation of the council to a leaner, fitter and more effective local public service: we can help!

What else is on your list?

The presentation of stage two transfers

Whilst this sounds like something organised by the Football Association, many will now know this is a process embedded in the new arrangements for police service governance. The APCC produced a helpful summary of some of the issues involved a while back which includes this paragraph:

The Police Reform and Social Responsibility Act (the Act) which creates PCCs also sets out a second  Stage 2’ transfer which refers to the subsequent movement of certain staff, property, rights and  liabilities from the PCC to the chief constable. The stage 2 transfer is designed to allow elected PCCs the freedom to make their own local arrangements about how their functions and those of the police force will be discharged in future. In establishing these local arrangements PCCs will wish to clarify with their chief constable who will employ which staff, hold which properties, liabilities and assets etc.

There is more information here as well. All PCCs and Chief Constables should by now have submitted their agreed plans to the Home Secretary. She will now review them.

I imagine the information will emerge soon as what the pattern is across England & Wales. What I am hoping for is to see it represented in this way:

	Contract of employment with PCC	Contract of employment with CC
Directed/managed by PCC	??	??
Directed/managed by CC	??	??
Jointly directed/managed by PCC & CC	??	??
Directed/managed by CC but with specified & agreed influence by PCC	??	??
Directed/managed by PCC but with specified & agreed influence by CC	??	??

This will not only provide useful comparisons across the country but also, I believe, clarity for the staff involved.

What do you think?

Eiji Toyoda, targets & PCCs

It might have escaped the notice of many PCCs (though not all I hope) that Eiji Toyoda died this week, aged 100. Reputedly, this is the man that led Toyota from producing as many cars a year as Ford were producing a day to the global business that it now occupies. Mr Toyoda has also been a beacon light in the quality and performance movement that has changed manufacturing processes across the world - the so called Toyota Way.

The first important thing to notice about the Toyota Way is that it is a philosophy of doing business, not a set of tools that some people mistake it for. You can read about it in any number of web sources and books: just use your search engine. But in summary there are 14 principles:

1. Base your management decisions on a long-term philosophy, even at the expense of short-term financial goals.
2. Create a continuous process flow to bring problems to the surface.
3. Use “pull” systems to avoid overproduction.
4. Level out the workload (heijunka). (Work like the tortoise, not the hare.)
5. Build a culture of stopping to fix problems, to get quality right the first time.
6. Standardized tasks and processes are the foundation for continuous improvement and employee empowerment.
7. Use visual control so no problems are hidden.
8. Use only reliable, thoroughly tested technology that serves your people and processes.
9. Grow leaders who thoroughly understand the work, live the philosophy, and teach it to others.
10. Develop exceptional people and teams who follow your company’s philosophy.
11. Respect your extended network of partners and suppliers by challenging them and helping them improve.
12. Go and see for yourself to thoroughly understand the situation (genchi genbutsu).
13. Make decisions slowly by consensus, thoroughly considering all options; implement decisions rapidly (nemawashi).
14. Become a learning organization through relentless reflection (hansei) and continuous improvement (kaizen).

One of the key elements of this philosophy is the use of targets. (Perhaps, you can see where I am going now: Police forces facing dozens of new performance targets BBC news) However, targets are a highly contentious issue in the 'right first time' approach to performance improvement. W. Edwards Deming, one of the gurus of the quality movement, consistently argued that companies should 'eliminate management by objectives' and 'eliminate fear' in his 14 points.

(Aside: why is 14 so special?)

The Toyota Production System certainly talks of targets (see, for example: "Kaizen also requires the setting of clear objectives and targets. It is very much a matter of positive attitude, with the focus on what should be done rather than what can be done" from this Toyota source.) But my understanding is this is mainly about work teams understanding what needs to be done and setting themselves goals, assessing their progress relative to these goals.

But please watch this: John Seddon (a follower of the Taiichi Ohno and the Toyota Production System & proponent of systems thinking) talking about "Cultural change is free: why targets make organisations worse" (09:50 in).

It seems clear to me that top down targets do not work. (I published my first article on the net about this back in 2003, the piece is still up there. I have blogged about this several times since as well.

And yet here we have being reported today, the fact that some (not all) PCCs find the lure of top down target setting irresistible: it seems to them to be the logical thing to do.

Despite the Home Secretary's protestations: what did she honestly expect? Create a system of political governance that focuses all the power on a single individual, emphasise the importance of local accountability, set up an electoral process that encourages a manifesto of pledges to be made, require the elected PCCs to produce a local Police & Crime Plan... and hey presto targets are set! Yes the Home Sec can proudly say that she no longer sets targets... but she has created a system which encourages their setting and deployment locally.

But what intrigues me most is not the fact that there are PCCs who have set targets but the ones who have not... There may be an element of politics here, as my colleague Bernard Rix has highlighted on his blog. It may also be that these PCCs are exhibiting a different kind leadership, and perhaps know a little bit about the stuff that Toyoda, Ohno & Seddon have discussed. Here is the list (from the BBC research) of the PCCs who have set targets:

• Avon and Somerset - 4
• Cambridgeshire - 12
• Cumbria - 20
• Devon and Cornwall - 4
• Hampshire - 5
• Hertfordshire - 14
• Kent - 5
• Leicestershire - 26
• Norfolk - 9
• Northamptonshire - 1
• Northumbria - 8
• Nottinghamshire - 21
• Thames Valley - 10
• Warwickshire - 6
• West Mercia - 15
• West Midlands - 4
• West Yorkshire - 1
• Wiltshire - 13

So, maybe, think about who is not on the list... instead. Are they using Toyoda's 14 principles I wonder?

Section 38 & misconduct

This morning I had the pleasure of reading through the John Harris Memorial Lecture that Tom Winsor (HMCIC) gave to assembled guests of the Police Foundation in the middle of July this year. He has joined an illustrious list of previous lecturers which you can see here.

In case you think I am being tongue in cheek: I am not. His words deserve your attention particularly in the light of events in some police areas in recent weeks. His lecture was entitled Operational independence and the new accountability of policing and he focused on the legalities underpinning the relationship between the PCCs and Chief Constables in the new governance environment.

I am told that the lecture is better read than listened to (an option on the Police Foundation site linked above) since it was quite intense. Apparently, even the tweeters had to give up their multi-tasking and listen closely! Certainly, I had to read a few paragraphs and sentences more than once to understand entirely the point he was making. But I do commend it to you.

If I was being picky, I was somewhat surprised he did not mention the Independent Police Complaints Commission nor reference the excellent words of the Patten Report on operational accountability.

In the lecture, Tom Winsor unpicks the precise pieces of legislation that relate to the PCC/CC relationship and the inviolability of operational independence (at least in legislative theory...)

I am not going to summarise the piece, but I will lift just one quote (with my added emphasis):

It follows that the section 38 power can only properly be exercised for reasons which are related to the performance by the chief constable of these duties and functions and which affect the achievement of local policing needs and related priorities, and not misconduct.

I think that all PCCs and Chief Constables would be well advised to read this lecture, if they have not done so already...

The lamp posts are not for bending....

In my ongoing explorations of the statistics and rationale underlying the Government's resolute commitment to Payment by Results (PbR), I asked them a series of further questions after their responses to my last set. (These two blog posts show their answers and my queries). Here is a copy of the letter I received yesterday (my questions in bold, their answers in italics):

Many thanks for your email & attached response to my questions. I note what you say about using section 22 not to answer my questions 15 & 16. I will await publication of the new batch of statistics on 25/7/13 naturally. I reserve the right to appeal against your decision subject to this information being able to answer my questions.

With regard to some of your other answers, I would see further clarification as follows:

a)      With regard to Q2, you have not answered my question. You have merely reported on a series of facts and not explained the reasoning behind the decision to produce the statistics early. Please, may I request again, for my question to be answered fully. If that is difficult, I am happy to submit an FoI inquiry requesting all the email correspondence between senior civil servants and the Minister which led up to the early publication of this data. Which would you prefer?

As set out in the publication, we published the figures in an ad hoc bulletin on 13 June, rather than waiting to publish them in the Proven Re-offending statistics bulletin on 25 July, to ensure the information was made public as soon as it was available. This is in accordance with the Code of Practice for Official Statistics (http://www.statisticsauthority.gov.uk/assessment/code-of-practice/) which requires us to “release statistical reports as soon as they are judged ready, so that there is no opportunity or perception of opportunity, for the release to be withheld or delayed”. 

Once the MoJ Chief Statistician had judged that we were in a position to publish statistically robust interim re-conviction figures, we published them at the earliest opportunity.

b)     In your answer to 6/7 (you correctly identified that this is one question where a rogue carriage return had crept in) you referred me to “Table B3 of annex B from the MoJ’s proven re-offending statistics quarterly bulletin”. I looked at this table carefully but I could not see how it answered my query about the extent of the difference between national stats and the pilot’s stats. Please could you be more precise and show me more clearly how this factor is likely to affect comparisons. Thank you.

The two middle columns of table B3 show (for offenders released from prison or starting court orders) re-offending figures including and excluding cautions. 

Looking at the top section headed ‘Proportion’, column 2 (“Previous measure: re-convictions (prison and probation offenders only), whole year”) is the re-conviction rate excluding cautions – i.e. the Doncaster measure. Column 3 (“New measure: re-offending (prison and probation offenders only), whole year”) is the re-offending rate, including cautions – i.e. the National Statistics measure. This shows, for example, for offenders discharged from prison or starting a court order in 2009 the proportion re-convicted was 34.7 per cent (column 2) but when we also count offences that receive a caution the proportion increases to 36.2 per cent (column 3). The difference over time is small - between 1 and 2 percentage points.

As noted in our previous response, we have not produced alternative interim figures on what the impact would be if different rules (such as including cautions) had applied to the pilots. However, the figures in table B3 show the impact at a national level of including/excluding cautions.

c)     You answer to Q9 confirms, I think, that there is no element of randomisation in the selection of the ‘control’ comparator groups. As a scientist I find this most disturbing and I am not sure about you, but I don’t I would be prepared to undergo a course of medical treatment that had gone through a “quasi-experiment”. As PbR spreads (as I assume the Governments wants), it will become increasingly difficult to find comparator groups on this basis. Moreover, I cannot see, no matter how independent are the people who are choosing the comparator groups that this process will control for hidden factors. As a consequence, I do not think that you have yet answered my query “what is your considered professional judgement as a statistician as to the validity of these results to guide future practice?” I look forward to your thoughts. Thanks

The Payment by Results pilots were set up to test a range of approaches to achieving reductions in re-offending through paying by results, and different pilots use different payment mechanism designs. 

Propensity Score Matching (PSM) is a well established statistical method for creating a control group when it is not possible to carry out a randomised control trial (as it is not in this case). As set out in our previous response, the control group will be selected, using the published PSM methodology, by an Independent Assessor.

The Ministry of Justice’s  consultation response, Transforming Rehabilitation: a Strategy for Reform, described how, under our proposals, to be fully rewarded, providers will need to achieve both an agreed reduction in the number of offenders who go on to commit further offences, and a reduction in the number of further offences committed by the cohort of offenders for which they are responsible.

The consultation response stated that we would discuss the final details of the payment mechanism with practitioners and potential providers. To support this engagement, we have since published a Payment Mechanism Straw Man- available at www.justice.gov.uk/downloads/rehab-prog/payment-mechanism.pdf. 

While the final design of the payment mechanism is still to be determined, the model set out in the straw man discusses setting a baseline for all reduction in re-offending targets for each Contract Package Area on the basis of average quarterly re-offending figures for the most recent year that data is available. 

d)    In answer to Q10 you say “The five percentage point reduction target was agreed after analysis of historic reconviction rates established that this would illustrate a demonstrable difference which could be attributed to the new system and not just natural variation” (with my added highlight). However later on you also say “we have not carried out statistical significance tests on the interim figures because, when it comes to the final results, neither pilot will be assessed on the basis of whether they have achieved a statistically significant change”. How can these two statements be compatible? Forgive me, but it seems to me you are using significant differences when it suits you and not when it does not…? Please justify this approach.
&
e)     Moreover, given this last statement, may I confirm that taxpayers’ money may well be doled out to the suppliers on what could be a random happenchance difference in results rather than one which is (say) beyond a standard 5% statistical threshold of significance? I am interested in your views here too.

I can confirm that testing was used in the design of the Payment by Results pilots at both Peterborough and Doncaster, to ensure that the minimum targets for outcome-based payments in each pilot are set at such a level that we can be confident that to achieve them a provider must achieve an improvement which is attributable to their interventions and not just natural variation.  Because this significance testing is built in at the target setting stage, there is then no need to conduct tests for significance again once outcomes are calculated; instead, outcomes can be judged on whether or not they exceed the targets. The benefit of carrying out the statistical significance testing prior to the start of the pilot rather than at the end is that the ‘goal posts’ can then be set and known by all parties at the outset. In addition, because these targets are set in terms of the final 12-month re-offending measure it is not helpful to carry out statistical significance tests on the interim figures, which measure re-offending over just 6 months and, in the case of the figures for the Doncaster pilot, have smaller offender cohorts than the final measure. 

f)       Your answer to my question 12 surprised me. It is well known, I thought, that certain crimes rise in the winter such as burglary due to the darker evenings etc. Whilst I recognise that you are comparing ‘like with like’ that does not exclude a seasonal effect, it could merely exacerbate one since your time sample is not across the whole year. Why not provide the summer six monthly data as well? 

Using Doncaster as an example, we are not saying that the re-conviction rate for the Oct-Mar 6 months will necessarily match the re-conviction rate for the Apr-Sep 6 months. In fact, because of seasonality it is more likely that they will differ, as you say. Therefore, because we want to compare re-conviction rates over time, we must use the same period for the comparison in each year – that is comparing the various Oct-Mar periods over time. If instead we compared the pilot period of Oct11-Mar12 with say Jan09-Jun09, any difference could reflect a real change, but it could also simply reflect seasonal effects. By comparing the equivalent period in each year, we eliminate this risk of seasonality. 

g) I hear what you say about the 19 month period but it really does look shady! Why not 18 months? Why not 6 months? Hopefully the overall data will clear all this up.

We process and analyse re-offending data on a quarterly basis. For the interim figures released on 13 June, the latest quarter for which we could provide 6 month re-conviction figures was the quarter ending March 2012. The Peterborough pilot started in September 2010 (partway through a quarter), which meant we were able to report on a maximum of 19 months of the first Peterborough cohort period. We could have chosen to round this down to a more conventional 18 months but we took the decision that we should include as much of the data as possible to maximise the robustness of the figures. The Doncaster pilot began in October 2011, at the start of a quarter, meaning we reported on a more conventional looking 6 month period.


It is all getting rather convoluted (which is one of the problems I have with PbR in that payments will steadily become more and more like arguments about how many angels can fit on a pin head). However, there are some points I will be raising from all this... (for another day)

But what are your thoughts? What questions now need to be asked?

Meanwhile, if you have not read it, here is my blog post about the next batch results that were published a couple of weeks ago.

Technology & the Queen's Peace: a survey

As preparation for a conference in the spring of next year, I want to carry out a survey. I need your help. I would be most grateful if you could answer the questions below. There are just three and you don't even have to answer all three if you do not want to.

You can post your answers below anonymously or using your name. Or you can email me (JonSHarvey@ymail.com), tweet me (either @JonSHarvey or @CllrJonSHarvey) or text or phone me (details here). I don't mind how.

In return, I promise to publish the survey results here as soon as I have a substantive pool of replies to make that worthwhile.

Now to the questions:

1) What existing technology (that you have come across) would, with substantial investment, be an extraordinarily cost effective way of improving community safety / crime reduction?

2) What possible technology (that is just beyond what we currently have) would, with substantial investment, be an extraordinarily cost effective way of improving community safety / crime reduction?

3) What 'sci fi' technology (that is well beyond what we currently have) would, with substantial investment, be an extraordinarily cost effective way of improving community safety / crime reduction?

You may well have more than one answer to each of these questions, but please try to restrict yourself to the best one, in your opinion, for each.

I look forward to reading your answers. Please spread this around to others who you think might also wish to participate.

Thank you!

More tilting at lamp posts...

Thanks to a heads up from Kyle McKay I see that the MoJ have now published an update on the Payment by Results pilots in Doncaster and Peterborough. You can access it here. (My previous blog posts can be accessed here.)

Proving the PbR pilots have worked is still a long way off, it would seem to me: the MoJ concedes they do need full 12 month post release data in order to do a full blown comparison ("final results will not be available until 2014"). However, this does not prevent them from interpreting the results (in my view) creatively to show how PbR is working in these two areas.

However, I would point out the following:

• They say the "interim re-conviction figures being published in this statistical bulletin are based on periods half the length of those that will be used for the final results".  I say this is not just half this is the first half of a 12 month period. Second halves are also a little harder...
• How typical are Doncaster and Peterborough compared to the rest of the country? All the comparisons made are with national data. Given the news about Doncaster in recent months & years, it is hardly a standard place. Also Peterbrough is a 'new town' and (according to Wikipedia) "Peterborough's population grew by 45.4% between 1971 and 1991" which I think makes it a somewhat unusual place. So are national comparisons really valid?
• They say that "Both PbR prison pilots use a 12 month re-conviction measure which differs from the National Statistics proven re-offending measure. The key difference is that re-convictions only count offences for which the offender was convicted at court, whereas the National Statistics proven re-offending measure also includes out of court disposals (cautions)" and "Additionally, there are a number of other differences between the pilots and the National Statistics proven re-offending measure in terms of which offenders are counted within the cohort". That all seems pretty important to me... does it to you?
• Indeed the whole document seems peppered with so many caveats, footnotes and explanations as to make me wonder just what we are being told.
• They say "Success of the Peterborough pilot will be measured against a control group of similar offenders released from other prisons, with the target met if the frequency of re-conviction events is 10 per cent lower for the Peterborough cohort than for the control group. It is not possible to replicate that comparison for these interim figures". I say: why not? Why is the 'control group' not being monitored in a similar way? It does not make it much of a control group..!
• They say "The national comparisons included with the previous interim figures published on 13 June 2013 included all prisons, not just local prisons. However, because Peterborough is a local prison, using national figures for other local prisons provides a better comparison". Huh? Why were the local prison data not used before? Are they just using whatever data seems to give the 'best' result?
• It would appear that frequency of conviction rates in Doncaster have been coming down since September 2007 whereas nationally during the same period, national figures have been showing a rise since September 2007. The Peteborough pilot began in October 2010. At the very least this shows that national comparisons are dodgy since the trends were going in opposite ways before all the PbR pilots began. It also potentially shows that other significant forces are present in Peterborough that could be creating the positive trends other than the PbR pilots... 
• Similarly the Donaster reconviction rates have been in a downward trend since October 2007, the data appears to suggest: again well before the pilots were begun. 
• (As an aside: the national data shows some very worrying trends: reconviction events per 100 offenders has gone from 66 to 84 between June 2007 and 2012. That is a rise of  over 27%. That is a bit concerning isn't it?)
• But for me the biggest problem with this whole comparison approach is that a potentially huge Hawthorne Effect is not being controlled for. In other words, the mere presence and attention being given to the PbR pilots is what is creating any positive effect, not the pilots themselves. This is not, I repeat NOT, being controlled for. For me this calls into question the whole edifice on which these pilots are based.

Anyone with an ounce of independent thought will understand that, at the very least, these results are not the basis which to build a whole reform of the offender management system. The comparisons are shaky and riven with cautions.

It is time to develop a better experimental framework.

British Remote & Online Police Service

I first came across the internet acronym 'irl' many years ago but it was not long after I began living some of my life on line. Part of who I am, my identity exists in the virtual world of the internet. There are large bits of me, as were, held in binary code on a whole variety of servers. Some of this is current, some of this is historical. I have friends on the internet whom I will probably never meet in real life. We exchange greetings. I have been trolled and verbally abused. I have probably made thousands of financial transactions. Last week I initiated court action against a person online. Last night I watched ten minutes of a programme about One Direction's fan base and how they felt connected to their idols in ways that teenage fans of Donny Osmond and David Cassidy could only have dreamt of...

In recent weeks, there have been several tragic stories of young people being driven to suicide by abuse and threats via the internet. No doubt, although much less reported, many hundreds of people will parted with thousands of pounds via various kinds of internet scams. Probably also a fair few (dozens perhaps?) of credit cards will have been stolen, skimmed or cloned and used to pay for all many of items from airline tickets and tube fares. Also sadly some more children will have been groomed and put in danger of abuse. I could go on.

But who polices this virtual world?

Who has the resources not only to tackle such crimes when they occur but also the resources to reduce the risk of such crimes in the future? Who has the ear of the internet industry be they service providers, web designers, cloud managers and all manner of commercial people who make the internet work, so that robust preventative action can be comprehensively taken? What is the internet equivalent of a car immobiliser?

Who is taking a joined up strategic view on all this?

And yes I know we have CEOP and Action Fraud, and probably other units that I do not know about but I do wonder whether we now need a single joined up virtual police service to assemble all these resources together into one centre of excellence? Just like we have the British Transport Police (which in my view ought to look after policing at all airports and sea ports too - but that is another blog post) why do we not have the British Online Police Service?

And to complete the picture as I suspect many of these crimes overlap, I have added in the idea of 'remote' crime which would include in my book rogue phone calls and mail order scams (etc.) which also cause huge distress.

And so I arrive at the idea of the British  Remote & Online Police Service as a new legal entity, probably with new enforcement powers, a governance structure that includes the internet industry (similar to BTP) and clear partnership liaison with 'irl' police services and financial regulators. In these straightened times this will need some imaginative sources of funding (a broad band / junk mail tax perhaps?) to ensure it is adequately resourced.

Can I interest one of the major political parties in this idea in time for the next election? Or even sooner perhaps...?

'Light touch' police and crime panels must shift scrutiny powers up a gear

Yesterday, The Guardian Public Leaders Network published my article about how Police & Crime Panels need to shift up a gear or three in their scrutiny of PCCs, especially their decisions about scarce resource allocation. You can read the whole article here.

In the article I say:

The question is, how well the police and crime panels (PCPs) will scrutinise these resource decisions over the coming months. Chief constables will be using all their considerable skills to ensure good and professional decisions are made about the operational deployment of tight police resources but they will be subject to the policy influence of their PCCs – and that influence needs to be carefully unpicked by the PCPs...

So how is your PCP doing..? 

The politics of stats, trends & probability

I am no statistical expert and it is many years since I studied the application of Student’s T test, two tails and correlation coefficients to psychological experiments. But I have retained just about enough of what I learnt then and since (about targets, trend analysis and statistical process control) to be pretty darn fed up with how most politicians and members of the media treat data.

The ridiculous way in which the media and certain sections of the government are treating the results of Dr Bruce Keogh’s investigation into hospital care is an object lesson in how complex data sets are twisted into political rhetoric. Some of this is clearly about politics and for that I can almost forgive them. But when it comes to the proportion of this twisting that is down to plain ordinary ignorance, I really can’t!

(And if you want to know what I mean about the Keogh report, read this blog post and please read it carefully.)

We entrust politicians with a huge amount of power which they wield on our behalf. They spend vast amounts of money, our money, on projects which the evidence shows (if they looked closely) were never going to work. It is time this ended. All politicians ought to have a good introduction to stats, trends and probability (and scientific methods, while they are about it) so that they are better able to make decisions that will actually make a real difference.

Now I am not saying that all politicians and members of the media are ignorant of such matters, but many are. The information on Payment by Results that I have uncovered in the last few days disturbs me. In fact it horrifies me that we may well be paying service providers for results that could simply be chance results rather than the robust outcomes of a better service.

I do not intend to make this blog post a long lesson in stats. Other people can do that far better than me. But here is just one idea: if you were to throw a dice a couple of dozen times, you would expect the numbers to come up in reasonably even quantities. Perhaps the four might have come up 5 times and the three just once. But you would not presume the dice was loaded. However, if you threw the dice a hundred times and the three only came up (say) 5 times and the four came up (say) 26 times you would begin to think something odd was happening.

Stats is simply about measuring when the threshold between chance and a real variation (a loaded dice in this example) is crossed. Without statistics, you cannot know whether an occurrence of (say) less teenage pregnancy (etc. etc.) is just a random chance or that something ‘significant’ has happened (i.e. it is NOT chance, or at least very unlikely to be).

I would hope that most people reading this, get this. But do most politicians? And journalists? What do you think?

And I won’t even start to talk about about systems theory, the role of blame and the fact that complex things really are complex!! (I will leave that for another blog post.)

But please… please can we have less of the ignorance around trends, evidence and chance occurrences and a bit more understanding that ‘wicked’ social (and medical) problems require some pretty darn ‘wicked’ solutions…

Tilting at lamposts

Yesterday I received a response to my inquiries concerning the pilot data being used by the MoJ to support the extension of  Payment by Results. I have reprinted the letter I received in the blog post below.

This blog contains some commentary on the replies I received.

• The Moj used Section 22 of the Freedom of Information Act to not answer two of my questions. They say that the information I was seeking is about to be published on July 25. I am prepared to wait until then before deciding to appeal their reply or not.
• I asked why they published the results early, they said they did this "to ensure the information was made public as soon as it was available" despite admitting in other answers that it was incomplete. I suspect they wanted to get in some 'good news' before the summer vacation and in advance of CSR negotiations. But in my view, it looks shoddy and is evidence of using data and stats for political purposes. However, I guess all governments do that... don't they?
• Their answers do seem to assert that they have sought to compare like with like in terms of cohort comparisons.
• Their answer to question 4 does evidence the fact that this data is incomplete and premature, in my view
• They originally said that a key difference between the cohort is that in this group "reconvictions only count offences for which the offender was convicted at court, whereas the National Statistics proven re-offending measure also includes out of court disposals (cautions)”. I asked what the impact of that difference was likely to be. They referred me to "Table B3 of annex B from the MoJ’s proven re-offending statistics quarterly bulletin: https://www.gov.uk/government/publications/proven-re-offending--2". I have looked at this table and it is not entirely clear so I think I am going to have go back to them on this and seek further clarification. But do note that they said "We have not produced alternative interim figures on what the impact would be if different rules (such as including cautions) had applied". Which seems a bit sloppy to me. This is a critical difference after all and I suspect that if the data was showing not in favour of the pilot providers, they would be seeking further clarification!
• I asked whether the comparison groups (to evidence that the pilot intervention was in fact working) were selected using some kind of randomised selection. They said "The control group will be selected by an Independent Assessor using Propensity Score Matching (PSM), the methodology for which has been published at: Peterborough Social Impact Bond: an independent ... - Gov.uk" So the answer is 'NO': comparator groups will be selected by 'independent' assessor (being paid by the government, I assume). I looked the reference document and here is a quote from it: "It should be noted that, unlike random control allocation, PSM cannot take account of unmeasured differences which may account for variation in reconviction aside from ‘treatment received’" Uh huh. But it goes onto assert that: "However, PSM [propensity score matching] is widely regarded as one of the best ways of matching quasi-experimentally (Rosenbaum, 2002), and it has been increasingly used in a criminological context (e.g. Wermink et al., 2010)." So that is alright then. Excuse me while I give you this new medicine that has been quasi-experimentally tested on people who are sort of similar to you...
• I asked "For Doncaster, success “will be determined by comparison with the reconviction rate in the baseline year of 2009”. How will this accommodate national and/or local trends in (say) sentencing practice or levels of crime?". They replied "The five percentage point reduction target was agreed after analysis of historic reconviction rates established that this would illustrate a demonstrable difference which could be attributed to the new system and not just natural variation." That is not an answer to my question, I will need to go back to them on this.
• I asked about the 6 versus 12 month comparison and how the headline data (based on six months) was going to look against the usual (12 month) data. They said in reply "The statistical notice made clear the limitations of the information presented and the care that should be taken in interpreting these interim figures." Remind me - was that subtlety in the press releases that went out when this interim data was released...?
• They missed the point completely on my question about seasonality...
• Please read their answer to my question about why 19 month data. Please let me know what you think. I am thinking 'wool', 'eyes' and 'what do you really mean?!'
• The maths question is funny. They said "The figures presented were the rounded versions of the actual figures, which were 68.53 and 79.29". So I have done the calculations again and the result I get this time is 15.7. So they are sort of correct - but why leave out the first decimal point?
• I asked about statistical significance (the test of whether a difference is just a chance difference or one that indicates a real effect is in play). This is what they said "We have not carried out statistical significance tests on the interim figures because, when it comes to the final results, neither pilot will be assessed on the basis of whether they have achieved a statistically significant change."

OK. Let me repeat that again in big and bold:

We have not carried out statistical significance tests on the interim figures because, when it comes to the final results, neither pilot will be assessed on the basis of whether they have achieved a statistically significant change.

So, Payment by Results could well be based upon purely random chance events that may just have happened.

Is that a solid basis for the distribution of taxpayers' money?

Payment by Results, lamp posts... lit?

The day after #tagginggate you would expect me to be somewhat sceptical about how well government manages complex contracts with external suppliers. Moreover, of course, questions remain about how well the external suppliers manage these contracts too! But that is for another blog post one day.

But meanwhile, I received a reply to my questions about the Payment by Results pilots (and if you thought tagging contacts were complex...!). Below I have reprinted in full the reply I have received from the relevant person in the Ministry of Justice. It is already quite a long piece, so I will leave my commentary to another posting. Please read what they have to say critically - you will be able to then to see whether your thoughts match, contradict or add to my interpretations.

Dear Mr Harvey,

Thank you for your email of 13th June 2013, in which you asked for the following information from the Ministry of Justice (MoJ):

(I have left out their repetition of the questions - as they are shown below anyway)

I can confirm that the department holds information that you have asked for, however, please be aware that questions 15 and 16 of your request [these are the questions in question: 15. Given that you must have the data for Peterborough for the missing 19 month period (September 08 to March 11), and acknowledging that this overlaps with the pilot beginning, please could I have this data nonetheless.
16. Likewise, please could I have the data for the quarter beginning April 2012] have been handled under the Freedom of Information Act 2000 (FOIA) and the remaining questions have been dealt with as normal business.  

Section 84 of the Act states that in order for a request for information to be handled as a Freedom of Information request, it must be for recorded information. For example, a Freedom of Information request would be for a copy of an HR policy, rather than an explanation as to why we have that policy in place. 

Following our assessment of your correspondence we believe that questions 1-14 and 17-21 relate to general questions and not recorded information.

The responses are as follows:

Questions 15 and 16 – Dealt with under the FOIA

I can confirm that the department holds information that you have asked for, but it is exempt from disclosure because it is intended for future publication.

We are not obliged to provide information that is intended for future publication (section 22 of the Act). In line with the terms of this exemption in the Freedom of Information Act, we have considered whether it would be in the public interest for us to provide you with the information ahead of publication, despite the exemption being applicable. In this case, I have concluded that the public interest favours withholding the information.

You can find out more about Section 22 by reading the extract from the Act and some guidance points we consider when applying this exemption, attached at the end of this letter.

You can also find more information by reading the full text of the Act, available at http://www.legislation.gov.uk/ukpga/2000/36/section/22.

When assessing whether or not it was in the public interest to disclose the information to you, we took into account the following factors:

Public interest considerations favouring disclosure
• There are public arguments in favour of disclosure of this information at the present time.  Disclosure would for example improve transparency in the operations of Government, and of the justice system in particular.

Public interest considerations favouring withholding the information
• There are public interest arguments against disclosure of this information at the present time.  These arguments include that is in the public interest to adhere to the existing publication process for official statistics, which includes time for the data to be collated and properly verified.

• It is also in the public interest to ensure that the publication of official information is a properly planned and managed process, to ensure that data are accurate once it is placed into the public domain.  It is also in the public interest to ensure that the information is available to all members of the public at the same time, and premature publication could undermine the principle of making the information available to all at the same time through the official publication process.

We reached the view that, on balance, the public interest is better served by withholding this information under Section 22 of the Act at this time.

You may be interested to know that this information is due to be published in the MoJ’s Proven Re-offending Statistics Quarterly bulletin on 25th July 2013 at the following link: 

https://www.gov.uk/government/organisations/ministry-of-justice/series/reoffending-statistics

[It seems reasonable to wait until 25/7/13 to decide whether I will appeal this decision or not.]

Questions 1-14 and 17-21 – Dealt with as normal business

As mentioned above, we have dealt with these questions under the provision of normal business.

1.  The pilots began on 9 September 2010 and the 1 October 2011 (Peterborough and Doncaster respectively.) Please can you qualify “began”?

This means that each pilot included eligible offenders (as defined in Table A1, Annex A of the statistical notice) discharged from the pilot prison from these dates onwards. 

2. Given that “the next Proven Reoffending Statistics quarterly bulletin will not be published until 25 July 2013”, why did you publish your results today rather than a few weeks from now?

As set out in the publication, rather than wait until 25 July, the results were published in this ad-hoc bulletin to ensure the information was made public as soon as it was available. In accordance with the Official Statistics Code of Practice the publication date was pre-announced by MoJ statisticians in May 2013. The next Proven Re-offending Statistics quarterly bulletin on 25 July will contain updated interim figures for the pilots, with quarterly updates thereafter.  

3. I understand that “the interim re-conviction figures being published in this statistical bulletin are based on periods half the length of those that will be used for the final results” – daft question I am sure, but presumably this applies to both the ‘experimental’ subject averages and the national comparators?

Yes, the interim re-conviction figures presented in this publication have been produced in exactly the same way for each pilot prison and its national comparator.

4. You say that these “interim 6 month re-conviction figures are available for almost all of Peterborough cohort 1 (around 850 offenders) and half of Doncaster cohort 1 (around 700 offenders)”, please can you explain what has happened to the other portions of the cohorts and why they are included?

The interim figures have been provided for as much of each cohort as possible, but at this stage they do not include all offenders in cohort 1 of either pilot. This is because some offenders were released from prison too recently to be measured on this basis (the 6 month re-offending window and 3 month waiting period have not yet elapsed). However, they will be included in the interim figures in future as soon as enough time has elapsed to allow us to measure them on a consistent basis.

5. In terms of methodology, you say “offenders enter the PbR pilots after their first eligible release from the prison within the cohort period”, please can you explain “eligible” in this context and whether the national comparator figures also cover the same “eligible” group?

Not all offenders released from the pilot prisons are eligible for the pilots. The Peterborough pilot for example, only includes adult males released from a custodial sentence of less than 12 months, so a prisoner released from a sentence of 2 years would not be eligible.  For each pilot, the national comparator figures have been produced on the same basis using the same eligibility criteria. More details on eligibility are available in Table A1, Annex A of the interim re-conviction figures publication:

https://www.gov.uk/government/publications/interim-re-conviction-figures-for-the-peterborough-and-doncaster-payment-by-results-pilots

6. You explain that the key difference is that reconvictions only count offences for which the offender was convicted at court, whereas the National Statistics proven re-offending measure also includes out of court disposals (cautions)” and “Additionally, there are a number of other differences between the pilots and the 
7. National Statistics proven re-offending measure in terms of which offenders are counted within the cohort”. Are you able to say what difference these differences might make to the figures? For example, what number of offenders per hundred are usually subject to a caution (or similar disposal) as opposed to a court conviction?

We have not produced alternative interim figures on what the impact would be if different rules (such as including cautions) had applied. However for information on the effect cautions have on re-offending, please see Table B3 of annex B from the MoJ’s proven re-offending statistics quarterly bulletin:

https://www.gov.uk/government/publications/proven-re-offending--2

8. Again I assume that given that the “Peterborough pilot includes offenders released from custodial sentences of less than 12 months, whereas the Doncaster pilot includes all offenders released from custody regardless of sentence length”, the national comparisons are on a like for like basis?

Yes, the figures for the national comparisons are calculated on the same basis as their respective pilots.

9. You explain that the “success of each Peterborough cohort will be determined by comparison with a control group (of comparable offenders from across the country)”. How will this ‘control’ group be selected to ensure there is no inadvertent or unknown bias? Indeed was there (will there be) any form of randomised control trial element to either of these two trials (and extensions)? If not, what is your considered professional judgement as a statistician as to the validity of these results to guide future practice?

The control group will be selected by an Independent Assessor using Propensity Score Matching (PSM), the methodology for which has been published at:

Peterborough Social Impact Bond: an independent ... - Gov.uk

10. For Doncaster, success “will be determined by comparison with the reconviction rate in the baseline year of 2009”. How will this accommodate national and/or local trends in (say) sentencing practice or levels of crime?

The five percentage point reduction target was agreed after analysis of historic reconviction rates established that this would illustrate a demonstrable difference which could be attributed to the new system and not just natural variation.

11. Given that normally reconviction rates are measured on a 12 month basis and these interim results are measured on a 6 month one, how much is that likely (based on past data) to have depressed the reconviction rates?

The figures presented are our best assessment of change in re-conviction figures at this time and have been provided as an early indication of each pilot’s progress. It is not possible to say at this stage what the final 12 month re-conviction figures will be, though naturally the final 12 month re-conviction figures will be higher than the interim 6 month figures simply because offenders will have had more time in which to commit offences.  The statistical notice made clear the limitations of the information presented and the care that should be taken in interpreting these interim figures. 

12. You say “Whereas in this publication, to eliminate the risk of seasonality and enable a consistent comparison over time, all figures relate to offenders released in the 6 month period from October to March”. I may well be missing something here, but by only using the six winter months, are you not likely to increase the risk of a seasonal effect in the data? Please explain further. 

We would be risking a seasonal effect if we took the 6 winter months for the pilot period and compared them to a different period in other years. For example if we had compared October 11 to March 12 with January to June 2009, it would be possible that any changes were simply the result of seasonal effects rather than a real change in re-offending. Whereas by only comparing the Oct-Mar pilot period with other Oct-Mar periods, we are comparing like with like and have therefore eliminated the risk of seasonality 

13. Given that the Peterborough cohort finished on 1/7/12, and allowing for the 6 months plus 3 (for court delays), this takes us up to March 2013. So on this basis, why have the last three months of data (April, May and June 2012) been excluded? (As far as I can see there is no explanation of this decision, but forgive me if I have overlooked it.)

Before releasing official statistics the information needs to be collated, processed and quality assured. Re-conviction data for offenders discharged in April, May and June for the Peterborough pilot had not been fully collated, processed and quality assured in time for this publication. However, re-conviction figures for the full Peterborough cohort (including all releases up to the end of June 2012) will be included in the next quarterly update to be published in July. 

14. Given that I assume that data is ordinarily collected on a quarterly basis, it would have been helpful to have presented your data in a similar way so that trends could be spotted over time rather than use the fairly arbitrary 19 month period to show the data. Why did you present it this way? Please could I have the data on a quarterly basis.

The 19 month period was chosen as this shows figures for as much of the cohort as possible as explained in the statistical notice. It is not an arbitrary cut off, but simply the period of the cohort for which we were able to provide interim re-conviction figures. 

The interim figures were published as soon as the MoJ Chief Statistician judged that we were in a position to produce statistically robust interim re-conviction figures, meaning that the number of offenders being reported on was a large enough sample for each pilot.  We have not produced any figures based on quarterly cohorts because the numbers involved would be too small to give statistically robust information. 

Additionally, reporting on the cohort by quarter would not show a like for like comparison across each quarter, and would therefore be more likely to confuse than to provide meaningful information. The reason for this is that offenders join the cohort after their first eligible discharge within the period. However some offenders will be released from the prison more than once within the cohort period. These more prolific offenders (who are more likely to re-offend) would therefore be more likely to appear in earlier quarters than later quarters.

15. Given that you must have the data for Peterborough for the missing 19 month period (September 08 to March 11), and acknowledging that this overlaps with the pilot beginning, please could I have this data nonetheless.

See earlier response.

16. Likewise, please could I have the data for the quarter beginning April 2012.

See earlier response.

17. You say “Nationally the equivalent figures show a rise of 16% from 69 to 79 re-conviction events per 100 offenders”. How do you get 16%? I can see a rise of 10 ‘points’ or a rise of (10/69*100) 14.5%. 

The figures presented were the rounded versions of the actual figures, which were 68.53 and 79.29. 

18. (As an aside, this is quite a large rise nationally in re-conviction rates comparing the period from just before the last election to period after. Have national rates continued to rise or have they levelled off now?)

Re-offending rates for all adult offenders have barely changed in a decade. Please see the quarterly re-offending bulletin for information on national re-offending levels.

https://www.gov.uk/government/publications/proven-re-offending--2

19. You say “these interim figures show a fall in the frequency of re-conviction events at Peterborough” which is drop from 41.6% to 39.2%. At what threshold of probability is this statistically significant?

We have not carried out statistical significance tests on the interim figures because, when it comes to the final results, neither pilot will be assessed on the basis of whether they have achieved a statistically significant change.  Peterborough will be assessed by comparison with a national matched control group using a PSM methodology. Doncaster will be assessed against a baseline of calendar year 2009.  

20. Please can you confirm that the OGRS scores cited relate to the cohort groups in both Peterborough and Doncaster (rather than all offenders who were released)?

The OGRS scores relate to the offenders within each cohort.

21. Why are the national re-conviction scores given next to Doncaster data (which average 32.9%) differ from the scores given next to the Peterborough data (average 37.9%)? I now the period is different and there is some missing data, but this still seems like a large difference…

The criteria used to create the national comparator figures for the Peterborough and Doncaster prisons are different because the 2 pilots have different criteria. For example, the national figures for the Peterborough comparison will only include adult males released from custodial sentences of less than 12 months, whereas the Doncaster comparison includes all prisoners released from custody regardless of sentence length. For more information on the differences between the two pilots please see Table A1, Annex A of the interim re-conviction figures statistical notice.

Generally, re-conviction rates are higher for offenders released from custodial sentences of less than a 12 months than for all offenders released from prison.  Hence the national comparator group for Peterborough have higher re-conviction rates than the national comparator group for Doncaster. 

You have the right to appeal our decision if you think it is incorrect. Details can be found in the ‘How to Appeal’ section attached at the end of this letter.

I will stop there - the rest is pretty standard boilerplate about how to appeal etc. I will say thank you to the Justice Statistics Analytical Services (who signed the letter) for their work in responding to my challenges.

So what do you think about these answers to my questions? 
What would you comment upon?