Lies, damned lies and statistics: How do you turn 61% into 95%?

Image from "Working together for a stronger NHS" Crown Copyright

Edit: 13/05/11 An analysis of the BSA 2007 dataset by Siobhan Farmer, Mark Hawker and myself has been published today in the new publication Lancet UK Policy Matters. You can find it here. We conclude that it is not possible to conclude that 95% of respondents wished for more choice. Using the kind of suppositions given below it may be possible to infer that between 61% and 72% may have thought there should be more choice, but the survey was not designed to answer this question. 

..........................................................................................................................................................

Edit: 17/4/11 I and a colleague have independently tried to verify Mark's analysis. We have reached similar conclusions but they don't corroborate Mark's results. Unfortunately he is currently outside the UK. We will update with our own results in the UK. We are in agreement that there is still no justification for claiming that "95% of people want more choice in healthcare". 

..................................................................................................................................................................

Today many of you will have read Ben Goldacre's excellent  analysis of the leaflet which the Department of Health issued last week to help the public understand why they were pursuing reforms of the NHS which are facing widespread opposition. 

Page 11 of the leaflet contains the graphic above stating that 95% of those surveyed in the 25th British Social Attitudes Survey  wanted MORE choice in the NHS. When Ben looked at the published reports he found that 'Do you want more choice in the NHS?' was not a question in the survey. Instead respondents were asked 'How much choice do you think you should have?' and 'How much choice do you actually have?'  But if you could establish how many people thought they should have choice but who currently think that they don't have choice then you might be able to say how many people think that they should have more choice. As Ben points out to answer this you would need to have individual level data. When he asked the DOH for the dataset they unfortunately pointed him to a book chapter.

But fortunately the day this leaflet was published, April 6th, 2011, Mark Hawker  started wondering if he could find out more about this dataset. And he did find more. The individual level data is available to download. So Mark did that. Then he analysed it. And what did he find?  Well, you can read a lot more in the blog post that he published on April 7th but here is a summary.

13% thought they had more choice that they thought they should have.

46% thought they had just the right amount of choice.

41% thought they should have more choice that they had.

So how did the DOH manage to get this so wrong? How did they confuse 41% with 95%? Why weren't they able to direct Ben Goldacre to the correct data source? And why have they decided not to fund this survey in the future?

Hopefully someone can help make the correct data look just as pretty as the incorrect infographic in the leaflet. In the meantime I think I'd like to thank Mark for his work, and to agree with this tweet: