Published on the LSE blog, 05 March 2012
The drip feed of pro-competition studies from Zack Cooper at LSE raises serious questions for the academic community and the public about what constitutes bad science and what to do about its politicisation. Recently, on 21 February in the columns of the FT, the Cooper and colleague Julian Le Grand warded off serious scientific criticisms of the studies with an ad hominem attack, categorising those in favour of competition as empiricists and those whose work is critical of markets in health care as intuitivists. In so doing they sweep aside decades of careful economic theory and evidence which shows why markets do not work in health services and distract the reader from the facts that their work is ungrounded and far from empirical. Their repeated claims that competition in the NHS saves lives and improves quality and productivity have no scientific basis.
In July 2011, Cooper and colleagues at the LSE press-released an unpublished paper to coincide with the prime minister’s announcement on the Future Forum which had been set up in response to deep public concerns about the Health and Social Care (HSC) Bill. These concerns resulted in the government suspending the legislative process for two months to undertake a ‘listening exercise’ with the public. The FT and The Economist put their paper centre stage in the HSC Bill debate. The authors were sufficiently persuasive for the prime minister to declare that “competition is one way we can make things work better for patients. This isn’t ideological theory. A study published by the London School of Economics found hospitals in areas with more choice had lower death rates.” The study in question claimed that “using [acute myocardial infarction] AMI mortality as a quality indicator, … mortality fell more quickly (i.e. quality improved) for patients living in more competitive markets after the introduction of hospital competition (to the NHS) in January 2006”.
The major improvements in outcome after acute myocardial infarction can be attributed to improvements in primary prevention in general practice and in hospital care, including the introduction of percutaneous IV angiography. The government’s own cardiac Tzar, Sir Roger Boyle, was sufficiently angered by their claims to respond with withering criticism: “AMI is a medical emergency: patients can’t choose where to have their heart attack or where to be treated!” It is “bizarre to choose a condition where choice by consumer can have virtually no effect”. Patients suffering “severe pain in emergencies clouded by strong analgesia don’t make choices. It’s the ambulance driver who follows the protocol and drives to the nearest heart attack centre”.
The intervention that the authors claimed reduced heart attacks and was a proxy for competition was patient choice. In 2006, patients were given choices of hospitals including private for-profit providers for some selected treatments. Less than the half patients surveyed in 2008 even remember being given a choice, and only a tiny proportion made those choices based on data from the NHS choices website. If patient choice was one of the two key elements of competition, it wasn’t prevalent and rather than being derived from the authors’ data, it was assumed.
Crucially, even if patient choice had occurred it does not explain why heart attack mortality rates fell. There is no biological mechanism to explain why having a choice of providers for elective hip and knee operations surgery (including hospitals which did not treat or admit acute MI patients) could affect the overall outcomes from acute myocardial infarction where patients do not exercise choice over where they are treated.
The problem of data dredging is well known; if you repeat an analysis often enough significant statistical associations will appear. But the authors make the cardinal error of not understanding their data and of confusing minor statistical associations with causation. Deaths from acute MI are not a measure of quality of hospital care, rather a measure of access to and quality of cardiology care. At best, what the paper appears to show is not the effect of choice on heart attacks but that if an individual has a heart attack in an area close to a hospital and their GP is near the hospital, then outcomes are better, but such findings are not new.
Cooper’s working paper which the government cited as supporting their reforms was subsequently published in the Economic Journal. That it got through that journal’s peer-review process is perhaps indicative of the poor understanding of healthcare and routine data from reviewers of that journal. Our response to their flawed work was published in a peer-reviewed piece in the Lancet. They responded with mainly ad hominem attack and we again responded with scientific criticism.
Last week Cooper and colleagues were at it again with another working paper (as yet unpublished in an academic journal) that was once again miraculously timed to coincide with an important event; the prime minister’s summit on the NHS Bill. This time the authors claim that length of stay fell more rapidly in NHS hospitals experiencing greater competition and that the risk of cherry picking by the private sector made a case for risk adjusted price. Once again, the authors were careless with the data and the study design.
There are three problems with their analysis of the data: they seem unaware that lengths of stay differ between the conditions they examine; they ignore the political context in which the data was generated; and finally, they show little knowledge of the particularities of the conditions they include and how these will affect the data.
Cooper and his colleagues use the average length of stay for four conditions, elective hip replacements, knee replacements, hernia repairs and arthroscopies, each of which differs widely in lengths of stay. Arthroscopy is usually done as an outpatient and may not be recorded on hospital episode statistics. Hernia repair is usually a day case although the average overall length of stay varies by type of procedure and with median lengths of stay of one or two days. In contrast, hip and knee replacements have median lengths of post-operative stay of four or five days again depending on the procedures and morbidity, with average lengths of stay in 2010-11 ranging from 5.9 to 8.2 days for hip replacements and 5.5 to 5.8 for knees. (See the Information Centre inpatient and outpatient data)
Thus, if providers have switched to arthroscopies and hernia repairs or to operating on patients who are well and healthy they will appear to have shortened their pre-operative and post-operative length of stay to less than a day. So a provider’s length of stay will depend on the mixture of operations and mixture of patients and how far they travel. The authors appear to have made no attempt to examine differences in case mix and length of stay. This is a serious error.
Equally, the authors do not look at how clinical coding changed following the introduction of the tariff in 2006. Gaming, upcoding and diagnostic drift are widely recognised in research on the NHS in the 2000s, with providers seeking to improve and increase their payments through fraudulent billing and accounting. This will apply especially where hospitals are under severe financial pressures and have strong motivations and perverse incentives to change the coding procedures. Arthroscopy procedures, which previously have been coded as an outpatient activity or not at all (i.e. it would not have been counted as an admission), may now be recorded separately as a day case inpatient procedure. These changes in coding distort measures of productivity so that providers may appear to be more efficient as they appear to do more work than they actually do.
Finally, length of stay is also a product of a range of factors related to the conditions in their data; pre-operative work for hip and knee replacement needs to take account of rurality and patient fitness for discharge, especially if patients live alone and have other co-morbidities and complexities. Patients who live close to a hospital may come in as an outpatient, while patients who live some distance away may require overnight stays. The authors should also have looked at readmission rates; premature discharge can result in readmission. The authors have not attempted to examine any of these factors, and neither have they considered the effect of hospital concentration on their data.
Le Grand and Cooper call themselves ‘empiricists’ and all those that disagree with them ‘intuitivists’. Unlike scientists, however, they have made no ‘real life’ observations themselves from which they have generated their theories. They do not appear to have the basic understanding of clinical practice. They have not made predictions, tested their theories with experiments, or adapted their models to see if they can do anything other than provide one explanation of many that could be derived from historical data. Moreover, they ignore the factors that underpin the generation of data and the need to understand how it is constructed and shaped. Data dredging has resulted in statistical associations but association is not causation. Bad science makes bad policy and bad policy leads to careless talk. Careless talk will cost lives especially when it is used in support of the HSC Bill.
1 thought on “Bad science concerning NHS competition is being used to support the controversial Health and Social Care Bill”
I have pondered with a degree of ambivalence for some time about making a comment. I don’t have any knowledge or involvement with medical matters, and so feel a little out of place here. However I read on the ‘Home Page’ of this website that your interests are more generic than just one subject. I have felt for some time a strong need to convey my thoughts towards someone with enough knowledge and wide ranging experience to understand my concerns. I have read your above article and the latter one with great interest. Many of your blogs seem to raise a real genuine concern regarding “privatization” and the perceived risks of less genuine care for patients rather than better care. I would have expected many false claims from private companies conveying that only with privatization can acceptable care be provided. I would have expected this issue to be around and much debate to be underway; with things even more vibrant under a Conservative government. However I do believe that anyone considering the future of the NHS, or lack of it; needs to look holistically and very generically at: international business, civil and military, politics and above all crime. I really do believe there is a vast difference between a simple privatize or not privatize situation and the one we have today. We could say that the news is privatized, and most mainstream media subvert by omitting important stories; and most of the general public are proud and willing participants; they will fiercely appose any other news or views. This situation has created a surreal world where even the obvious is unseen or perceived in a distorted way. We have seen how the banks have behaved over the last few years and the rise to power of not only those too big to fail; but those too big to prosecute. With the coming of the end of the Soviet Union we had a terrible crisis for those who understood real skulduggery; now we have a vast private military and security complex that no doubt want’s and will demand the NHS along with anything else desirable. I may be wrong, but I do not think so, if one looks outside mainstream media and cross checks the validity of things as carefully as possible; there is intelligence, rather than evidence, out there. Thank you for reading this comment.