Just over three years ago, when few had heard of him, Dr No wrote a post called The Collapse of the Probability Function. At its heart lies the troublesome paradox that, while we might know how a group of patients might fare, we have no way of knowing how individual patients will fare. We might know that of a hundred patients, five will die in the next ten years from a heart attack. What we don’t know is who of the hundred will be the five; and the flip side of that is, when as doctors we choose to intervene, as increasingly we do, there are ninety five souls now tangled in our medical web, with all that that entails, be it tests, treatments and general apprehension, who were never going to have a heart attack anyway, let alone die from one in the next ten years. That’s a whole lot of medical intervention without any benefit whatsoever – but what the heck – overall, we might save a handful of lives – or so the hopeful reasoning goes.
This problem – and major problem it is – of not knowing who will benefit, and for that matter who will be harmed also lies at the heart of the screening debate, which has once again been re-ignited by a ‘new’ report on the benefits and harms of breast cancer screening, ‘new’ being qualified because, though the report is new, the data it is based on is old. The arguments for and against screening symptom free women of a certain age for breast cancer have gone up and down like a tired see-saw for decades. Screening evangelists, cancer charities and cancer specialists, will insist that not to be screened is to play Russian roulette, just as the sceptics will warn that to be screened enters you in an alternative game of Russian roulette, with its risk of over-diagnosis and unnecessary, and potentially harmful of itself, treatment. But what neither side tells you is that, for the vast majority of those screened, screening will make not one jot of difference.
Exactly the same dilemma applies to the vast majority of patients who take medication: for most patients, the drugs don’t work. To many, this might seem a counter-intuitive, not to mention heretical, statement, but it is true. To understand why it is true, we need to consider the science – evidence – that we are supposed, in this enlightened age of evidence based medicine, to use in deciding whether or not to prescribe.
The evidence comes from clinical trials. The best of these are, to coin a bit of a mouthful, randomised double blind placebo controlled trials. What the mouthful is about is doing our best to remove the influence of bias (systemic influences that twist) on our results. Patients are randomly allocated to receive the active drug or a dummy pill, and in each case neither the patients nor their doctors know who gets the drug, and who the dummy. Patients are followed for a time, and the number of events – say heart attacks – counted for each group. The results can be expressed in various ways – a relative risk reduction (ten percent fewer deaths in patients receiving the active drug), an absolute risk reduction (sixteen fewer deaths) or, increasingly, as the NNT, the number needed to treat (we need to treat forty patient to prevent one death). The figures are mathematically related, but of the three, the NNT is, in information terms, the richest, but not always the most popular. Fifty percent fewer deaths, or sixteen hundred lives saved, sounds racier than the NNT is forty.
Now, let us consider what an NNT actually tells us. Real world NNTs vary tremendously. Those in single figures are considered excellent. Many common preventative drugs have NNTs in tens or even hundreds of patients: the NNT for a statin to prevent a heart attack over five years in people without a history of heart disease is, for example, around (estimates, unsurprisingly, vary) sixty. Around sixty patients need to take a statin for one to benefit; and that means – Dr No hopes by now you are getting the point – in fifty nine there was no benefit: the drug didn’t work as intended. In fact, some patients are harmed, but that is another story for another day, for today’s post is about whether drugs work, not whether they do harm.
The perfect NNT is one: you only need treat one patient to have one patient benefit. Such a drug, we can confidently say, does work. But the moment the NNT starts to rise above one, we find increasingly that for most patients, the drug doesn’t work. Even when the NNT is excellent, in single figures, for most patients the drug doesn’t work.
Let us consider – as an example – the NNT for aspirin for period pain, which is around 9 (never mind ibuprofen has a better NNT, at around 3 – the numbers are easier to appreciate with a slightly bigger NNT). Trials, of course, are ideally done on large numbers of patients, but what happens if we boil down the figures to the minimum needed to generate an NNT? [Empirical note: those who want to play around with the numbers can find an online RRR/ARR/NNT calculator here: select ‘Randomized Controlled Trial’ in the drop down box, and note that with small numbers of patients you will get comedy p/CI values.]
The trial, recall, has two groups: those who receive the active drug, and those who don’t. In our hypothetical boiled down example, we might have nine patients who receive aspirin, and nine who do not. Of those who do not receive aspirin, four are pain free at two hours, and five are not; of those who did, five are pain free, and four are not. Taking aspirin has converted one patient from still in pain to pain free.
But what of the other patients? From the control (no aspirin) group we know that four would be pain free anyway, even without aspirin; and from the treated group we know that four will still be in pain, despite taking aspirin. For eight of the nine patients in the active drug group, taking aspirin made no difference: they were going to be in pain or pain free, whether they took aspirin or not. The aspirin made no difference, which is why Dr No can confidently say, by a generous – some might say wild, even reckless – extrapolation, for most patients, most drugs don’t work.
This, then, is the paradox at the heart of modern evidence based population focused medicine. Whether we are looking at screening or treatment, we have to accept that, for most of those we screen or treat, our intervention makes not one jot of difference. They were going to get better or die, whatever we did.
That is not to say we should never screen or treat. Of course not. But it does mean perhaps we should cut the paternalistic crap, and fess up that, most of the time, for most patients, medical intervention makes no difference. Somewhat paradoxically, the more we understand what evidence based medicine is really telling us, the more we should dump the bluff, provide the facts, and let the patient decide.