Nature Cooks the Books

Even at the best of times, epidemiology can seem as dry as old biscuits, and when it starts counting stiffs – as it so often does – it can smell not just dry and old, but musty too. But it is an important ology, and when done well, which is surprisingly difficult, it can tell us useful things.

This morning on Radio 4’s Today programme, we had Professor Bruce Keogh, a Department of Health chief pongo, putting the icing on an epidemiological cake he had himself helped bake. Having started soundly enough, he suddenly veered off-piste, gathered speed, and delivered a shopping list of deliverables that bore no relation to the study he and Wingnut had only a moment ago been discussing. “For me,” Prof Cough said, “the big prize …is about our NHS catching up with other service industries and offering a routine six or seven day week…where people can have routine operations over the weekend…at their convenience…access to expert advice…that’ll put the compassion back into the NHS”. Compassion? It sounded more like commerce to Dr No. It was, of course, a government inspired vision for the 24/7 hospital, where every day is Wednesday, every patient a consumer, and all the clocks strike thirteen.

The study, available here, appears sound in its data collection and analysis. Unlike recent comedy research by Dr Foster, which relied dodgy DIY trust data to show that a lower hospital consultant presence at weekends was linked to higher patient death rates, Prof Cough’s study relies on national data (albeit the hospital admissions data is self-reported by trusts), and the analysis was, so far as it was able to go, commendable. The researchers found that, for every 100 deaths to patients admitted on a weekday (they chose Wednesday), there would be 116 deaths to patients admitted on a weekend day. Put bluntly: you are more likely to come out of hospital feet first if you go in at the weekend. So far, so good – or bad, if you happen to be admitted over a weekend.

But – and is a big but – this nicely baked epidemiological cake is only good so far as it goes. One can’t go around layering it with speculative icing, and expect the icing to look good. And yet that is just what Prof Cough did: he leapt from an established association, via a speculative (he himself described the situation as “complex”) causation (not enough senior doctors around at weekends), to an imaginary solution: the 24/7 hospital.

There are at least two reasons why the cake cannot take the icing put upon it. The first is that none of the data collected measured actual clinical activity – number of consultants and other grade staff present, ward round frequency and diagnostic and operative activity, for example – so the study is incapable of showing an association between lack of clinical activity and subsequent mortality, even if one existed. But even if that data had been collected, and used in the analysis, there is another potential flaw, the bane of the epidemiologist’s life: bias. Unlike random error, which can be largely dealt with by increasing sample size, bias causes a regular, systematic error. It does so because something is wrong with the conduct and/or analysis of the study.

Much of the art and science of epidemiology is directed towards spotting and countering bias. Unspotted, un-countered bias remains an ever present danger; the black ice on which even the soundest epidemiologist can slip, and leave the road of truth and unwittingly end up in a duffer’s ditch of despair.

The varieties of bias are as they say legion, but it seems to Dr No that – just for starters – there is a common form of bias, selection bias, that could very easily apply to this study. Selection bias occurs when something causes – selects – patients with differing and, crucially, relevant characteristics to be unequally distributed between differing groups in a study. More patients with more severe disease, for example, end up in group A, and it is this, rather than differences in treatment between group A and group B, that explains why there are more deaths in group A.

Now, if we think about our own behaviour when we are ill, we can see all too easily how just such a selection might happen in the present study. Many people, especially the elderly, who are likely to figure large in this study, are reluctant to call the doctor out of hours, all the more so at the weekend; those that can will hold out until Monday morning, as any GP knows only too well. Those with more severe illness, however, will not stand the wait, and so will call the doctor straight away and so be admitted to hospital there and then. And so it is that more patients with more severe illness may be selectively admitted at weekends; and this in turn may account for more deaths in those admitted at weekends.

None of the data collected in the study measures of the severity of actual illness giving rise to admission, and so we cannot allow for it in our analysis. We do not know whether Sunday’s chest pain turned out to be indigestion, or Wednesday’s pain the start of a massive heart attack, or, of course, the other way round. If, as seems very possible, Sunday’s pain is indeed more likely to be cardiac, then it is equally possible that greater severity of illness, rather than differences in staffing, at weekends could accounts for the higher mortality following weekend admission.

Not for the first time, and certainly not for the last time, we may find that nature, through bias, has cooked our books.