Paul Corrigan, whose posts show a worrying trend towards titles so long they stand as posts in their own right, has declared himself a friend of FFT, the punter-friendly friends and family test based on asking patients at or soon after discharge whether they would recommend the unit they have just left to friends and family. The test is popular with government for its apparent simplicity, resented by managers for the real extra burden it imposes, and derided by front-line staff, for whom the test might be better known as the Flying F*ck Test: the punters don’t give a FF about responding, and we don’t give a FF about the results. Although first announced last year, FF testing was back in the news last week after friendly we’re all in this together Dave announced plans to extend FF testing to general practice. The news got a cool response from senior GPs; others went further. One called the test ‘meaningless’; another dubbed it ‘trite’.
Dr No approached this post expecting to do a routine shaft of a daft idea. Despite the superficial appeal of a neat single global measure of patient satisfaction, the fact is that if it is to do with the health service, and we’re all in this together Dave is in favour of it, then there is usually something wrong with it. It didn’t take Dr No long to find the idea’s Achilles Heel. Like most customer surveys, the response rate for FF testing is abysmal. The governments own FFT guidance ‘expects’ rates of ‘around 15%’, adding, as another porker roars by on after-burn, ‘for the majority, this figure could be much higher’. The overall (but with wide variation between trusts, with privately run Hinchingbrooke being notably high, but that’s another story for another day) response rate in a pilot study was in fact slightly better (current latest figure 18.99% for Nov 2012), but that doesn’t alter the fact that more than eight out of ten punters didn’t give a FF. Dr No raised the shaft of non-response bias, and was about to impale the daft idea, when he suddenly thought: careful.
Careful, because the last paragraph contains two assumptions, and assumptions are as we know the mother of all. The first assumption is that Tory health service ideas are routinely bad – an easy enough habit to get into given recent performance, but not necessarily the case. The second is that low response rates will result in non-response bias: the two out of ten who did respond are by definition exceptions, and therefore somehow freaks. Even if the gross demographics of responders and non-responders are shown to be the same, the discrepancy between two and eight out of ten is so striking as to give rise to a presumption of divergence of opinion on the matter of substantive interest between the two groups. How could a feeble two out of ten minority possibly represent an whopping eight out of ten majority? The responders, we feel intuitively, are bound to be unrepresentative; and the bigger the non-response rate, the bigger the bias. Low response rates, prima facie, are bad.
Well, not necessarily so. It all depends on whether there is a material – that is, relevant to the question at hand – reason governing whether a person responds or not. Such material reasons are very plausible: non-responders may not respond because they are disgruntled, and fear a black mark if they express their disgrunt. Responders may be toadies, eager to please. All manner of imaginary reasons can be imagined. But what is the evidence?
The evidence, somewhat counter-intuitively, supports ‘not necessarily so’. As humans, we naturally ascribe reasons to behaviour, but what if response/non-response behaviour is not reasoned behaviour, but random? What evidence there is suggests that the hand that guides response can indeed be the hand that rolls dice rather than the hand of reason. Although the setting, which may or may not matter, is different, eye-popping research by Keeter, Curtin and others on telephone surveys shows response rates appear to have little meaningful impact on results. One typical study, comparing ‘standard’ (25%) with ‘rigorous’ (50%) rates, found that, for ‘77 out of 84 comparable items, the two surveys yielded results that were statistically indistinguishable’. Perhaps low response rates aren’t an Achilles heel after all.
So Dr No has decided to put the shaft back in its box, and consider that FF testing may not be so daft after all. In fact, so long as the natural constraints of a single punter-based quirks-mode assessment are always borne in mind, Dr No finds the idea of FF testing appealing, precisely because it is a single comprehensible to all global assessment. Whether it would have turned on the red light at Mid Staffs faster than Sir Jar’s FosterKit is a moot point: its appeal, in contrast to Foster, is in its simplicity. Rest assured that, should it ever get ideas above its station and claim to measure complex detail beyond its capabilities, then Dr No will have that shaft back out of its box before you can say daft.