Selective reporting bias: do you really need to report every outcome?

A large part of being a scientist is venturing into the unknown. You come up with hypotheses and test them through experiments. The problem is that more often than not, the experiments don’t work. By that I mean that they don’t give you the BIG interesting outcome you were hoping for, the one that will really make a difference to clinical practice, and instead often give you an equivocal (not significant) result.

Occasionally though, the experiment might give you an unexpected outcome that you didn’t plan for or were seeking, but might make the experiment more appealing for say publication. Wouldn’t it be nice to publish just that result and not the others that didn’t work or showed no effect? You could even make out that this unexpected result was the one you were looking for all along. Tempting isn’t it.

No, don’t go there.

In fact, what you’d be doing is introducing selective outcome reporting bias into your experiment, devaluing the validity of your results.

What’s the harm?

Selective outcome reporting can lead to wasted resources and potentially harms patients. Here’s why. Let’s say, as a very simple example, you were interested in doing a systematic review of RCTs for a new blood pressure lowering drug, let’s call it “SwitchBP”.

switch BP

You think there are likely to be enough comparisons of SwitchBP versus a current standard blood pressure lowering drug to do a meta-analysis, and your main outcome of interest is a reduction in blood pressure. Your systematic search finds 5 RCTs (Studies A, B, C D, E) of SwitchBP versus the standard drug. Four of them (Studies A, B, C, D) are smaller, but one (Study E) has a larger sample size and contributes the most to the overall effect size.

BP efffect

So it turns out that SwichBP is better than the comparator at lowering blood pressure.

However, you’ve read some case reports which suggested that SwitchBP might be associated with a number of adverse events, so you particularly want to analyse those as well.  But as far as you can tell only 4 RCTs provide any quantitative data on adverse events (all showing no significant difference from the control arm) and Study E simply states in the text “no significant difference was noted in adverse events between each arm”.

adverse events no data

You are a little suspicious of the trend and contact the authors of study E for any quantitative data they can share, but get no response. So you continue with your review and inevitably conclude no overall increase in adverse events from SwitchBP based on the published data.

A short time later, you are thrilled when your review gets taken up by the new blood pressure lowering guidance and SwitchBP is recommended as a “safe and effective” drug to lower blood pressure.

But here’s the rub. How comfortable would you feel if you then found out that the data you hadn’t seen (from Study E) might change your overall conclusions? Because in Study E (the largest), the authors had chosen not to report their actual data on adverse events, possibly because they thought there was no significant difference between arms.  However, including these data in your meta-analysis would have shown that, overall, SwitchBP was in fact associated with a significant increase in adverse events.

adverse events data


However, this is possibly too late for your published systematic review, which is already included in guidance, supporting the use of SwitchBP in thousands of patients, despite an increased risk of adverse events. Of course the benefit to harm balance may still favour using SwitchBP, but were you able to generate conclusions based on all the data in the first place?

Of course this is a very simplistic example, but one where the principles and effects of not disclosing full outcome data reflects a not uncommon practice. This and suggestions of what to do to reduce selective outcome reporting are discussed in the next blog.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s