Selective reporting bias: types, impact and ways to reduce it

Selective reporting of outcomes is just one type of reporting bias and there are a number of ways in which it can arise. In the previous linked blog we gave an example of the effect of selective reporting bias through under-reporting of data. So what could have been done to avoid the “SwitchBP” scenario?

Perhaps the first step is to recognize, in a little more detail, the types of selective reporting that exist. These are summarised in the Cochrane handbook but include:

  • Selective omission of outcomes from the published study;g. results that are not significant are deliberately left out
  • Selective choice of data for an outcome;g. outcomes are taken at different time points and only some, usually those that show a favorable result, are reported
  • Selective reporting of different analyses using the same data;g. a study that is measuring changes in say kilograms of weight (a continuous variable) shows that the results have more impact when reported as BMI <25 or BMI >25 (dichotomous variables)
  • Selective reporting of subsets of the data;g. a study that plans on reporting the total number of strokes but ends up reporting only ischaemic strokes and not haemorrhagic ones
  • Selective under-reporting of data;g. a study that plans to report a specific outcome, for example a change in blood pressure, shows no differences between interventions, and the authors choose to avoid reporting the actual data and instead just state that the difference was “not significant”

Prevalence and impact of selective reporting

A 2009 study showed that in a cohort of registered trials, 31% (46 of 147) had some form of discrepancy between the outcomes registered and the outcomes published. Furthermore, when it could be assessed, the studies with changes were more likely to report a statistically significant result.  In a of Cochrane systematic reviews, over a third were suspected to have at least one RCT containing selective outcome reporting bias. The authors also demonstrated the impact of this bias and found that selective outcome reporting can produce a median change in the treatment effect size of 39% (IQR 18% to 67%).

The authors of a 2015 systematic review of studies that examined selective outcome reporting (a meta-epidemiology study) found 27 studies. The median proportion of trials with an identified discrepancy between the registered and published primary outcome was 31% (although there was large variability between them). Four studies observed outcome changes in more than 50 % of trials.

Reducing selective outcome reporting

Recently The Compare Project was launched. The aim is to prospectively audit the presence of outcome switching in all RCTs published in the top five medical journals (NEJM, JAMA, The Lancet, Annals of Internal Medicine, BMJ). The outcomes presented in each published trial are compared with the stated outcomes in the clinical trial registry and protocol. When any outcome switching is noted, a letter is sent to the journal editors to inform them. As stated in the project approach: “Through increased awareness of misreported outcomes, individual accountability, and feedback for specific journals, we hope to fix this ongoing problem”. The results are being posted and updated live, and as I write the overall mean proportion of prespecified outcomes reported is 58.9% of 44 trials assessed. Look and see if the results have changed.

But the long term “fix” to reduce selective outcome reporting bias shouldn’t actually be that difficult. Reviewers and editors should make it routine practice (and many do) to check that the published outcomes match those in the trial protocol or registry (e.g. the ISRCTN or ClinicalTrials.gov). If the protocol is not available it should be requested directly from the corresponding author. If all that is not possible (which would be a concern in itself), then as a reader and at a minimum, check the outcomes reported in the methods section with the outcomes reported in the results section of the published article.

The CONSORT statement is a minimum set of recommendations for the reporting of randomized trials and it specifically includes the following:

 

Outcomes 6a Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed
6b Any changes to trial outcomes after the trial commenced, with reasons

 

Cross-checking of CONSORT statements completed by authors is also good practice, to ensure accurate reporting.

However, although many journals endorse the CONSORT statement, not all enforce it.

Trialists should be ensuring that all data are as transparent as humanly possible. Open access journals and open repositories for data deposition aim to ensure transparency and access to all data. There will be little place to hide should you choose otherwise.

But the bottom line, if you want to be a good experimenter, is that if you say you’re going to do something:

  1. do it
  2. show that you did it, and
  3. show how you did it.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s