While some people put a lot of faith in published scientific and medical research, there are a number of reasons why we sometimes need to be a bit sceptical about what we read in scientific journals. One fundamental issue here concerns what is known as ‘publication bias’ ” the tendency for favourable results to be more likely to be published than less favourable findings. For example, let’s say a drug company funds a study into one of it products and this yields results which put the product in a favourable light. You can imagine that the drug company will be keen to have this study published, preferably in a prestigious or ‘high impact factor’ journal. However, if the results were unfavourable, however, you might imagine that the drug company may be none too keen for the study to see the light of day.
A few years back, I was having lunch after giving a lecture to a group of business consultants. While chatting one of my lunch-mates volunteered that one of her clients, a drug company, had some years before embarked on a study involving one of its products. The lead investigator had insisted before the study began that the trial be published on completion, whatever its findings. The drug company agreed to this proviso. However, the preliminary results were not good. According to my lunch-mate, the drug company committed to keep the study running so that, in essence, it was never concluded. The expense of this was viewed to be much more attractive than the loss of revenue the poor study results would likely have induced. You can read about publication bias here.
Some medical journals have attempted to reduce the risk of publication bias by encouraging that trials are officially registered (preferably prior to them starting, and hopefully before the study ends) as a precondition for publication. This way, it has become harder for the results of unknown studies to be buried. These new recommendations, set by the International Committee of Medical Journal Editors, came into force in 2005.
However, despite this step, it seems there is still potential for researchers and their sponsors muddy the waters. Here are some ways they can do this:
Don’t register the trial (as yet, it’s not mandatory to do so)
Register the trial, but wait till after it has been completed (this gives an opportunity to choose the ‘favourable’ outcomes to register)
Be unclear in the registration what outcomes are to be measured
Register certain outcomes, but present different outcomes in the published study
In a study published this week in the Journal of the American Medical Association, French, Canadian and British researchers attempted to quantify these problems exist in published medical literature [1]. They assessed this on the basis of 323 studies (in the areas of cardiology, rheumatology and gastroenterology) published in 2008 in the 10 medical journals with the highest impact factors.
Here, in summary, is what they discovered:
Less than half (45.5 per cent) of all trials had been adequately registered (i.e. registered prior to completion of the trial, with the measured outcomes clearly stated)
27.6 per cent of studies were not registered at all
13.9 per cent of studies were registered after they had been completed
12 per cent of studies were registered without a clear description of the outcomes being measured
Of the adequately registered trials, 31 per cent showed evidence of discrepancy between the outcomes that were registered and the outcomes the were published
In half of these studies it was possible to assess the influence these discrepancies had: in 82.6 per cent of these studies, there was evidence that the discrepancies favoured the publication of statistically significant results
So many trials remain to be adequately registered (which means that there is still much potential for publication bias). And of those studies that were adequately registered, there’s still a tendency for research (and likely, their sponsors) to shift the goalposts in terms of what the trial was seeking to show.
The authors of the JAMA study make some recommendations to help stamp out these sources of bias. They include:
The sponsor and principal investigator should ensure that the trial details are registered before enrolling participants.
The comprehensiveness of the registration should be routinely checked, especially regarding the reporting of important items such as the primary outcome.
Editors and, possibly, peer reviewers should systematically check the consistency between the registered protocol and the submitted manuscript to identify any discrepancies and, if necessary, require explanations from the authors.
Along with this, the authors point out that mandatory publication of results would also make it much harder for researchers and their sponsors to ‘hide information’.
References:
1. Mathieu S, et al. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 2009;302(9):977-984
