Jefferson T (2019). Sponsorship bias in clinical trials – growing menace or dawning realisation?

© Tom Jefferson, Email: tj@cochrane.dk


Cite as: Jefferson T (2019). Sponsorship bias in clinical trials – growing menace or dawning realisation? JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/sponsorship-bias-in-clinical-trials-growing-menace-or-dawning-realisation/)


Introduction

If therapeutic decisions in healthcare are to be informed by the results of clinical research, patients and prescribers must be able to trust the research evidence presented to them. In recent decades the credibility of much of the evidence base for some of the most popular therapeutic and preventive interventions has been undermined by the identification of sponsorship bias.

Sponsorship bias is the distortion of design and reporting of clinical experiments to favour the sponsor’s aims. By using the word ‘sponsor’ I am not implying here that the origins of bias are solely or principally commercial. Sponsors are the funders and stakeholders active within the design, setting-up, running and reporting of clinical trials, including the members of research teams.

Until recently the distortions introduced by sponsorship bias were recognised as important but difficult to identify with certainty because of the secrecy surrounding pharmaceutical trials. Recent developments, such as relaxation of access to regulatory material (Gøtzsche and Jørgensen 2011), have led to relatively successful efforts to identify and describe sponsorship bias.

The problem of sponsorship bias was recognised a century ago. In 1917, Torald Sollman warned of the effects of secrecy and closeness between those who manufacture drugs, those who test them, and those who publicise them. As a member of the Council on Pharmacy and Chemistry of the American Medical Association (forerunner of the US Food and Drug Agency), Sollman was able to identify “poor quality” and secrecy (publication bias) as major threats to the credibility of the reports submitted to the Council.

Some of the papers masquerade as “clinical reports,” sometimes with a splendid disregard for all details that could enable one to judge of their value and bearing, sometimes with the most tedious presentation of all sorts of routine observations that have no relation to the problem”…

…“when commercial firms claim to base their conclusions on clinical reports, the profession has a right to expect that these reports should be submitted to competent and independent review. When such reports are kept secret, it is impossible for anyone to decide what proportion of them are trustworthy, and what proportion thoughtless, incompetent or accommodating

…“Those who collaborate should realize frankly that under present conditions they are collaborating, not so much in determining the scientific value, but rather in establishing the commercial value of the article.” (Sollmann 1917)

In Sollman’s simple observations lie the nub of the problem. When the clinical trial is designed to advance public understanding, its design and reporting are less likely to be distorted by bias. However, when trials are run for reasons other than advancing knowledge – such as licensing, profit, or career enhancement – bias is likely to creep in, even if it has not been present from the start.

It was not until six decades after Sollmann’s article that empirical investigations of sponsor bias began. In 1980, Elina Hemminki reported her pioneering study of 566 reports of clinical trials of psychotropic drugs submitted in support of licensing applications in Sweden and Finland over a period of four non-consecutive years in the 60s and 70s (Hemminki 1980). Hemminki’s aims were threefold: to define the number of trials accompanying applications, to determine the quality and publication fate of these trials, and to assess whether data contained in the reports could be used to identify harms. Her research may have been the first use of regulatory documents to explore reporting bias and its association with the contents of reports. The list of problems she identified might well have been compiled 30 years later: secrecy, cherry picking of results, publication bias, ghost authorship, distorted design in favour of testing short term effectiveness, and the inverse association between the reporting of harms and the likelihood of publication (Dickersin and Chalmers 2010).

To investigate the specific nature of sponsorship bias in this article I review subsequent applications of Hemminki’s conceptually simple expedient of comparing the reports of clinical trials of medicinal products submitted to regulators with subsequently published reports of the same trials. Although the available evidence relates principally to clinical trials of commercial interest this should not be taken to imply that academia has higher standards of reporting – available evidence suggests that it does not (Goldacre et al. 2018) – but rather that sponsorship bias is more difficult to study in non-commercial trials.

One manifestation of sponsor bias is choosing comparators to give the impression that new drugs are more effective or safer than existing alternatives (Mann and Djulbegovic 2012). Further on in this essay I present examples showing how design distortions can alter the conclusion of a clinical trial or mislead readers.

Sponsorship bias reflected in reporting biases

In the two decades that followed Hemminki’s observations, the problems of sponsorship bias and its most important consequences were investigated and defined by a handful of investigators who assessed projects submitted to regulators or research ethics committees. Easterbrook and colleagues found that only 48% of trials submitted to an ethics committee between 1984 and 1987 were published, and that publication was more likely if statistically significant differences had been found (Easterbrook et al. 1991).

In 1992 Dickersin and colleagues added another dimension. Using a similar sample, they documented publication bias associated with external sponsorship. Their questionnaire survey found that the investigators were reluctant to submit reports of disappointing research for publication, although NIH-funded studies had a higher publication rate than industry-backed studies (Dickersin et al. 1992).

Melander and colleagues assessed 42 placebo-controlled trials of selective serotonin reuptake inhibitors (SSRIs) submitted to the Swedish regulators between 1983 and 1999. They found that analysis methods set out in the protocols were ignored and instead the analyses most favourable to new products were presented. Multiple publications of the same ‘positive’ trials were also frequent (Melander et al. 2003).

Chan and colleagues followed up a cohort of 274 trial protocols submitted to a Danish research ethics committee in the 1990s and compared their content with subsequent reports. This comparison revealed systematic differences in outcome reporting associated with statistical significance of the comparisons. Higher rates of significant harms were associated with a lower likelihood that these would be reported publicly. The investigators denied supressing data despite clear evidence to the contrary (Chan et al. 2004).

An important innovation in methods used to study sponsorship bias was reported in two papers published in 2008 (Turner et al. 2008; Rising et al. 2008). These compared information in freely available Food and Drug Administration (FDA) regulators’ reports on the strengths and weaknesses of the population of clinical trials submitted by sponsors to support applications for licensing with information in publications. This investigative innovation was important: these FDA reports are often very detailed and list all the trials relevant to the indications specified in the applications for licensing. Although those investigating possible sponsorship bias did not see the actual submission, they had access to the views of regulatory reviewers. In some instances, they had re-run their original analyses with data provided by the sponsor with each submission. Both these assessments (Turner et al. 2008; Rising et al. 2008) revealed major discrepancies between FDA assessments and conclusions in subsequently published reports of the same trials.

In a further important analysis, Psaty and Kronmal compared internal sponsor documents with mortality data presented to the FDA with two published trials of the drug rofecoxib for Alzheimer’s Disease and cognitive impairment. The sponsor concealed from the FDA their own internal analysis showing excess mortality in the intervention arms, while claiming no excess mortality by the simple expedient of ignoring the deaths in the 2 off-treatment weeks of follow up. Neither publication contained any statistical analysis and the reporting of the time between exposure and death was unclear (Psaty and Kronmal 2008).

Understanding sponsorship bias

Thanks to the work of groups who accessed regulatory material by statutory means, litigation or media pressure, the decade beginning in 2009 witnessed major advances in understanding sponsorship bias and its effects.  All three of the examples which follow have used Hemminki’s comparison method.

The group working in the German Institute for Quality and Efficiency in Health Care (IQWIG), empowered by legislation to access regulatory submissions, has produced cogent evidence of systematic selective reporting of data, both to regulators and in publications, with overemphasis on benefits and under-reporting or non-reporting of harms (Eyding et al. 2010, Wieseler et al. 2012, Wieseler et al. 2013, Köhler et al. 2015).

By highlighting the discrepancies among different sources of data for the same trials, primarily publications (when available) and regulatory documents, other assessments have led to a fundamental reappraisal of the properties of drugs and biologics (Coyne 2012; Jefferson et al. 2012; Rodgers et al. 2013; Fu et al. 2013; Jefferson et al. 2014; Vedula et al. 2013; Maund et al. 2014; Lawrence et al. 2015; Cosgrove et al. 2016; Hodgkinson et al. 2016; Jureidini et al. 2016; Schroll et al. 2016; Mayo-Wilson et al. 2017). More detail of each of these assessments is provided in Table 1 (adapted from Jefferson et al. 2018).

Two other reviews of comparison studies looked specifically at how harms were reported in publications and their regulatory submission counterparts and came to similar conclusions (Hughes et al. 2014; Golder et al. 2016). At the time of writing, the assessment by Golder et al. is more up-to-date and comprehensive (it includes some of the studies already cited), and concluded that “The percentage of adverse events that would have been missed had each analysis relied only on the published versions ranged between 43% and 100%, with a median of 64%.”

In summary, this comprehensive body of evidence documented the presence of distortions by sourcing published records and comparing them in detail with information that is not usually visible. It documented the presence of the bias introduced by sponsors’ attempts at “establishing the commercial value of the article” – to use Sollman’s words written over a century ago.

Although evidence of sponsor bias is now undeniable, however, it does not detail how distortions are generated, and how they are sometimes hidden or misunderstood. I shall illustrate this with three examples with which I am particularly familiar.

Tamiflu

The influenza antiviral Tamiflu (oseltamivir, Roche) is a drug that has been extensively stockpiled as a precaution against a predicted influenza pandemic. In 2010, Roche published a randomised, double blind trial of Tamiflu (WP 16263) in three different strengths, daily for 5 days, versus placebo, in 395 volunteers (Dutkowski et al. 2010). The Methods section of the publication reported that:

The synopsis of the matching 8544-page clinical study report similarly reported:

Synopses of clinical study reports are usually very useful, exhaustive and highly structured, and the content usually coherent with the rest of the document. However when we reviewed the certificates of analysis for oseltamivir, which are part of the regulatory statutory requirements in which manufacturers describe the active intervention and its comparator, we discovered that the capsules containing oseltamivir and its placebo, although of the same size, had different coloured caps. This meant that the trial could not have been double blind, thus raising the likelihood of bias in assessing outcomes. It is difficult to know whether ‘mistakes’ of this kind are simple errors undetected and unremarked upon by regulators. It is clearly not possible for any editor or peer reviewer to identify them either.

The Figures that follow below are the certificates of analysis from Tamiflu trial WP 16263 describing the active capsule and its placebo. The relevant passages have been enlarged and bordered in red.

Gardasil

The quadrivalent Human Papilloma Virus (qHPV) vaccine Gardasil made by Merck was launched to provide protection against papilloma viral infection, a precursor of cancer of the cervix. The publication of pivotal trial V501-020 in the New England Journal of Medicine in 2011 reports the comparator to the vaccine as “AAHS containing placebo”. Reports of the same item from other sources are notable (Table 2). All contain the words “placebo”

Trial Clinical Study Report Register

 

Publication
V501-020 225 mcg of aluminium as AAHS in normal saline No ingredients listed; only states “placebo”

(NCT00090285)

AAHS containing placebo

Giuliano et al 2011

 

 

 

 

 

 

 

Table 2. Reporting format for the control used in Gardasil trial V501-020.

Despite its description as a “placebo”, and being mentioned as many as 50 times in the publication, a placebo has by definition no active ingredient. Merck’s adjuvant Aluminium Hydroxyphosphate Sulphate (AAHS) is neither a placebo nor inactive but is actually a very potent adjuvant contained in the vaccine. Its purpose is to stimulate immunity and maintain a high and prolonged immune response. Its use as a control may mask both harms and differences in effectiveness between arms, something that none of the four data sources report. The manuscript does not explain any of this nor are readers warned of the presence of fragments of DNA in the AAHS mesh, of possible recombinant origin.

The practical consequence of using the Gardasil adjuvant as a control is that the clinical difference being estimated is Gardasil plus adjuvant versus adjuvant alone. Mistakes and misreporting of this type seem unlikely to have occurred by chance and leave readers wondering how regulators and journal editors could have missed the facts, considering that all the pivotal trials of Gardasil had the same comparator. What is needed is a trial comparing Gardasil (whicj contains AAHS adjuvant) with an inactive placebo control.

We have drawn the attention to similar reporting bias in four other major HPV trials (Doshi P et al. 2019).

Statins

The third and last example of sponsor bias comes from the JUPITER trial of the cholesterol lowering drug rosuvastatin. JUPITER tested the effects of the statin in preventing primary cardiovascular events in asymptomatic people with elevated C-reactive protein (Ridker et al. 2008). JUPITER is a significant trial as it was the first trial to test statin use in primary prevention. It is on the basis of the results of trials like JUPITER that the indications for statin use have been expanded to include primary prevention. Although there is little debate about the benefits of statin therapy in those at higher risk of cardiovascular disease, recently attention has shifted to the trade-off between benefits and harms in those prescribed statins for primary prevention. Higher doses of statins are known to be associated with Rhabdomyolysis, leading to renal and respiratory failure, but the debate is now on the uncertainty about rates of less serious harms (especially myalgia and low grade myopathy) in populations at lower risk of cardiovascular disease. Although not immediately life threatening, the impact of these potential harms on quality of life and remaining mobility, especially in frail elderly people, as statins are extended to wider and older populations, make them potentially very important (CTT 2019). The debate swings between the conclusions drawn by the influential Oxford-based Cholesterol Treatment Trialists’ (CTT) Collaboration on the basis of their series of individual participants data (IPD) meta-analysis (Collins et al. 2016) on one side, and evidence from observational studies on the other (Abramson et al.2013, Malhotra 2013). The CTT director insists that the benefits of statin use in primary prevention of cardiovascular disease outweigh their harms (the incidence of which they estimate at 1 in 10,000 users) (Demasi 2018). These observations seem to be contradicted by numerous large surveys, and observational studies report that users quit mainly because of harms.

The original CTT protocol (CTT 1995) made no mention of harms, concentrating instead only on potential benefits (see Table III), while by 2016 the CTT was actively seeking harms data from their trials holdings, as they had based their analyses of IPD mostly on publications.  (CTT 2016). No results are available from this proposed analysis at the time of writing. Bias and distortion lie in originally ignoring harms and in analyzing individual data without reference to the source clinical study reports and especially in not checking the definitions, severity scales and case report forms. This is where such harms would be recorded. Examination of the relevant parts of the JUPITER protocol and blank or model case report form would provide the answer. The JUPITER protocol suggests recording adverse events as follows:


and defines non cardiovascular adverse events as:

Extracts are taken from Astra Zeneca’s protocol for JUPITER.

The publication by Ridker et al. 2008 states that “There were no significant differences between the two study groups with regard to muscle weakness,” although this refers to serious muscular weakness. It is also of note that all specific definitions regarding possible harms are related to cardiovascular endpoints (i.e. benefits). Absence of common methods and definitions across trials are likely to raise questions about the validity of general statements such as those by Ridker et al.

These examples of distortions in designing and reporting trials of these three blockbuster interventions suggest deeply ingrained sponsorship bias in everyday clinical trial science. Whereas the first two examples could be ascribed to Sollman’s “establishing the commercial value of the article”, the statin meta-studies were organised and run by well-respected academic centres in a network of trialists, which is only partially pharma funded. The dangers of not referring to clinical study reports, protocols and manuals of operations has not been highlighted here. It is important to recognise that undetected but plausible adverse effects of statins in the hundreds of millions of people who are now being prescribed these drugs could be causing low grade harms making their lives more difficult.

Tackling sponsorship bias

In any list of priorities for preventing sponsorship bias, disclosure and openness and the avoidance of Sollman’s “secrecy” ranks as the first and most obvious measure. The example and snapshots of regulatory documents presented in this paper did not fall out of the sky. They were the results of many years of work and effort by my colleagues and me. The degree of scrutiny involved in the review of regulatory documents also points to the necessity of quick access (when the intervention has only just been licensed); a sufficient body of researchers willing and capable of doing this work; and the identification of priority topics (perhaps on the basis of cost and potential benefit) on which to concentrate scarce reviewing efforts.

Legislation would probably be necessary to enable early access by accredited groups worldwide to regulatory submissions and to open up the editorial process completely to the use of regulatory material.

Most of these measures were proposed by Garattini and Chalmers more than a decade ago, with little political engagement (Garattini and Chalmers 2009). In 1968, at US Congressional Hearings, the statistician Donald Mainland suggested that Congress, “take the evaluation of drugs entirely out of the producer’s hands,” after the completion of toxicological testing on animals” (Altman 2017). Ultimately, separation of those who are keen on establishing the value of healthcare interventions for health and those who promote the commercial worth of interventions will be the only efficient way protecting the interests of the public.

Disclosure:
Tom Jefferson received a fee from the James Lind Initiative for preparing this article. Tom Jefferson is funded by the Nordic Cochrane Centre and NIHR for his work on statins. Other disclosures are here.

Acknowledgements
The Dutch Medicines Evaluation Board for providing the JUPITER trial clinical study report.

This James Lind Library article has been republished in the Journal of the Royal Society of Medicine 2020;113:148-157. Print PDF

References

Abramson JD, Rosenberg HG, Jewell N, Wright JM (2013). Should people at low risk of cardiovascular disease take a statin? BMJ 347:f6123.

Altman DG (2017). Donald Mainland: anatomist, educator, thinker, medical statistician, trialist, rheumatologist. JLL Bulletin: Commentaries on the history of treatment evaluation (http://www.jameslindlibrary.org/articles/donald-mainland-anatomist-educator-thinker-medical-statistician-trialist-rheumatologist/  )

Chan A-W, Hròbjartsson A, Haahr M, Gøtzsche PC, Altman DG (2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to publications. JAMA 291:2457-2465.

Cholesterol Treatment Trialists’ (CTT) Collaboration (1995). Protocol for a prospective collaborative overview of all current and planned randomized trials of cholesterol treatment regimens. Am J Cardiol 75:1130-1134.

Cholesterol Treatment Trialists’ (CTT) Collaboration (2016). Protocol for analyses of adverse event data from randomized controlled trials of statin therapy. Am Heart J 176:63-9.

Cholesterol Treatment Trialists’ (CTT) Collaboration (2019). Efficacy and safety of statin therapy in older people: a meta-analysis of individual participant data from 28 randomised controlled trials. Lancet 393:407–15.

Collins R, Reith C, Emberson J, Armitage J, Baigent C, Blackwell L, Blumenthal R, Danesh J, Smith GD, DeMets D, Evans S, Law M, MacMahon S, Martin S, Neal B, Poulter N, Preiss D, Ridker P, Roberts I, Rodgers A, Sandercock P, Schulz K, Sever P, Simes J, Smeeth L, Wald N, Yusuf S, Peto R (2016). Interpretation of the evidence for the efficacy and safety of statin therapy. Lancet 388 :2532-61. doi: https://doi.org/10.1016/S0140-6736(16)31357-5 https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(16)31357-5/fulltext

Cosgrove L, Vannoy S, Mintzes B, Shaughnessy AF (2016). Under the Influence: The Interplay among Industry, Publishing, and Drug Regulation. Accountability in research 23:257-279. 23. 10.1080/08989621.2016.1153971.

Coyne DW (2012). The health-related quality of life was not improved by targeting higher hemoglobin in the Normal Hematocrit Trial. Kidney International 82:235-41.

Dickersin K, Chalmers I (2010). Recognising, investigating and dealing with incomplete and biased reporting of clinical research: from Francis Bacon to the World Health Organisation. JLL Bulletin: Commentaries on the history of treatment evaluation (https://www.jameslindlibrary.org/articles/recognising-investigating-and-dealing-with-incomplete-and-biased-reporting-of-clinical-research-from-francis-bacon-to-the-world-health-organisation/).

Dickersin K, Min YI, Meinert CL (1992). Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA 267:374-8.

Demasi M (2018). Statin wars: have we been misled about the evidence? A narrative review. Br J Sports Med 52:905-909. doi:10.1136/bjsports-2017-098497.

Doshi P, Jefferson T (2013). Clinical study reports of randomised controlled trials: an exploratory review of previously confidential industry reports. BMJ Open 26;3(2):e002496.

Doshi P, Bourgeois F, Hong K, Jones M, Lee H, Shamseer L, Spence O, Jefferson T (2019) Additional trials within scope: a follow-up to our “Call to action: RIAT restoration of a previously unpublished methodology in Gardasil vaccine trials”. https://www.bmj.com/content/346/bmj.f2865/rr-9

Dutkowski R, Smith JR, Davies BE (2010). Safety and pharmacokinetics of oseltamivir at standard and high dosages. International Journal of Antimicrobial Agents 35:461–7.

Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR (1991). Publication bias in clinical research. Lancet 37:867-72.

Eyding D, Lelgemann M, Grouven U, Harter M, Kromp M, Kaiser T, Kerekes MF, Gerken M, Wieseler B (2010). Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ 341:c4737.

Fu R, Selph S, McDonagh M, Peterson K, Tiwari A, Chou R, Helfand M (2013). Effectiveness and harms of recombinant human bone morphogenetic protein-2 in spine fusion: a systematic review and meta-analysis. Annals of Internal Medicine 158:890-902.

Garattini S, Chalmers I (2009). Patients and the public deserve big changes in evaluation of drugs BMJ 338:b1025.

Giuliano AR, Palefsky JM, Goldstone S, Moreira ED Jr, Penny ME, Aranda C, Vardas E, Moi H, Jessen H, Hillman R, Chang YH, Ferris D, Rouleau D, Bryan J, Marshall JB, Vuocolo S, Barr E, Radley D, Haupt RM, Guris D. (2011). Efficacy of quadrivalent HPV vaccine against HPV infection and disease in males. N Engl J Med 364:401–11.

Goldacre B, DeVito NJ, Heneghan C, Irving F, Bacon S, Fleminger J, Curtis (2018). Compliance with requirement to report results on the EU Clinical Trials Register: cohort study and web resource. BMJ 362:k3218. https://www.bmj.com/content/bmj/362/bmj.k3218.full.pdf

Golder S, Loke YK, Wright K, Norman G (2016). Reporting of adverse events in published and unpublished studies of health care interventions: a systematic review. PLoS Med 13(9): e1002127. doi:10.1371/journal.pmed.1002127.

Gøtzsche PC, Jørgensen AW (2011). Opening up data at the European Medicines Agency. BMJ 342:d2686.

Hemminki H (1980). Study of information submitted by drug companies to licensing authorities. BMJ 280:833-6.

Hodkinson A, Gamble C, Smith CT (2016). Reporting of harms outcomes: a comparison of journal publications with unpublished clinical study reports of orlistat trials. Trials 17:207.

Hughes S, Cohen D, Jaggi R. Differences in reporting serious adverse events in industry sponsored clinical trial registries and journal articles on antidepressant and antipsychotic drugs: a cross-sectional study. BMJ Open. 2014;4:e005535. doi:10.1136/bmjopen-2014-005535.

Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, Hama R, Thompson MJ (2012). Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database of Systematic Reviews Issue 1. Art. No.: CD008965. DOI: 10.1002/14651858.CD008965.pub3.

Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R, Thompson MJ Spencer EA, Onakpoya I, Mahtani KR, Nunan D, Howick J, Heneghan CJ (2014). Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database of Systematic Reviews Issue 4. Art. No.: CD008965 DOI: 10.1002/14651858.CD008965.pub4.

Jefferson T, Doshi P, Boutron I, Golder S, Heneghan C, Hodkinson A, Jones M, Lefebvre C, Stewart LA (2018). When to include clinical study reports and regulatory documents in systematic reviews. BMJ Evidence-Based Medicine 10.1136/bmjebm-2018-110963. https://ebm.bmj.com/content/ebmed/23/6/210.full.pdf

JUPITER (Justification for the Use of statins in Primary prevention: an Intervention Trial Evaluating Rosuvastatin). A randomized, double-blind, placebo controlled, multicenter, Phase III Study of rosuvastatin (CRESTOR) 20 mg in the primary prevention of cardiovascular events among subjects with low levels of LDL-cholesterol and elevated levels of C-Reactive Protein. JUPITER trial protocol (2003 and 2006 versions).

Jureidini J, Amsterdam J, McHenry L (2016). The citalopram CIT-MD-18 pediatric depression trial: deconstruction of medical ghostwriting, data mischaracterisation and academic malfeasance. International Journal of Risk & Safety in Medicine 28:33-43.

Köhler M, Haag S, Biester K, Brockhaus AC, McGauran N, Grouven U, Kölsch H, Seay U, Hörn H, Moritz G, Staeck K, Wieseler B (2015). Information on new drugs at market entry: retrospective analysis of health technology assessment reports versus regulatory reports, journal publications, and registry reports. BMJ. 350:h796. doi: https://doi.org/10.1136/bmj.h796. https://www.bmj.com/content/bmj/350/bmj.h796.full.pdf

Lawrence K, Beaumier J, Wright J, Perry T, Puil L Turner E, Mintzes B (2015). Olanzapine for schizophrenia: what do the unpublished clinical trials reveal? Presented at the 23rd Cochrane Colloquium, Vienna, October 2015 (presentation 03.1). https://abstracts.cochrane.org/2015-vienna/olanzapine-schizophrenia-what-do-unpublished-clinical-trials-reveal

Malhotra A (2013). Saturated fat is not the major issue BMJ 347:f6340

Mann H, Djulbegovic B (2012). Comparator bias: why comparisons must address genuine uncertainties. JLL Bulletin: Commentaries on the history of treatment evaluation (http://www.jameslindlibrary.org/articles/comparator-bias-why-comparisons-must-address-genuine-uncertainties/)

Maund E, Tendal B, Hróbjartsson A, Jørgensen KJ, Lundh A, Schroll J, Gøtzsche PG (2014). Benefits and harms in clinical trials of duloxetine for treatment of major depressive disorder: comparison of clinical study reports, trial registries, and publications. BMJ 348:g3510 doi: 10.1136/bmj.g3510.

Mayo-Wilson E, Li T, Fusco N, Bertizzolo L, Canner JK, Cowley T, Doshi P, Ehmsen J, Gresham G, Guo N, Haythornthwaite JA, Heyward J, Hong H, Pham D, Payne JL, Rosman L, Stuart EA, Suarez-Cuervo C, Tolbert E, Twose C, Vedula S, Dickersin K (2017). Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy. J Clin Epidemiol 91:95–110.

Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B (2003). Evidence b(i)ased medicine – selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ 326:1171-3. https://www.bmj.com/content/326/7400/1171.full.pdf+html)

Psaty BM, Kronmal RA (2008). Reporting mortality findings in trials of Rofecoxib for Alzheimer disease or cognitive impairment. JAMA 299:1813-1817.

Ridker PM, Danielson E, Fonseca FAH Genest J, Gotto AM Jr, Kastelein JJ, Koenig W, Libby P, Lorenzatti AJ, MacFadyen JG, Nordestgaard BG, Shepherd J, Willerson JT, Glynn RJ; JUPITER Study Group (2008). Rosuvastatin to prevent vascular events in men and women with elevated C-Reactive Protein. New England Journal of Medicine 359:2195-207.

Rising K, Bacchetti P, Bero L (2008). Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Med 5(11): e217. doi:10.1371/journal.pmed.0050217

Rodgers MA, Brown JVE, Heirs MK, Higgins JPT, Mannion RJ, Simmonds MA, Stewart LA (2013). Reporting of industry funded study outcome data: comparison of confidential and published data on the safety and effectiveness of rhBMP-2 for spinal fusion. BMJ 346: f3981.

Schroll JB, Penninga EI, Gøtzsche PC (2016). Assessment of adverse events in protocols, clinical study reports, and published papers of trials of orlistat: a document analysis. PLoS Med 13:e1002101.

Sollmann T (1917). The crucial test of therapeutic evidence. JAMA 69:198-199. (

Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine 358:252-60.

Vedula SS, Li T, Dickersin K (2013). Differences in reporting of analyses in internal company documents versus published trial reports: comparisons in industry-sponsored trials in off-label uses of gabapentin. PLoS Medicine 10:e1001378.

Wieseler B, Kerekes MF, Vervoelgyi V, McGauran N, Kaiser T (2012). Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications. BMJ 344:d8141.

Wieseler B, Wolfram N, McGauran N, Kerekes MF, Vervölgyi V, Kohlepp P, Kamphuis M, Grouven U (2013). Completeness of reporting of patient-relevant clinical trial outcomes: comparison of unpublished clinical study reports with publicly available data. PLoS Medicine 10:e1001526. doi:10.1371/journal.pmed.1001526.

WP16263. A randomized, double blind, parallel group, placebo-controlled study of the effect of oseltamivir on ECG intervals in healthy subjects. Available from: https://datadryad.org/resource/doi:10.5061/dryad.77471.