United States

&lt;p&gt;Ayu Putu Madri Dewi,&lt;sup&gt;1&lt;/sup&gt; Nicholas J. DeVito,&lt;sup&gt;2&lt;/sup&gt; Gowri Gopalakrishna,&lt;sup&gt;1,3&lt;/sup&gt; Inge Stegeman,&lt;sup&gt;4,5&lt;/sup&gt; Mariska Leeflang&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective&lt;/h4&gt;
&lt;p&gt; To improve research reproducibility, referees of grant proposals could assess the expected reproducibility of a proposed project. A simple Open Science checklist may assist referees in assessing whether referees are more capable of predicting the reproducibility of research proposals with an Open Science checklist than without one.&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Design&lt;/h4&gt;
&lt;p&gt; This is a nonrandomized controlled trial and a comparative prediction study&lt;sup&gt; &lt;/sup&gt;using a minimum of 100 granted research proposals and their corresponding published outputs (α = 5%, power = 80%, correlation = 0.6).&lt;sup&gt;2&lt;/sup&gt; The study focuses on empirical research with quantitative data across diverse fields. A mock grant review was started on April 7, 2025, with a minimum of 4 referees divided into 2 groups: one using an Open Science checklist and the other not. Each proposal will be assessed under both conditions (&lt;b&gt;Figure 25-0934&lt;/b&gt;). The checklist, developed in a previous study&lt;sup&gt;3&lt;/sup&gt; and adapted for grant proposal evaluation, serves as the intervention. Referees are blinded to final study outcomes to minimize bias. The primary outcome is predictive accuracy, assessed using the area under the curve (AUC) from receiver operating characteristic (ROC) analysis. Referees’ predictions on the actual reproducibility of the published studies will be compared. Open Science scores are numerical (0-14), while reproducibility outcomes are binary (yes or no). The AUC from logistic regression will be used and visualized in ROC curves. Comparative predictive accuracy will be calculated for both groups. Sensitivity analyses will explore variations in checklist scoring and study reproducibility across different research fields. A database including grant proposal data, Open Science scores, reproducibility status, and analytical code in R will be shared in an open repository. We hypothesize that referees without the checklist will predict reproducibility no better than random chance (with an AUC of approximately 0.50), while those using the checklist will achieve an AUC of approximately 0.65, indicating improvement while acknowledging that the checklist is not a perfect tool.&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/17200ac9fe2e7dc26b84dba55df1facc.png)

&lt;h4&gt;Results&lt;/h4&gt;
&lt;p&gt; As of June 5, 2025, a total of 15 mock grant reviewers have completed 89 mock reviews across 60 unique grant proposals: 46 reviews in the checklist group and 43 in the control group. Twenty-nine proposals were reviewed in both groups, forming a fully paired subset. Analysis is ongoing.&lt;/p&gt;
&lt;h4&gt;Conclusions&lt;/h4&gt;
&lt;p&gt; We will share our results with funding agencies to explore integrating the Open Science checklist into grant review processes to enhance reproducibility.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Parker TH, Griffith SC, Bronstein JL, et al. Empowering peer reviewers with a checklist to improve transparency. &lt;i&gt;Nat Ecol Evol&lt;/i&gt;. 2018;2(6):929-935. doi:10.1038/s41559-018-0545-z&lt;/p&gt;
&lt;p&gt;2. Fraser H, Bush M, Wintle BC, et al. Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process. &lt;i&gt;PLoS One&lt;/i&gt;. 2023;18(1):e0274429. doi:10.1371/journal.pone.0274429&lt;/p&gt;
&lt;p&gt;3. Dewi APM, Rethlefsen M, Schroter S, et al. Measuring the efficacy of an intervention to improve reproducibility of scientific manuscripts: study protocol for a randomized controlled trial. OSFHOME. October 31, 2024. Accessed July 2, 2025. &lt;a href=&quot;https://osf.io/b5g6y&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://osf.io/b5g6y&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Amsterdam UMC, Epidemiology and Data Science Department, Amsterdam, The Netherlands; &lt;sup&gt;2&lt;/sup&gt;Bennett Institute for Applied Data Science, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK, nicholas.devito@phc.ox.ac.uk; &lt;sup&gt;3&lt;/sup&gt;Department of Epidemiology, Faculty of Health, Medicine, and Life Sciences Maastricht University, The Netherlands; &lt;sup&gt;4&lt;/sup&gt;Department of Otorhinolaryngology and Head &amp;amp; Neck Surgery University Medical Center Utrecht, The Netherlands; &lt;sup&gt;5&lt;/sup&gt;Brain Center, University Medical Center Utrecht, The Netherlands.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures &lt;/h4&gt;
&lt;p&gt;None reported.&lt;/p&gt;


The 10th Peer Review Congress

Perception of Open Science Practices on Reproducibility Among Reviewers of Grant Proposals at Research Funding Organizations

funding/grant peer review

open science

reproducibility

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Ayu Putu Madri Dewi,<sup>1</sup> Nicholas J. DeVito,<sup>2</sup> Gowri Gopalakrishna,<sup>1,3</sup> Inge Stegeman,<sup>4,5</sup> Mariska Leeflang<sup>1</sup></p>
<h4>Objective</h4>
<p> To improve research reproducibility, referees of grant proposals could assess the expected reproducibility of a proposed project. A simple Open Science checklist may assist referees in assessing whether referees are more capable of predicting the reproducibility of research proposals with an Open Science checklist than without one.<sup>1</sup></p>
<h4>Design</h4>
<p> This is a nonrandomized controlled trial and a comparative prediction study<sup> </sup>using a minimum of 100 granted research proposals and their corresponding published outputs (α = 5%, power = 80%, correlation = 0.6).<sup>2</sup> The study focuses on empirical research with quantitative data across diverse fields. A mock grant review was started on April 7, 2025, with a minimum of 4 referees divided into 2 groups: one using an Open Science checklist and the other not. Each proposal will be assessed under both conditions (<b>Figure 25-0934</b>). The checklist, developed in a previous study<sup>3</sup> and adapted for grant proposal evaluation, serves as the intervention. Referees are blinded to final study outcomes to minimize bias. The primary outcome is predictive accuracy, assessed using the area under the curve (AUC) from receiver operating characteristic (ROC) analysis. Referees’ predictions on the actual reproducibility of the published studies will be compared. Open Science scores are numerical (0-14), while reproducibility outcomes are binary (yes or no). The AUC from logistic regression will be used and visualized in ROC curves. Comparative predictive accuracy will be calculated for both groups. Sensitivity analyses will explore variations in checklist scoring and study reproducibility across different research fields. A database including grant proposal data, Open Science scores, reproducibility status, and analytical code in R will be shared in an open repository. We hypothesize that referees without the checklist will predict reproducibility no better than random chance (with an AUC of approximately 0.50), while those using the checklist will achieve an AUC of approximately 0.65, indicating improvement while acknowledging that the checklist is not a perfect tool.</p>

![](https://assets.underline.io/markdown_image/1/image/17200ac9fe2e7dc26b84dba55df1facc.png)

<h4>Results</h4>
<p> As of June 5, 2025, a total of 15 mock grant reviewers have completed 89 mock reviews across 60 unique grant proposals: 46 reviews in the checklist group and 43 in the control group. Twenty-nine proposals were reviewed in both groups, forming a fully paired subset. Analysis is ongoing.</p>
<h4>Conclusions</h4>
<p> We will share our results with funding agencies to explore integrating the Open Science checklist into grant review processes to enhance reproducibility.</p>
<h4>References</h4>
<p>1. Parker TH, Griffith SC, Bronstein JL, et al. Empowering peer reviewers with a checklist to improve transparency. <i>Nat Ecol Evol</i>. 2018;2(6):929-935. doi:10.1038/s41559-018-0545-z</p>
<p>2. Fraser H, Bush M, Wintle BC, et al. Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process. <i>PLoS One</i>. 2023;18(1):e0274429. doi:10.1371/journal.pone.0274429</p>
<p>3. Dewi APM, Rethlefsen M, Schroter S, et al. Measuring the efficacy of an intervention to improve reproducibility of scientific manuscripts: study protocol for a randomized controlled trial. OSFHOME. October 31, 2024. Accessed July 2, 2025. <a href="https://osf.io/b5g6y"><span class="Hyperlink CharOverride-6">https://osf.io/b5g6y</span></a></p>
<sup>1</sup>Amsterdam UMC, Epidemiology and Data Science Department, Amsterdam, The Netherlands; <sup>2</sup>Bennett Institute for Applied Data Science, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK, nicholas.devito@phc.ox.ac.uk; <sup>3</sup>Department of Epidemiology, Faculty of Health, Medicine, and Life Sciences Maastricht University, The Netherlands; <sup>4</sup>Department of Otorhinolaryngology and Head &amp; Neck Surgery University Medical Center Utrecht, The Netherlands; <sup>5</sup>Brain Center, University Medical Center Utrecht, The Netherlands.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>


poster

<p>Kyungwan Hong,<sup>1</sup> Mark Basista,<sup>1</sup> Tony Tse<sup>1</sup></p>
<h4>Objective </h4>
<p>Information about how adverse events (AEs) were specified, collected, and assessed for clinical trials (ie, AE methods) is essential to accurately interpret the safety profiles of studied interventions.<sup>1</sup> ClinicalTrials.gov provides a structured format for trialists to report aggregate results information and AE methods that complement the published literature. Prior work compared reporting of AEs in publications and ClinicalTrials.gov in terms of completeness and consistency,<sup>2</sup> examined differences between sources in reporting AE methods for systematic collection,<sup>3</sup> and assessed AE methods reporting in publications and identified methodologic weaknesses.<sup>1</sup> This cross-sectional analysis explored the extent that AE methods reported in ClinicalTrials.gov complemented corresponding publications.</p>
<h4>Design </h4>
<p>ClinicalTrials.gov was searched on August 1, 2024, for drug and biologic trials with primary completion dates between January 18, 2017, and August 1, 2021, to allow 3 years for results posting and publication. A total of 9659 records were initially retrieved. After randomization, the records were reviewed sequentially until a convenience sample of 40 trials with summary results posted on ClinicalTrials.gov and corresponding primary results articles in PubMed were identified. The identified trials were then analyzed. All authors reviewed and extracted text for 5 ClinicalTrials.gov AE method data elements: collection period (time frame), vocabulary source for AE terms (how AE terms were standardized), collection approach (systematic or nonsystematic), analysis population, and whether participants withdrew due to harms. Each set of trial results was independently reviewed by 2 authors, with discrepancies resolved by consensus.</p>
<h4>Results </h4>
<p>Collection period (28 [70.0%]), analysis population (27 [67.5%]), and withdrawal for harm (35 [87.5%]) were described for over half of the 40 sampled trials in ClinicalTrials.gov and/or publications (<b>Table 24-0831</b>). However, consistency varied; for example, of the 35 trials with number of participants withdrawn for harm described in both sources, 20 (57.1%) provided inconsistent information. Collection approach was described in both sources infrequently (4 [10.0%]). Other AE methods (eg, vocabulary source) were reported for some trials in only one or neither source. </p>

![](https://assets.underline.io/markdown_image/1/image/570e4b44fadda46a96c1215d9717877c.png)

<h4>Conclusions </h4>
<p>A few AE methods were not reported on ClinicalTrials.gov and/or publications, indicating a need for improvement. Complete AE methods reporting is needed for accurate safety profiles of study interventions and unbiased risk-benefit assessments. Even when available, reported AE methods were frequently inconsistent between sources, potentially due to variations in time frames used for AE reporting. Limitations included excluding supplementary appendices and study protocols that may provide additional methodologic details on AE methods. Future research is needed to review publicly available study protocols for AE methods information, compare the quality of AE methods reporting by source, and characterize trials in which AE methods have and have not been completely reported (eg, by funding source, study design).</p>
<h4>References</h4>
<p>1. Phillips R, Hazell L, Sauzet O, Cornelius V. Analysis and reporting of adverse events in randomised controlled trials: a review. <i>BMJ Open</i>. 2019;9(2):e024537. doi:10.1136/bmjopen-2018-024537</p>
<p>2. Krešo A, Grahovac M, Znaor L, Marušić A. Safety reporting in trials on glaucoma interventions registered in ClinicalTrials.gov and corresponding publications. <i>Sci Rep</i>. 2024;14(1):27762. doi:10.1038/s41598-024-79394-z</p>
<p>3. Mayo-Wilson E, Fusco N, Li T, Hong H, Canner JK, Dickersin K; MUDS Investigators. Harms are assessed inconsistently and reported inadequately part 1: systematic adverse events. <i>J Clin Epidemiol</i>. 2019;113:20-27. doi:10.1016/j.jclinepi.2019.04.022</p>
<sup>1</sup>National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, MD, US, kyungwan.hong@nih.gov.</p>
<h4>Conflict of interest Disclosures </h4>
<p>Kyungwan Hong reported being employed by the National Institutes of Health (NIH) as a policy analyst with the ClinicalTrials.gov program. Mark Basista reported conducting this work while contracting with ICF. Tony Tse reported being employed by the NIH as an analyst with the ClinicalTrials.gov program. No other disclosures were reported.</p>
<h4>Funding/Support</h4>
<p> This work was supported by the National Center for Biotechnology Information of the National Library of Medicine, NIH.</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funder was not directly involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the abstract for presentation.</p>
<h4>Disclaimer </h4>
<p>The views expressed in this article are those of the authors and do not necessarily reflect the views or policies of the NIH.</p>


Characterizing Adverse Event Methods Reported in ClinicalTrials.gov and Publications

<p>Randall J. Ellis,<sup>1</sup> Chirag J. Patel<sup>1</sup></p>
<h4>Objective </h4>
<p>New omics modalities hold promise for biomarker discovery for disease prediction. Accessible biobank samples have fueled a massive literature, some of which may be irreproducible, be subject to lack of generalizability, or report inflated results due to data leakage,<sup>1</sup> ie, spurious relationships between input and target variables that arise as artifacts of data collection, sampling, or preprocessing. These biases frequently result in models developed in one context not generalizing to real-world contexts. Here, we demonstrated and quantified bias in biomedical multiomic machine learning results due to irrelevant or unreported baselines, data leakage, and inadequate performance metrics reporting.</p>
<h4>Design </h4>
<p>In November 2024, using data from the UK Biobank, we conducted a cohort study of all 607 disease outcomes defined in the dataset, using blood plasma proteomics data (collected from 2006-2010) and demographics data. We demonstrated how simple demographics (age, sex, education) compare competitively with putatively novel models that incorporate an expansive number of omics input features, how randomly chosen omics features compare with those chosen according to data-driven feature selection methods, the impact of data leakage on performance (eg, normalizing the data before making train-test cross-validation splits), and how presenting the area under the receiver operating characteristic curve (AUROC) gives a biased view of performance. We presented additional case studies of using baseline risk calculators for cardiovascular disease and atherosclerotic cardiovascular disease to assess how omics factors compared with traditional risk scores for associations with 10-year risk of heart disease or stroke. We used a stratified cross-validation approach and assessed the AUROC, sensitivity, specificity, and positive and negative predictive value.</p>
<h4>Results </h4>
<p>Across a majority of 607 disease outcomes, demographic baselines performed competitively (0%-10% difference in mean AUROC) in comparison with the combination of demographics and omics factors when looking at the variability of performance across cross-validation folds, which are underreported in the literature. We found that data leakage influenced predictive performance (mean AUROC increases of 5%-30%). Accounting for disease prevalence and class balance played a significant role in the interpretation of machine learning results, particularly for positive predictive value. Omics factors contributed marginal increases in performance for cardiovascular disease and atherosclerotic cardiovascular disease compared with traditional risk score calculators (less than 5% mean AUROC).</p>
<h4>Conclusions </h4>
<p>The robustness and transparency of biobank-based omics research is enhanced if several practices are followed: (1) rigorous and relevant baselines would provide critical tests of candidate models to justify their clinical use and value, (2) preventing data leakage would preclude inflated results, and (3) comprehensive performance metrics reporting would give transparent measures of progress for researchers in the omics community. Our results complement other work showing the impact of data leakage<sup>1</sup> and the marginal improvements of plasma proteomics beyond demographic baselines.<sup>2</sup></p>
<h4>References</h4>
<p>1. Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. <i>Patterns</i>. 2023;4(9):100804. doi:10.1016/j.patter.2023.100804</p>
<p>2. Deng YT, You J, He Y, et al. Atlas of the plasma proteome in health and disease in 53,026 adults. <i>Cell</i>. 2025;188(1):253-271. doi:10.1016/j.cell.2024.10.045</p>
<sup>1</sup>Department of Biomedical Informatics, Harvard Medical School, Boston, MA, US, chirag_patel@hms.harvard.edu.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>
<h4>Funding/Support </h4>
<p>This work was supported by grants from the National Institute on Aging (RF1AG074372) and National Library of Medicine (5T15LM007092-33).</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funders were not involved in the development or implementation of this study.</p>
<h4>Acknowledgment </h4>
<p>We acknowledge the support of our funders and Audrey Airaud for early-stage input on these experiments.</p>


Bias in Machine Learning Associated With Weak Baselines, Data Leakage, and Inadequate Measures Reporting

<p>Edith Otalike,<sup>1</sup> Mike Clarke,<sup>2</sup> Ngianga-Bakwin Kandala,<sup>1</sup> Joel J. Gagnier<sup>1,3</sup></p>
<h4>Objective</h4>
<p> Individual-participant data meta-analysis (IPD-MA) is regarded as the criterion standard in evidence synthesis, but it is resource intensive. While there is consensus on reporting items, a consensus-based tool for the critical appraisal of the methodological quality of IPD-MA does not currently exist. We undertook a systematic methodology review as the initial phase in the development of a critical appraisal checklist. This review collated and summarized the available methodological guidance on IPD-MA of randomized and observational studies. </p>
<h4>Design</h4>
<p> We followed the guidelines for Cochrane Methodology Reviews and reported following the PRISMA 2020 guidance. We performed an electronic search of MEDLINE, Embase, CINAHL, Web of Science Scopus, Cochrane Methodology Registry, CONSORT Database of Methodological Papers, Health Technology Assessment Review Database, <i>Research Synthesis Methods</i>, and <i>Journal of the Royal Statistical Society</i> covering publications from 1946 to June 2024. We included studies published in English that addressed any methodological guidance and essential statistical and software requirements for IPD-MA. Data extraction focused on study characteristics, domain of the review process, and the specific recommendations. Risk of bias was assessed using resources relevant to the study design. A thematic synthesis was performed to group recurring themes into domains. For each domain, signalling questions were generated to develop a preliminary checklist for assessment and refinement in a 2-round e-Delphi survey involving international IPD-MA experts.</p>
<h4>Results</h4>
<p> The literature search yielded 13,589 citations. After screening 9436 unique abstracts and reviewing 286 full texts, we included 130 articles that met our inclusion criteria. These articles consisted of narrative reviews, handbooks, critical reviews, empirical studies, and statistical method articles. They were published between 1995 and 2024, with most originating from the UK (62 [48%]), the US (20 [15%]), and the Netherlands (16 [12%]) and 50 (38%) originating from 10 other countries. Most of these studies had a low risk of bias. We identified 14 domains of guidance for conducting and reporting of IPD-MA, and we categorized them into 5 sections (<b>Table 25-0999</b>). This finding informed the initial version of the checklist to be evaluated in the e-Delphi survey.</p>

![](https://assets.underline.io/markdown_image/1/image/ae40b29d7f5b93606f06cb13df104504.png)

<h4>Conclusions</h4>
<p> There are many recommendations in the literature on the general conduct of IPD-MA and on specific aspects of this research, which would benefit from consensus recommendations for all aspects of IPD-MA and critical appraisal of reports. This review provides many suggestions for these recommendations, and our e-Delphi survey will seek consensus on items to include in the critical appraisal tool, which we expect to be completed in 2025.</p>
<sup>1</sup>Department of Epidemiology &amp; Biostatistics, Schulich School of Medicine &amp; Dentistry, Western University, London, Ontario, Canada, eotalike@uwo.ca; <sup>2</sup>Northern Ireland Methodology Hub, Queen’s University Belfast, Northern Ireland, UK; <sup>3</sup>Department of Surgery, Schulich School of Medicine &amp; Dentistry, Western University, London, Ontario, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Edith Otalike receives internal funding from Western University and the Dean’s Research Scholarship award. No other disclosures were reported.</p>
<h4>Additional Information</h4>
<p> Joel J. Gagnier is a co–corresponding author (jgagnie4@uwo.ca).</p>


Individual-Participant Data Meta-Analysis Methodological Guidance: A Systematic Review

<p>Luis C. L. Correia,<sup>1</sup> Rafael F. Mascarenhas,<sup>2</sup> Felipe S. C. de Menezes,<sup>2</sup> Jeronimo S. Oliveira Júnior,<sup>2</sup> Marcus V. Almeida,<sup>2</sup> Caio F. Azevedo,<sup>2</sup> Naieli M. de Andrade,<sup>2</sup> Viola Vaccarino,<sup>1</sup> Joseph S. Ross,<sup>3</sup>< Joshua D. Wallach<sup>1</sup></p>
<h4>Objective</h4>
<p> Several approaches exist for selecting confounders for adjustment in observational studies, including subjective judgment, statistical criteria, or the use of a causal model, usually represented by directed acyclic graphs (DAGs). We assessed the characteristics and trends in the reported methods for selecting confounders to control for in observational studies published in the highest impact factor medical and epidemiological journals.</p>
<h4>Design </h4>
<p>We identified the 10 highest Impact Factor medical (n = 5) and epidemiological (n = 5) journals according to InCites Journal Citation Reports. For each journal, we reviewed all PubMed-indexed articles published in 2003, 2013, and 2023 to identify observational studies evaluating exposure-outcome relationships in which confounder adjustment would be expected. We excluded articles that were descriptive, predictive, and quasi-experimental (ie, that primarily address confounding through design-based approaches). We randomly selected half of the articles in each journal and publication year for full-text evaluation and identified key study characteristics. We classified the methods reported by each study to select confounders: no confounder adjustment; adjustment but confounders not specified; confounders selected without justification; confounders selected based on an established association with the outcome; confounders selected based on statistical criteria (ie, imbalance between exposure groups, change-in-estimate strategy, or application of a stepwise regression); or confounders selected based on a causal model, either depicted by a DAG or explained in the text. We followed the STROBE reporting guideline for cross-sectional studies.</p>
<h4>Results</h4>
<p> We identified 623 eligible observational studies, including 197 (31.6%) published in medical journals and 426 (68.4%) published in epidemiological journals. Of these, 22 (3.5%) did not report adjusting for confounders, 18 (2.9%) did not specify which confounders were selected, 281 (45.1%) reported selection of confounders without justification, 139 (22.3%) reported selection of confounders based on an established association with the outcome, 121 (19.4%) reported selection of confounders based on statistical criteria, and 42 (6.7%) reported selection of confounders based on a causal model (35 used a DAG and 7 provided an explanation in their text) (<b>Table 25-1006</b>). The selection of confounders without justification remained relatively stable between 2003 and 2023 (from 111 of 228 [48.7%] to 68 of 164 [41.5%]), while the use of a causal model to identify confounders increased (from 0 of 228 to 37 of 164 [22.6%]). Differences in the methods used to select confounders were observed across journal type and study design, but not exposure type or funding source.</p>

![](https://assets.underline.io/markdown_image/1/image/b3e83419c11953083cbf73a2ad1b02b7.png)

<h4>Conclusions</h4>
<p> Although our analysis was limited to high-impact medical and epidemiological journals, we found that the reporting of causal models—such as DAGs—has increased over time. However, fewer than one-fourth of studies reported such models in 2023, raising concerns about how confounders are selected and justified in observational research.</p>
<sup>1</sup>Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, GA, US, joshua.wallach@emory.edu; <sup>2</sup>Bahiana School of Medicine and Public Health, Salvador, BA, Brazil; <sup>3</sup>Department of Medicine, Yale School of Medicine, New Haven, CT, US.</p>
<h4>Conflicts of Interest</h4>
<p> Joseph S. Ross reported receiving grants from the US Food and Drug Administration; Johnson &amp; Johnson; the Medical Device Innovation Consortium; the Agency for Healthcare Research and Quality; the National Heart, Lung, and Blood Institute; and Arnold Ventures outside the submitted work and is also is an expert witness at the request of relator attorneys, the Greene Law Firm, in a qui tam suit alleging violations of the False Claims Act and Anti-Kickback Statute against Biogen Inc that was settled in September 2022. Dr Ross is a Deputy Editor at <i>JAMA</i>. Joshua D. Wallach is supported by Arnold Ventures, Johnson &amp; Johnson through the Yale Open Data Access project, and the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under award 1K01AA028258 and previously served as a consultant to Hagens Berman Sobol Shapiro LLP and Dugan Law Firm APLC. Dr Wallach is a Specialty Associated Editor at <i>JACC</i>.</p>


Reporting of Confounder Selection in Observational Studies in High Impact Medical and Epidemiological Journals, 2003-2023

<p>Ashkan Hashemi,<sup>1</sup> Isaac Dreyfus,<sup>2</sup> Nicholas Varunok,<sup>3</sup> John Burton,<sup>4</sup> Sina Rashedi,<sup>5</sup> Farbod Zahedi Tajrishi,<sup>6</sup> Behnood Bikdeli<sup>5</sup></p>
<h4>Objective </h4>
<p>The role of noninferiority trials in cardiovascular research remains unclear. We sought to determine the proportion of noninferiority vs superiority trials, their likelihood of meeting primary outcomes, and their academic impact and online engagement.</p>
<h4>Design</h4>
<p> We reviewed cardiovascular randomized controlled trials (RCTs) published in <i>The New England Journal of Medicine</i>, <i>JAMA</i>, and <i>The Lancet</i> between January 2014 and December 2019. Cardiovascular trials were defined based on review by 2 independent reviewers for topics that included patients with cardiovascular diseases or tested cardiovascular outcomes. Trials were categorized by design (superiority vs noninferiority), and assessed for primary outcome success, academic impact (Google Scholar citations), and online engagement (Altmetric Attention Score). Linear trends were assessed via linear regression. An interaction term was tested between trial design (superiority vs noninferiority) and outcome success (meeting the primary outcome or not) with respect to citation count.</p>
<h4>Results</h4>
<p> Among 429 cardiovascular RCTs, 100 (23.3%) were noninferiority trials, with their proportion increasing over time (<i>P</i> = .009). Noninferiority trials were significantly more likely to meet their primary outcome than superiority trials (79% vs 54%;<i> P </i>&lt; .001). Median citations were comparable between noninferiority and superiority trials (560 [IQR, 248-851] and 505 [IQR, [274-886], respectively; <i>P</i> = .83), as were Altmetric Attention Scores (226 [IQR, 124-481] and 234 [IQR, 120-542], respectively; <i>P</i> = .70) (<b>Figure 25-1150</b>). Regardless of design, trials that met their primary outcome had higher citation counts than those that did not (superiority trials, 615 [IQR, 280-1200] vs 423 [IQR, 267-734], respectively; <i>P</i> = .003; noninferiority trials, 613 [IQR, 261-998] vs 338 [IQR, 201-635], respectively; <i>P</i> = .049). There was no significant interaction between trial design and outcome success with respect to citation counts (<i>P</i> for interaction = .87).</p>

![](https://assets.underline.io/markdown_image/1/image/f38f193f9c584e236deb6401cf062bab.png)

<h4>Conclusions</h4>
<p> Noninferiority trials are increasingly shaping cardiovascular research and are more likely to meet their primary outcome, with academic engagement and online attention comparable to superiority trials. The higher success rate observed in noninferiority trials may, in part, reflect the discretionary—and at times overly permissive—approach to noninferiority margin selection, warranting further study. Finally, Altmetric Attention Scores reflect online attention but not clinical impact; future work will assess influence on guidelines and US Food and Drug Administration labeling.</p>
<sup>1</sup>Weill Cornell Medicine, New York, NY, US; <sup>2</sup>David Geffen School of Medicine, University of California, Los Angeles, CA, US; <sup>3</sup>Vanderbilt University Medical Center, Nashville, TN, US; <sup>4</sup>Keck School of Medicine, University of Southern California, Los Angeles, CA, US; <sup>5</sup>Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, US, bbikdeli@bwh.harvard.edu; <sup>6</sup>Tulane University School of Medicine, New Orleans, LA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


Non-Inferiority vs. Superiority Trials in Cardiovascular Research: Trends, Success in Meeting the Primary Outcome, and Online Engagement

<p>Michael C. Ferraro,<sup>1,2</sup> Aidan G. Cashin,<sup>1,2</sup> Amanda C. de C. Williams,<sup>3</sup> Emma Fisher,<sup>4</sup> Gavin Stewart,<sup>5</sup> Christopher Eccleston,<sup>4,6,7</sup> Neil E. O’Connell<sup>8</sup></p>
<h4>Objective </h4>
<p>Continued citation of retracted literature is a threat to scientific integrity. Most authors who cite retractions are unaware the article has been withdrawn.<sup>1</sup> While the International Committee of Medical Journal Editors (ICMJE) requires authors to check that no references in a manuscript cite retracted articles,<sup>2</sup> few journals include this requirement in author submission guidelines.<sup>3</sup> This study evaluated the effect of an audit and feedback intervention on retraction checking instructions in the top 20 pain and top 20 anesthesiology journal author submission guidelines.</p>
<h4>Design </h4>
<p>This pre-post audit and feedback intervention evaluated author submission guidelines of the top 20 pain and top 20 anesthesiology journals (according to Journal Citation Reports 2022 Impact Factor) for (1) recommendations to check manuscript references for retractions and (2) guidance on how to identify retractions. Each item was judged as yes or no (1 or 0, respectively) and given a total score out of 2. The feedback intervention was targeted at the journal editors in chief via a single email contact. Based on the journal’s score, a tailored letter was constructed (for pain and anesthesiology journals separately) including the journal score, comparison with other journals evaluated, and guidance (including a template) to implement ICMJE recommendations within author guidelines. A follow-up audit was performed at 6 months. This study was approved by University of New South Wales ethics and registered on Open Science Framework on April 2, 2024.</p>
<h4>Results </h4>
<p>The baseline audit was performed in April 2024. The mean (SD) impact factor was 3.6 (1.5) for pain journals and 4.6 (2.6) for anesthesiology journals. Twelve of 20 pain journals and 9 of 20 anesthesiology journals were ICMJE signatories. At baseline, 0 of 20 pain journals and 3 of 20 anesthesiology journals included recommendations to check references for retractions and 0 of 20 pain journals and 1 of 20 anesthesiology journals provided guidance on how to identify retractions. At 6-month follow-up in October 2024, 3 of 20 pain journals and 5 of 20 anesthesiology journals included recommendations to check references for retractions and 3 of 20 pain journals and 3 of 20 anesthesiology journals provided guidance on how to identify retractions (<b>Figure 25-0881</b>). Overall, this represented an absolute score change of 25%. Correspondence with editors of 2 pain journals and incorporation of the guidance template in 1 pain journal confirmed that these changes resulted from the feedback intervention.</p>

![](https://assets.underline.io/markdown_image/1/image/c3e027bdf09d715677071bd08372e17c.png)

<h4>Conclusions </h4>
<p>The number of pain and anesthesiology journals with author instructions to check references for retractions increased following a tailored audit and feedback intervention. However, overall uptake of the feedback was low, suggesting that more sustained editorial engagement strategies may be necessary to facilitate implementation of recommendations. Future studies should evaluate whether implementation of retraction screening recommendations into author submission guidelines reduces citation of retracted literature.</p>
<h4>References</h4>
<p>1. De Cassai A, Geraldini F, De Pinto S, et al. Inappropriate citation of retracted articles in anesthesiology and intensive care medicine publications. <i>Anesthesiology</i>. 2022;137(3):341-350. doi:10.1097/ALN.0000000000004302</p>
<p>2. International Committee of Medical Journal Editors. Preparing a manuscript for submission to a medical journal. Accessed January 15, 2025. <a href="https://www.icmje.org/recommendations/browse/manuscript-preparation/preparing-for-submission.html"><span class="Hyperlink CharOverride-6">https://www.icmje.org/recommendations/browse/manuscript-preparation/preparing-for-submission.html</span></a></p>
<p>3. Boudry C, Howard K, Mouriaux F. Poor visibility of retracted articles: a problem that should no longer be ignored. <i>BMJ</i>. 2023;381:e072929. doi:10.1136/bmj-2022-072929</p>
<sup>1</sup>Centre for Pain IMPACT, Neuroscience Research Australia, Sydney, New South Wales, Australia, m.ferraro@neura.edu.au; <sup>2</sup>School of Health Sciences, Faculty of Medicine and Health, University of New South Wales, Sydney, New South Wales, Australia; <sup>3</sup>Research Department of Clinical, Educational and Health Psychology, University College London, London, UK; <sup>4</sup>Centre for Pain Research, University of Bath, Bath, UK; <sup>5</sup>School of Natural and Environmental Sciences, University of Newcastle upon Tyne, Newcastle, UK; <sup>6</sup>Department of Health and Clinical Psychology, the University of Ghent, Ghent, Belgium; <sup>7</sup>Department of Psychology, The University of Helsinki, Helsinki, Finland; <sup>8</sup>Department of Health Sciences, Centre for Wellbeing Across the Lifecourse, Brunel University of London, Uxbridge, UK.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Aidan G. Cashin, Amanda C. de C. Williams, Emma Fisher, and Christopher Eccleston are editorial board members for 2 of the pain journals (<i>PAIN, Journal of Pain</i>) included in this audit. No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>No specific funding was received for this project. Michael C. Ferraro was supported by an Australian Government Research Training Program PhD scholarship and a Neuroscience Research Australia top-up scholarship.</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The sponsor, Neuroscience Research Australia, had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and decision to submit the manuscript.</p>
<h4>Additional Information</h4>
<p> This study was registered on Open Science Framework at <a href="https://osf.io/gbkjy/"><span class="Hyperlink CharOverride-6">https://osf.io/gbkjy/</span></a>.</p>


An audit and feedback intervention to reduce inappropriate citation of retracted literature in the pain and anaesthesiology fields

<p>Jing Cui,<sup>1</sup> Nan Yang,<sup>1</sup> Kexin Ji,<sup>1</sup> Dingran Yin,<sup>1</sup> Chen Shen,<sup>2</sup> Zhaolan Liu,<sup>2</sup> Han Tan,<sup>3</sup> Yaxin Sun,<sup>4</sup> Zhaoqi Huo,<sup>5</sup> Shuo Liu,<sup>5</sup> Huiyu Wang,<sup>5</sup> Xintong Zhang,<sup>6</sup> Jing Guo,<sup>1</sup> Yufei Wang,<sup>7</sup> Xiaoqi Ren,<sup>5</sup> Vincent C. H. Chung,<sup>8</sup> Jianping Liu<sup>2</sup></p>
<h4>Objective</h4>
<p> Publication retractions, especially in the field of medicine, have long been a matter of significant concern.<sup>1</sup> The trends and characteristics of retractions of publications on Traditional Chinese medicine (TCM) by authors from mainland China remain unclear. This study is dedicated to exploring the prevalence and underlying reasons of retractions in TCM publication.</p>
<h4>Design </h4>
<p>This cross-sectional study identified and analyzed publications that were marked as retractions or withdrawal. It encompassed retractions of TCM literature published by mainland Chinese authors in international peer-reviewed journals from January 2000 to November 2024. Seven medical databases were systematically searched, including 3 English-language databases (PubMed, Embase, and Cochrane Library) and 4 Chinese-language databases (China National Knowledge Infrastructure, Wanfang Data, VIP Database, SinoMed), as well as the Retraction Watch Database. Duplicate data were excluded. Article screening and data extraction were independently conducted by 2 authors. Discrepancies were resolved through discussion. The retractions of TCM publications by mainland Chinese authors were thoroughly analyzed. The numbers and characters of retractions across different study types were compared, and retraction reasons were comprehensively summarized according to Committee on Publication Ethics (COPE) guidelines. Main outcomes included the prevalence, reasons, and relevant factors of retraction (year, authors’ region and affiliation, study types, study subjects, and reasons).</p>
<h4>Results </h4>
<p>A total of 24,479 retracted articles were initially identified, and 679 retracted publications on TCM met inclusion criteria. Retractions encompassed laboratory studies (381 [56.1%]), clinical studies (189 [27.8%]), and secondary studies (109 [16.1%]). Notably, the number of retractions surged to 354 articles (52.1%) in 2023 (<b>Figure 25-0930</b>). Most first authors were affiliated with Grade A tertiary hospitals (the highest accreditation level in China’s health care system; 415 [61.1%]). Active-ingredient extracts of TCM (289 [42.6%]) and neoplasms (145 [22.6%]) emerged as the most prevalent study subjects. A total of 77.6% of retractions were attributed to multiple reasons, so that the sum of the percentage of reasons may exceed 100%. A total of 12 retracted articles had no specific reasons found. Author misconduct drove 576 retractions (86.4%), predominantly involving peer review manipulation (374 [56.1%]), falsification or fabrication of data (262 [39.3%]), and paper mills (244 [36.6%]).</p>

![](https://assets.underline.io/markdown_image/1/image/1dd3a7c51afdb9fddf1603eab8ac2367.png)

<h4>Conclusions </h4>
<p>Currently, retractions of TCM publications have become increasingly conspicuous, particularly those from higher-level academic institutions and hospitals. Systemic author misconduct, notably compromised peer review processes and industrialized fraud, may constitute the primary driver. However, some publishers regularly remove retracted literature from the platform, which results in omissions in the data. To promote scientific credibility in TCM publications, we recommend (1) enhanced institutional accountability mechanisms, (2) standardized retraction policies aligned with COPE guidelines, (3) mandatory research integrity training, and (4) transparent postretraction investigations.<sup>2</sup></p>
<h4>References</h4>
<p>1. Else H. Biomedical paper retractions have quadrupled in 20 years—why? <i>Nature</i>. 2024;630(8016):280-281. doi:10.1038/d41586-024-01609-0</p>
<p>2. Bakker CJ, Reardon EE, Brown SJ, et al. Identification of retracted publications and completeness of retraction notices in public health. <i>J Clin Epidemiol</i>. 2024;173:111427. doi:10.1016/j.jclinepi.2024.111427</p>
<sup>1</sup>Qi-Huang Chinese Medicine School, Beijing University of Chinese Medicine, Beijing, China; <sup>2</sup>Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, Beijing, China, liujp@bucm.edu.cn; <sup>3</sup>School of Acupuncture-Moxibustion and Tuina, Beijing University of Chinese Medicine, Beijing, China; <sup>4</sup>School of Management, Beijing University of Chinese Medicine, Beijing, China; <sup>5</sup>School of Traditional Chinese Medicine, Beijing University of Chinese Medicine, Beijing, China; <sup>6</sup>School of Humanities, Beijing University of Chinese Medicine, Beijing, China; <sup>7</sup>College of Continuing Education, Hebei University of Chinese Medicine, Shijiazhuang, China; <sup>8</sup>Jockey Club School of Public Health and Primary Care, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong SAR, China.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> High-level Traditional Chinese medicine key subjects construction project of National Administration of Traditional Chinese Medicine—Evidence-Based Traditional Chinese Medicine (award zyyzdxk-2023249).</p>
<h4>Role of the Funder/Sponsor </h4>
<p>The sponsor contributed to the decision to submit the abstract for presentation.</p>
<h4>Additional Information</h4>
<p> Jing Cui and Nan Yang contributed equally to this work, and Kexin Ji and Dingran Yin contributed equally to this work. Vincent C. H. Chung is a co–corresponding author (vchung@cuhk.edu.hk).</p>


Prevalence of and Reasons for Retractions on Traditional Chinese Medicine Research Publications by Authors from Mainland China in International Peer-review Journals

<p>Yagmur Ozturk,<sup>1</sup> Frédérique Bordignon,<sup>2,3</sup> Cyril Labbé,<sup>1</sup> François Portet<sup>1</sup></p>
<h4>Objective</h4>
<p> To address the challenge of tracking the use of unreliable sources in scientific literature, we analyzed citations to retracted articles (CRAs) using retraction reasons (RRs) and citation context (CC) analysis. Our aim was to inform the development of automated tools for postpublication peer review (PPPR) by identifying instances in which a CRA is not only an unreliable source, but may also introduce unreliability into the citing work itself, depending on the way it is cited. We examined the degree to which the content of the citation context aligns with the RR, indicating potential risk for the integrity of the citing article.</p>
<h4>Design</h4>
<p> RRs were obtained from the Retraction Watch database<sup>1</sup> for a set of 22,558 publications that cite retracted sources accessed from the Problematic Paper Screener.<sup>2</sup> We selected 29 of 108 retraction reasons that directly relate to scientific content of the article (eg, unreliable or fabricated data or results). To ensure access to CCs, we limited our dataset to open access citing articles and excluded those that had themselves been retracted. This yielded 4979 citing articles and 7424 unique CRA instances; 88% of these were preretraction citations. We then limited the rest of the analysis to a subset of 3049 CCs from the PubMed Central Open Access corpus.<sup>3</sup> We used regular expressions to search for terms referencing concepts related to the retraction reasons (eg, “data,” “results,” “findings”) and manually validated their use in the CCs.</p>
<h4>Results</h4>
<p> Of 3049 CCs, we identified 57 that explicitly mentioned the exact issue mentioned in the RR. Among these, 12 emphasized similarity or consistency with the retracted findings, which are instances we interpret as potential indicators of unreliability in the citing paper, meriting PPPR. These 57 CCs span 18 research fields and involve 60 distinct RRs (some overlapping with the selected 29). The cases are distributed across a range of publishers and fields, offering a basis to be used in the development of automatic systems.</p>

![](https://assets.underline.io/markdown_image/1/image/c740d6d31d9977a9f35654827d4b160b.png)

<h4>Conclusions</h4>
<p> While retracted publications are unreliable sources, we argue that how the retracted work is used and why it was retracted matter more than the fact that a retracted reference is found in a publication. If the retracted source is used to support the citing work, this calls for a secondary assessment when it is detected. By combining RRs with CC analysis, our approach informs a practical method to flag such cases for PPPR and supports the development of tools for large-scale CRA assessment.</p>
<h4>References</h4>
<p>1. The Center for Scientific Integrity. Retraction Watch Database.<a href="https://retractiondatabase.org/"><span class="Hyperlink CharOverride-6"> </span></a><a href="https://retractiondatabase.org/"></a></p>
<p>2. Cabanac G, Labbé C, Magazinov A. The ‘Problematic Paper Screener’ automatically selects suspect publications for post-publication (re)assessment. <i>arXiv</i>. Preprint posted online October 7, 2022. doi:10.48550/arXiv.2210.04895</p>
<p>3. National Library of Medicine. PMC Open Access Subset. Accessed January 30, 2025. <a href="https://pmc.ncbi.nlm.nih.gov/tools/openftlist/"><span class="Hyperlink CharOverride-6">https://pmc.ncbi.nlm.nih.gov/tools/openftlist/</span></a></p>
<sup>1</sup>Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, Grenoble, France, yagmur.ozturk@univ-grenoble-alpes.fr; <sup>2</sup>Ecole Nationale des Ponts et Chaussées, Institut Polytechnique de Paris, Marne-la-Vallée, France; <sup>3</sup>LISIS, INRAE, Univ Gustave Eiffel, CNRS, Marne-la-Vallée, France.</p>
<h4><a id="_idTextAnchor124"></a>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> We acknowledge the NanoBubbles project that has received Synergy grant funding from the European Research Council within the European Union’s Horizon 2020 program (grant agreement number 951393) (<a href="https://cordis.europa.eu/project/id/951393"><span class="Hyperlink CharOverride-6">https://cordis.europa.eu/project/id/951393</span></a>).</p>


Citation Context Analysis of Retracted Articles: Leveraging Retraction Reasons to Track Unreliability in Citing Literature

<p>Caitlin J. Bakker,<sup>1,2</sup> Erin E. Reardon,<sup>3</sup> Nicole Theis-Mahon,<sup>4</sup> Sara Schroter,<sup>5,6</sup> Lex Bouter,<sup>7,8</sup> Maurice P. Zeegers<sup>2</sup></p>
<h4>Objective</h4>
<p> Systematic reviews gather, appraise, and synthesize studies to inform research, practice, and policy. However, the inclusion of retracted articles, which often contain flaws and falsified or fabricated data, undermines the credibility of systematic reviews. Identifying retracted articles is challenging, as they are inconsistently flagged.<sup>1,2</sup> Our study validated and compared approaches to identify retracted articles and their retraction notices.</p>
<h4>Design</h4>
<p> Our study, guided by an advisory panel of information specialists and researchers, evaluated approaches for identifying retracted publications from 8 health sciences databases (Cochrane Library, Embase.com, Ovid Embase, Ovid Medline, Ovid PsycINFO, PubMed, Scopus, and Web of Science). Using a reference set of 43,544 retracted publications and 27,755 associated retraction notices from Retraction Watch, we identified items found in each database. From August 10 to 14, 2024, we determined how many of the available items could be retrieved using each approach per database. Two search strategies, database indexing, and 2 citation managers were evaluated. The complete methodology, including search strategies, is available in our protocol.<sup>3</sup> Recall (sensitivity) was calculated to evaluate identification effectiveness.</p>
<h4>Results</h4>
<p> Recall of retracted publications and notices varied across databases and retrieval approaches. Across databases, search strategy 2 consistently achieved the highest recall, with values ranging from 74.6% to 96.9%. Search strategy 1 also performed strongly, particularly in PubMed and Web of Science (both &gt;93%). In contrast, indexing-based retrieval showed variable performance, with high recall in PubMed (94.6%) and Ovid Medline (94.5%) but much lower in Embase.com (40.6%) and PsycINFO (34%). Citation manager tools (EndNote and Zotero) yielded lower recall, with values rarely exceeding 64%. Recall was lowest for Ovid Embase and PsycINFO regardless of method, while PubMed and Web of Science showed the highest recall.</p>

![](https://assets.underline.io/markdown_image/1/image/b3957ae8a92f24f08736a44da92e7d56.png)

<h4>Conclusions</h4>
<p> There was substantial variability in the ability of databases and retrieval approaches to identify retracted publications and notices. No single approach captured all items, underscoring the need for multiple approaches in an iterative identification process. </p>
<h4>References</h4>
<p>1. Bakker CJ, Reardon EE, Brown SJ, et al. Identification of retracted publications and completeness of retraction notices in public health. <i>J Clin Epidemiol</i>. 2024;173:111427. doi:10.1016/j.jclinepi.2024.111427</p>
<p>2. Boudry C, Howard K, Mouriaux F. Poor visibility of retracted articles: a problem that should no longer be ignored. <i>BMJ</i>. 2023;381:e072929. doi:10.1136/bmj-2022-072929</p>
<p>3. Bakker C, Reardon EE, Theis-Mahon NR, Schroter S, Bouter L, Zeegers M. Validation and comparison of methods to identify retracted publications during information retrieval. Open Science Framework. Cited June 7, 2025. <a href="https://osf.io/rwzym/"><span class="Hyperlink CharOverride-6">https://osf.io/rwzym/</span></a></p>
<sup>1</sup>University of Regina, Regina, SK, Canada, caitlin.bakker@uregina.ca; <sup>2</sup>Maastricht University, Maastricht, the Netherlands; <sup>3</sup>Emory University, Atlanta, GA, US; <sup>4</sup>University of Minnesota, Minneapolis, US; <sup>5</sup>BMJ, London, UK; <sup>6</sup>London School of Hygiene and Tropical Medicine, London, UK; <sup>7</sup>Amsterdam University Medical Center, Amsterdam, the Netherlands; <sup>8</sup>Vrije Universiteit Amsterdam, Amsterdam, the Netherlands.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Caitlin J. Bakker is cochair of the National Information Standards Organization Communication of Retractions, Removals and Expressions of Concern Standing Committee. No other disclosures were reported.</p>
<h4>Funding/Support </h4>
<p>This research is part of an ongoing PhD collaboration between BMJ and the team Meta-Research at Maastricht University (UM) on the responsible conduct of publishing scientific research. BMJ is published by BMJ Group, a wholly owned subsidiary of the British Medical Association. UM is a public legal entity in the Netherlands. This study is part of Caitlin J. Bakker’s self-funded BMJ/UM PhD. No exchange of funds has taken place for this research project.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The authors are wholly responsible for the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, and approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Additional Information</h4>
<p> All authors express their own opinions and not necessarily that of their employers.</p>


Evaluating Approaches for Identifying Retracted Articles and Retraction Notices in Systematic Review Searching

<p>Guillaume Cabanac,<sup>1,2</sup> Kathryn Weber-Boer<sup>3</sup></p>
<h4>Objective</h4>
<p> We analyzed postretraction citations of retracted publications in biomedical and clinical sciences (BCS). Previous studies have investigated the citation of retracted clinical trials<sup>1</sup> and the citation of retracted publications in policy documents,<sup>2</sup> but the share of references cited after their retraction in BCS literature is unknown. We identified the articles citing postretracted references and present the results aggregated at the journal and publisher levels.</p>
<h4>Design</h4>
<p> We used Dimensions data accessed in BigQuery to delineate the corpus, including publications of type ‘article’, classified as “review article” or “research article”, with the field of research category BCS (32, in the ANZSRC 2020 classification), published between 2005 and 2024. We used the snapshot of the Dimensions dataset from 1 June 2025. These 12,240,884 articles were published in 44,901 journals from 11,961 publishers, and contained 401,909,282 references of 28,058,673 unique publications. We extracted the year of publication, journal titles, publisher names, and list of references cited. We retained those references which were removed, retracted, or withdrawn (henceforth ‘retracted’), based on data derived from Crossref, PubMed, Retraction Watch, and retraction notices issued by publishers. We excluded publications which cited retractions which had a retraction year earlier than or equal to the year of publication, to consider only “post-retraction references.”</p>
<h4>Results</h4>
<p> We observed a growing number of publications in the BSC literature that cite retracted publications. In 2005, there were 674 publications referencing 638 retracted publications 688 times. This grew to 13,927 publications referencing 14,214 retracted publications 15,339 times in 2024: a 2066% increase in publications and retracted references and a 2228% increase in retracted publications cited. This compared to an increase over the same period of 267% for publications in BSC (from 368,338 to 983,837) and 390% in citations (8,495,152 to 33,143,231). This is an increase in the proportion of publications with retracted references from to 0.18% to 1.69% of all BSC publications. The number of retracted publications referenced doubled since 2019. The connection between journal and publisher reflected the ownership relationship as of approximately January 2025. The journals in which post-retraction referencing of retracted literature occurred are many (7225 distinct journals, 17%), but over half of the publications (57,292 of 110,492) are currently owned by four major publishers: Elsevier (23,725; 21%), Springer Nature (17,727; 16%), Wiley (9488; 9%), and Frontiers (6352; 6%).<b>(Table 25-1151)</b></p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1151t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1151t1.png"> </a>
<h4>Conclusions</h4>
<p> The publishers of articles responsible for the most citations of postretracted references are the largest publishers of scientific literature by any measure. We could have ranked or displayed the results by publisher or journal as a percentage of publications overall; however, in this case, quantity matters more than share. This practice is preventable, with system-wide corrective efforts by the publishing industry, such as prepublication checks for retracted references, as a complement to the postpublication checks of retracted references that are also needed.<sup>3</sup></p>
<h4>References</h4>
<p>1. Kataoka Y, Banno M, Tsujimoto Y, et al. Retracted randomized controlled trials were cited and not corrected in systematic reviews and clinical practice guidelines. <i>J Clin Epidemiol</i>. 2022;150:90-97. doi:10.1016/j.jclinepi.2022.06.015</p>
<p>2. Malkov D, Yaqub O, Siepel J. The spread of retracted research into policy literature. <i>Quant Sci Stud</i>. 2023;4(1):68-90. doi:10.1162/qss_a_00243</p>
<p>3. Cabanac G. Chain retraction: how to stop bad science propagating through the literature. <i>Nature</i>. 2024;632(8027):977-979. doi:10.1038/d41586-024-02747-1</p>
<sup>1</sup>Université de Toulouse, IRIT (UMR 5505 CNRS), Toulouse, France; <sup>2</sup>Institut Universitaire de France, Paris, France; <sup>3</sup>Digital Science, London, UK, k.weberboer@digital-science.com.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Kathryn Weber-Boer is employed by Digital Science, which provided access to the Dimensions datasets that this work analyzed. Guillaume Cabanac received funding from the Institut Universitaire de France.</p>
<h4>Additional Information </h4>
<p>Guillaume Cabanac and Kathryn Weber-Boer are co–corresponding authors.</p>


Next from The 10th Peer Review Congress

Characterizing Adverse Event Methods Reported in ClinicalTrials.gov and Publications

Similar lecture

Use of an Open Science Checklist and Reproducibility of Findings: A Randomized Controlled Trial