2022-23 Seminars and Recordings

May 19, 2023

MetaReviewer: A Free, Collaborative, Browser-Based Tool for Conducting Data Extraction

  • Speaker: Joshua Polanin, American Institutes for Research

  • Description: Collecting evidence synthesis data from even a few studies presents numerous logistical and organizational challenges. Reviewers with the expertise or resources turn to bespoke database software or costly meta-analysis programs. In the absence of resources or expertise, many reviewers turn to a spreadsheet, which can lead to inaccurate or unreliable data collection. With this in mind, we created a program for the collection of evidence synthesis information called MetaReviewer. Dr. Polanin will demonstrate some of its capabilities and functionalities and discuss potential future upgrades. Although the Beta testing period is over and we currently are not accepting new users, you can also check out the beta version here: https://www.metareviewer.org/ We expect to release an updated version in the fall of 2023.

  • Slides

  • Video Recording

March 17, 2023

Lessons Learned from a Participatory Evidence Synthesis

  • Speaker: Sarah Peko-Spicer, American Institutes for Research

  • Description: Education policy has long been plagued by a knowledge-to-action gap. Researchers have minimized this gap by engaging individuals who are directly affected by policy in primary research studies using participatory research methods (PRM). PRM emphasize academic researchers working alongside youth, practitioners, and community members to co-construct knowledge and effect change through localized, community-driven implementation. Many studies that use PRM are qualitative in nature, leaving many researchers wondering about the extent to which PRM can be used to support quantitative inquiries, including quantitative syntheses. In this talk, the authors will discuss a 10-session training, developed to facilitate a participatory evidence synthesis on social and emotional learning programs and strategies.

  • Slides

  • Video Recording

February 17, 2023

Meta-Analytic QuantCrit: Exploring the Possibilities

  • Speaker: Jamaal Young, Texas A&M University

  • Description: Meta-analyses of critical issues are worthwhile in the field of education because they provide evidence that informs strategies and policies to support statistically stigmatized and stereotyped students (4S). QuantCrit can guide the design, implementation, and interpretation of meta-analyses of critical issues as a research paradigm to support equitable learning outcomes. Nevertheless, I am unaware of any approaches to meta-analytic QuantCrit. Thus, the goal of this session is to set the stage for exploring how QuantCrit could be applied to meta-analytic thinking and research. To accomplish this goal, I will provide an overview of QuantCrit and its five tenets: 1) the centrality of racism; 2) numbers are not neutral; 3) categories are neither ‘natural’ nor given, and models that treat categories as causal often conceal the operation of power and ideology; 4) voice and insight (data cannot speak for itself); and 5) a social justice/equity orientation (Gillborn et al., 2018). I will conclude by providing specific examples of each tenet within the context of meta-analytic thinking and practice.

  • Slides

  • Video Recording

January 20, 2023

Your New Best Friend, Heterogeneity: Diverse Evidence Types and Diverse Methods to Address Explanatory Questions in Systematic Reviews

  • Speaker: G.J. Melendez-Torres, University of Exeter

  • Description: Heterogeneity is, to many meta-analysts, an irritation or an afterthought: a nuisance parameter, an opportunity for misinterpretation, or (for the workshy) an excuse to avoid meta-analysis completely. But viewed another way, heterogeneity provides an essential basis for meta-analyses that go beyond inferential tests and summative answers to explanatory insights. In this discussion, I consider deductive, abductive and inductive methods that move past ‘exploring’ heterogeneity to capitalising on heterogeneity as an opportunity to deliver better, more useful findings for policy and practice impact, in bodies of evidence both large and small.

  • Slides

  • Video Recording

December 16, 2022

Comprehensive and Transparently Reported Search Strategies

  • Speaker: Amy Riegelman, University of Minnesota

  • Description: By definition, systematic reviews and meta-analyses are supposed to have exhaustive and comprehensive literature searches. Additionally, the strategies used to identify relevant literature need to be transparently reported in order to facilitate reproducibility. While some research topics are well indexed and therefore readily discoverable, literature on others research topics is dispersed across many databases and search engines, which poses challenges for systematic searching. Furthermore, grey literature is often sought for reviews, but grey sources can be very challenging to identify. This talk will describe guidelines and best practices for targeting the relevant literature while balancing precision and recall. This seminar will also include tips and tricks for conducting comprehensive searches while adhering to guidelines (e.g., PRISMA) to help ensure reproducibility of search methods.

  • Slides

  • Video Recording

November 18, 2022

Synthesizing Individual Participant Data Obtained From Complex Sampling Surveys: A Two-Stage IPD Meta-Analysis Approach

  • Speaker: Diego Campos, University of Oslo

  • Description: The growing availability of individual participant data (IPD) shared by researchers and larger institutions provides new opportunities for meta-analyses. However, some of the readily accessible data sets are based on complex sampling designs—that is, designs that typically involve multi-stage sampling, clustering, and stratification. Accounting for different sample procedures, stages, clusters, and weights across datasets and dealing with the hierarchical structure of the primary and meta-analytic data are some of the methodological challenges in synthesizing individual participant data from complex sampling studies. Thus, the analysis and synthesis of complex survey data need to consider the sample design features of the primary studies and the dependencies among effect sizes during the meta-analysis. We propose a two-stage individual participant data meta-analysis approach to address these challenges. This talk will illustrate this approach and present hands-on steps for synthesizing IPD from complex sampling surveys.

  • Slides

  • Video Recording

October 21, 2022

Advances in Methods for Overviews of Reviews

  • Speaker: Carole Lunny, Knowledge Translation program at the University of Toronto and Cochrane Hypertension Review Group at the University of British Columbia

  • Description: Since 2018, significant methodological advances in how to conduct overviews of reviews have been published. This presentation briefly orients participants to some of the guidance, tools and methods recently published on overviews of reviews. I will present the methods work by our team on: (i) the prevalence of overviews, journal impact factors and number of citations, (ii) a taxonomy for categorising overviews with similar eligibility criteria, (iii) how to handle overlap in primary study data across systematic reviews on the same topic, (iv) replication of the Jadad algorithm to assess discordance, (v) survey of decision makers to determine how they compare and choose amongst competing systematic review on the same topic, and (vi) development of the WISEST AI tool to automatically quality assess and compare the PICO, methods and results across systematic reviews on the same topic.

  • Slides

  • Video Recording

References to these studies are found below:

  1. Lunny C, Thirugnanasampanthar SS, Kanji S, Ferri N, Thabet P, Pieper D, et al. Identifying and addressing conflicting results across multiple discordant systematic reviews on the same question: protocol for a replication study of the Jadad algorithm. BMJ Open. 2022;12(4):e054223.

  2. Lunny C, Reid EK, Neelakant T, Chen A, Zhang JH, Shinger G, et al. A new taxonomy was developed for overlap across ‘overviews of systematic reviews’: A meta-research study of research waste. Res Synth Methods. 2022;13(3):315-29.

  3. Lunny C, Neelakant T, Chen A, Shinger G, Stevens A, Tasnim S, et al. Bibliometric study of ‘overviews of systematic reviews’ of health interventions: Evaluation of prevalence, citation and journal impact factor. Res Synth Methods. 2022;13(1):109-20.

  4. Lunny C, Thirugnanasampanthar, S.S., Kanji, S., Pieper, D., Whitelaw, S., Tasnim, S., Reid, E., Zhang, J.H.J., Kalkat, B., Chi, Y., Abdoulrezzak, R., Zheng, D.W.W., Pangka, L., Wang, D.X.R., Safavi, P., Sooch, A., Kang, K.T., Ferri, N., Nelson, H., Tricco, A.C. How can clinicians choose between conflicting and discordant systematic reviews? A replication study of the Jadad algorithm. In process at BMC Medical Research Methodology, and preprint at https://www.researchsquare.com/article/rs-1143357/v3. 2022.

  5. Lunny C, Whitelaw; C., Chi, Y., Kanji, S., Pieper, D., Shea, B., Veroniki, A-A., Ardern, C., Pham, B., Reid, E., Ferri, N., Bagheri, E., Zhang, J., Tricco, A.C. . Decision maker opinions on how they use and compare the strengths and weaknesses of one or more systematic reviews on a similar topic: a cross-sectional study. Preprint at https://osf.io/gr42d/. 2022.

  6. Lunny C, Pieper D, Thabet P, Kanji S. Managing overlap of primary study results across systematic reviews: practical considerations for authors of overviews of reviews. BMC Med Res Methodol. 2021;21(1):140.

  7. Goossen K, Hess S, Lunny C, Pieper D. Database combinations to retrieve systematic reviews in overviews of reviews: a methodological study. BMC Med Res Methodol. 2020;20(1):138.

September 30, 2022

Usage of Individual Patient Data Meta-Analysis to Synthesize Single-Case Research Data

  • Speaker: Mariola Moeyaert, SUNY Albany

  • Description: We have entered an era in which scientific knowledge and evidence increasingly informs research practice and policy. As the use of personalized trials increases, there is a remarkable and growing interest in the quantitative synthesis of single-case design (SCD) data. One technique that is developed and can be applied for this purpose is individual patient data (IPD) meta-analysis (also called raw SCD data meta-analysis). The IPD approach provides detailed information about intervention effectiveness and intervention heterogeneity. IPD is a flexible modeling approach, allowing for a variety of modeling options such as modeling moderators to explain intervention heterogeneity. During the seminar, the IPD meta-analytic technique (with the inclusion of moderators) will be introduced and demonstrated, and statistical properties of the model parameters will be discussed.

  • Slides

  • Video Recording