2024-25 Seminars and Recordings

September 30, 2024

April 18, 2025, 11 am CST

Synthesizing Qualitative Evidence in Education: Opportunities and Challenges of Meta-Aggregation

Speaker: Dr. Yukiko Maeda, Purdue University
Description: This presentation introduces meta-aggregation as a systematic methodology for synthesizing qualitative evidence in education. Unlike interpretive approaches such as meta-ethnography, meta-aggregation is grounded in pragmatism and aims to preserve the original meanings of findings without reinterpretation, generating actionable recommendations that support evidence-based practice. Drawing on the recent applications in mathematics education, we outline the step-by-step process of meta-aggregation from extracting findings with supporting evidence, to categorizing them by shared meaning, to developing actionable recommendations. We discuss the promise and potential of this method for enhancing the transparency, usability, and practical relevance of qualitative synthesis in applied educational contexts. At the same time, we address key methodological challenges, including inconsistent reporting standards and the need for nuanced judgment when evaluating credibility. This session is designed for researchers and graduate students seeking structured, rigorous approaches to qualitative synthesis and for those aiming to better connect qualitative insights with educational policy and practice.
Slides
Video Recording
```
  
```

Launching a Living Systematic Review of SEL in a Politically Charged Climate: From Evidence Synthesis to Practical Significance

Speaker: Dr. Christina Cipriano, Yale School of Medicine
Description: Improving student academic, social, emotional, and behavioral outcomes in school through efficacious social and emotional learning programs is simultaneously an area of national urgency and political criticism. Towards this end, a first of its kind living systematic review was launched to monitor the evolving inputs and outcomes of social and emotional learning programs in a dynamic educational landscape. In this session, Dr. Cipriano will discuss (1) what is a living systematic review and how it differs from a traditional systematic review and meta-analysis, (2) why this methodology holds utility for social and emotional learning evidence, and (3) critical challenges as opportunities to advance access and use of social and emotional learning evidence in a political charged climate.

Adapting Methods for Correcting Selective Reporting Bias in Meta-Analysis of Dependent Effect Sizes

Speaker: Dr. Man Chen, University of Texas at Austin
Description: Various methods have been proposed to adjust for selective reporting bias in average effect size estimates under the assumption that each study produces one effect size. However, there remains a need to investigate the selection bias adjustment methods in meta-analysis with multiple, dependent effect sizes. In this presentation, I will demonstrate novel adaptations of several adjustment methods based on a multivariate working model and weighting scheme that correct for selective reporting bias while handling dependencies among effect sizes. I will present the performance of existing adjustment methods and the novel adaptations based on an extensive Monte Carlo simulation study. I will conclude with a discussion of limitations, future directions, and suggestions for correcting selection bias in the context of dependent effect sizes. The presentation will cover research described in a pre-print available at https://doi.org/10.31222/osf.io/jq52s.
Slides
Video Recording
```
  
```

Friday the 13th & Single Case Experimental Designs: Allegedly Unlucky Encounters for Meta-Analysts

Speakers: Dr. John Ferron, University of South Florida, and Dr. Megan Kirby, Language Dynamics Group
Description: With increased use of single-case experimental designs to study intervention effects, there is a greater need to include single-case experimental design studies in meta-analyses. In this session, we will discuss alternative methods for estimating effects from single-case studies, illustrate the estimation of these effects from freely available software apps, and provide guidance about choosing among the effect size options.
Slides
Video Recording
```
  
```

Using GPT API Models as Second Screeners of Titles and Abstracts in High-Quality Systematic Reviews

Speaker: Dr. Mikkel Vembye, Danish Center for Social Science Research
Description: Independent human double screening of titles and abstracts is a critical step to ensure the quality of systematic reviews and meta-analyses. However, double screening is a resource-demanding procedure that slows down the review process. In this presentation, I will demonstrate how reviewers reliably can implement OpenAI’s GPT models as second screeners of title and abstracts to mitigate this resource demand. I will provide a practical demonstration of how to conduct title and abstract screening with GPT API models through the newly developed R package AIscreenR and highlight how the approach differs from simply using the ChatGPT interface. I will discuss implementation requirements for ensuring performance that is at least on par with typical human screening. To facilitate this, I will introduce a new screening benchmark scheme that is based on common human screening performance in 22 large-scale systematic reviews. As the initial proof of concept, I will present results of using GPT API screening in three large classification experiments. I will conclude by discussing limitations and future directions for this screening method and how it can be combined with traditional (semi-)automated screening tools. The background for this presentation can be found at https://osf.io/preprints/osf/yrhzm.
Slides
Video Recording
```
  
```

Fitting Flexible Meta-Analytic Models with Structural Equation Modeling

Speaker: Dr. Mike Cheung, National University of Singapore
Description: Understanding the differences between various meta-analytic models, such as the fixed-effect, random-effects, and multiplicative error models, can be challenging for researchers. This presentation explains these models using a graphical representation within a structural equation modeling (SEM) framework. Users specify meta-analytic models using lavaan syntax. The metaSEM package in R can convert these models to graphical models, model-implied mean and variance, and fit models using the Full Information Maximum Likelihood (FIML) estimation method. Complex meta-analytic models, such as multivariate meta-analysis and mediation models, can also be fitted. Users without strong statistical and programming backgrounds can still develop and implement novel meta-analytic models that are not available in standard meta-analysis software.
Slides
Video Recording
```
  
```

Getting Meta-Analyses Done: Practical Perspectives from Experienced Reviewers

Panelists:
- Dr. Carlton Fong, Texas State University
- Dr. Amanda Neitzel, Johns Hopkins University
- Dr. Peng Peng, University of Texas at Austin
Description: Our panel of prolific and experienced systematic reviewers will be discussing the very practical side of systematic reviews and meta-analyses, common challenges and roadblocks, and tips for getting meta-analyses done, with plenty of time for audience questions and discussion.
Video Recording
```
  
```