ONH Metascience Symposium 2026

Sign-up form and time schedule

Symposium contents:

Metacheck workshop (Cristian Mesquida & Daniel Lakens) 
We will present our beta version of the tool Metacheck, an open-source tool that assists researchers and reviewers to check scientific manuscripts for the presence or absence of best practices in open science. Similar to tools such as ‘Statcheck’ the tool will allow users to run a set of automated checks for anything we can detect with sufficient accuracy, such failures to follow reporting guidelines, best practices in data and code sharing, and a range of checks for references. We will present a demo of the validated modules, discuss our future plans, and highlight the usefulness of the tool for metascience research.
 

Talks
The meaningful interpretation of effect sizes (Daniel Lakens)
For more than a century, scientists have emphasized the importance of not only knowing whether an effect exists, but also interpreting the size of the effect. Is the effect of an experiment large enough to matter theoretically or practically? Or is the effect small enough that we can ignore it? For many psychological measures (such as subjective happiness on a 7-point scale), these questions are not at all easy to answer. In this talk, we will discuss a framework to guide a research program with the goal to help researchers in psychology not only report effect sizes, but also interpret them in a more meaningful way.

Meta-analytic approaches to testing models of energy expenditure, model riskiness, and the need to appropriately propagate error (James Steele)
Competing theories are often evaluated using quantitative predictions, yet conclusions can depend critically on how uncertainty is handled in the estimation strategies used to test these predictions. Using the debate between additive and constrained models of total energy expenditure (TEE) as a case study, this talk revisits recent evidence claimed to falsify the additive model in favour of the constrained. I re-analyse the same exercise-intervention studies used in a recent evidence synthesis to support constrained TEE but explicitly propagate and model uncertainty in both expected and observed energy expenditure responses. Appropriate propagation of error, in this case sampling error, substantially alters the evidential picture: estimates become compatible with additive predictions and provide little basis for strong rejection of the additive model. Beyond the substantive implications for human energy expenditure, this case illustrates broader metascientific issues concerning error propagation, regression dilution, model riskiness, and Meehlian corroboration. I argue that stronger theory appraisal requires tests that appropriately incorporate uncertainty and fairly evaluate the competing predictions implied by alternative models.

Improving Coordination in Psychological Science (Sajedeh Rasti)
The need for coordination in science is increasingly acknowledged. However, relatively little research has examined the various dimensions and nuances of coordination in science. This is especially relevant for psychological science, a field that continues to face systemic challenges and multiple crises. In this talk, we will explore what coordination means in the context of science, examine its potential advantages and disadvantages, and discuss a recent initiative aimed at fostering coordination in metascience.

A tutorial for calculating field-specific effect size distributions (Ben Glaser)
Effect sizes are useful for understanding the magnitude of study results and for planning new studies via power analysis. However, despite their wide usage, effect sizes are often misinterpreted. This is mostly due to an over-reliance on general effect size benchmarks that were not intended for broad application across diverse research fields. Inaccurate effect size interpretations can lead to incorrect conclusions about the magnitude of study results and incorrect sample size estimates, thereby increasing the likelihood of false-positive results. This talk introduces the ESDist R package, which is designed to calculate empirically derived effect-size benchmarks or a range of reliably detectable empirical effect sizes for a specific research question or field of interest by computing effect size distributions (ESDs). This package can be used on data that can be easily extracted from pre-existing meta-analyses to help researchers more accurately plan new studies or to better understand how an individual study might relate to other studies in their field. ESDist includes a set of features that make it easy to use in a priori power analysis. Moreover, the package includes a feature for estimating effect size benchmarks that account for publication bias and are weighted by effect sizes' variances, which addresses existing limitations of using ESDs for study planning or interpretation.

Using open weight AI models to extract statistical information from published articles at scale (Peder M Isager)
Recent advances in artificial intelligence have made it possible for researchers to make use of AI for a wide variety of research tasks, including metanalytic data curation and systematic review. A challenge in utilizing AI for data collection is that current frontier models are closed source, expensive to run, and too large to be run locally to ensure data protection. To compensate for this, we can make use of smaller open-weight Large Language Models (LLMs) that can be run for free on local machines. In this talk I will describe an ongoing pilot project to extract sample size information from published articles at scale. I will cover the current open-weight LLM landscape, benefits and challenges in using open-weight LLMs, how to implement such LLMs in data curation, and how accuracy of AI model output can be validated against human-coded data.

From Guesswork to Rigor: Improving Power Analysis in Sports and Exercise Science Research (Cristian Mequida)
A priori power analyses are widely used to justify sample sizes in sports and exercise science, yet they are often poorly implemented. Common issues include reliance on inflated or inappropriate effect sizes, retrospective justification of sample sizes, and limited consideration of study design. In this talk, I critically examine these limitations and show how current practices can lead to systematically underpowered or misleadingly justified studies. I then discuss practical improvements, including greater transparency in assumptions, the use of structured frameworks such as PICOS, and increased adoption of simulation-based approaches to sample size planning.