News

Scientific conclusions depend on who performs the analysis

Published: 2.4.2026

Researchers reanalysed hundreds of previously published studies in the social and behavioural sciences. Only one in three re-analyses reached the same conclusions.

Research often involves choosing a single analytic path, but there are other options available, Picture: Matti Ahlgren, Aalto University.

A new study published in Nature, "Investigating the analytic robustness of the social and behavioural sciences," finds that scientific conclusions can shift dramatically depending on who conducts the analysis.

More than 450 independent researchers from around the world conducted over 500 re-analyses of datasets from one hundred previously published studies in the social and behavioural sciences. All analysts received the same data and the same central research question, but they were free to carry out the analysis based on their own expert judgment.

The study revealed that scientific conclusions can change substantially depending on who performs the analysis.

‘All analysts reached the same conclusion as the original authors in about one third of cases,’ says data scientist Enrico Glerean from Aalto University.

The results come from a large-scale international collaboration led by Balázs Aczél and Barnabás Szászi from Eötvös Loránd University and Corvinus University, conducted as part of the Systematizing Confidence in Open Research and Evidence (SCORE) program with altogether 865 researchers.

‘It was truly a gigantic effort, with hundreds of people re-analysing published studies to assess how replicable their findings were,’ says Glerean.

All analysts reached the same conclusion as the original authors in about one third of cases

Enrico Glerean

Discrepancies are not due to lack of expertise

Over the past decade, the social and behavioural sciences have undergone substantial reforms aimed at making research more transparent, rigorous and reliable. One important question, however, is receiving increasing attention: to what extent do research findings depend on the specific way in which data are analysed?

‘Research often involves choosing a single analytic path, where one dataset is typically analysed by one researcher or one team. But there are also other reasonable ways to analyse the same data,’ says Aalto University Assistant Professor Christoph Huber.

While peer review assesses methodological acceptability, it rarely reveals what results might have emerged under alternative, yet equally defensible, statistical decisions.

‘Sometimes researchers disagree about how certain types of data should be analysed in the first place. At other times, the phenomena themselves do not permit a single, unequivocal interpretation,’ says Enrico Glerean.

Person in dark suit and pink tie leaning against a tiled wall, hands in pockets — Christoph Huber

Yet empirical research involves numerous decision points: how data are cleaned, how variables are defined, which statistical models or software are used, and how results are interpreted. Together, these choices constitute what is known as analytic variability — the flexibility that can fundamentally influence final conclusions.

‘Perhaps many findings rest on choices that don’t actually hold up under scrutiny. It’s worth having a closer look and rethinking how we’re doing science. Hopefully, this means that we will have more credible results in the future,’ Huber says.

The discrepancies were not due to a lack of expertise. Experienced researchers with strong statistical backgrounds were just as likely to arrive at divergent results as others. At the same time, observational studies proved less robust than experimental ones.

Research often involves choosing a single analytic path. But there are also other reasonable ways to analyse the same data

Christoph Huber

‘I personally reanalysed an experimental study and reached the same results. This may be because experimental data allow less room for analysis-dependent, open-ended conclusions,’ Huber notes.

We need humans more than machines

One way to increase the reliability of research findings is a so-called multiverse and multi-analysts analysis approach, in which the same dataset is analysed by different teams using different methodological choices, rather than relying on a single team and a single set of analytic decisions.

‘Multiverse and multi-analysts analyses can help show which results are robust regardless of the methods used. This also means that the social side of science becomes even more important: researchers should discuss different approaches and challenge one another,’ Glerean says.

Glerean also notes that popular generative AI tools are exerting an ever-greater influence on everything, and it is often trained on vast amounts of old material. This means it also draws on findings that have been shown to be irreproducible, misleading or simply wrong.

‘Science advances faster, and it self-corrects better than large language models trained on everything that was ever written. We will still need humans more than machines when it comes to understanding what counts as the best explanation of a phenomenon,’ Glerean says.

Original news article: Large-Scale Collaboration Releases New Findings on Research Credibility

Nature: Investigating the analytical robustness of the social and behavioural sciences