School of Science
Science for tomorrow’s technology, innovations and businesses
Scientific conclusions depend on who performs the analysis
Researchers reanalysed hundreds of previously published studies in the social and behavioural sciences. Only one in three re-analyses reached the same conclusions.
A new study published in Nature, "Investigating the analytic robustness of the social and behavioural sciences," finds that scientific conclusions can shift dramatically depending on who conducts the analysis.
More than 450 independent researchers from around the world conducted over 500 re-analyses of datasets from one hundred previously published studies in the social and behavioural sciences. All analysts received the same data and the same central research question, but they were free to carry out the analysis based on their own expert judgment.
The study revealed that scientific conclusions can change substantially depending on who performs the analysis.
‘All analysts reached the same conclusion as the original authors in about one third of cases,’ says data scientist Enrico Glerean from Aalto University.
The results come from a large-scale international collaboration led by Balázs Aczél and Barnabás Szászi from Eötvös Loránd University and Corvinus University, conducted as part of the Systematizing Confidence in Open Research and Evidence (SCORE) program with altogether 865 researchers.
‘It was truly a gigantic effort, with hundreds of people re-analysing published studies to assess how replicable their findings were,’ says Glerean.
Enrico GlereanAll analysts reached the same conclusion as the original authors in about one third of cases
Discrepancies are not due to lack of expertise
Over the past decade, the social and behavioural sciences have undergone substantial reforms aimed at making research more transparent, rigorous and reliable. One important question, however, is receiving increasing attention: to what extent do research findings depend on the specific way in which data are analysed?
‘Research often involves choosing a single analytic path, where one dataset is typically analysed by one researcher or one team. But there are also other reasonable ways to analyse the same data,’ says Aalto University Assistant Professor Christoph Huber.
While peer review assesses methodological acceptability, it rarely reveals what results might have emerged under alternative, yet equally defensible, statistical decisions.
‘Sometimes researchers disagree about how certain types of data should be analysed in the first place. At other times, the phenomena themselves do not permit a single, unequivocal interpretation,’ says Enrico Glerean.
Yet empirical research involves numerous decision points: how data are cleaned, how variables are defined, which statistical models or software are used, and how results are interpreted. Together, these choices constitute what is known as analytic variability — the flexibility that can fundamentally influence final conclusions.
‘Perhaps many findings rest on choices that don’t actually hold up under scrutiny. It’s worth having a closer look and rethinking how we’re doing science. Hopefully, this means that we will have more credible results in the future,’ Huber says.
The discrepancies were not due to a lack of expertise. Experienced researchers with strong statistical backgrounds were just as likely to arrive at divergent results as others. At the same time, observational studies proved less robust than experimental ones.
Christoph HuberResearch often involves choosing a single analytic path. But there are also other reasonable ways to analyse the same data
‘I personally reanalysed an experimental study and reached the same results. This may be because experimental data allow less room for analysis-dependent, open-ended conclusions,’ Huber notes.
We need humans more than machines
One way to increase the reliability of research findings is a so-called multiverse and multi-analysts analysis approach, in which the same dataset is analysed by different teams using different methodological choices, rather than relying on a single team and a single set of analytic decisions.
‘Multiverse and multi-analysts analyses can help show which results are robust regardless of the methods used. This also means that the social side of science becomes even more important: researchers should discuss different approaches and challenge one another,’ Glerean says.
Glerean also notes that popular generative AI tools are exerting an ever-greater influence on everything, and it is often trained on vast amounts of old material. This means it also draws on findings that have been shown to be irreproducible, misleading or simply wrong.
‘Science advances faster, and it self-corrects better than large language models trained on everything that was ever written. We will still need humans more than machines when it comes to understanding what counts as the best explanation of a phenomenon,’ Glerean says.
Original news article: Large-Scale Collaboration Releases New Findings on Research Credibility
Nature: Investigating the analytical robustness of the social and behavioural sciences
Contact information:
Department of Finance
The Department of Finance at Aalto University School of Business is one of the leading finance departments in Europe.
Read more news
Positive communication and improvisation help build students’ communication skills to meet employer needs
The School of Business redesigned its mandatory first-year communication course
Startups succeed despite headwinds – revenue of Aalto Startup Center’s high-growth companies increased by 355% in three years
Growth companies emerging from Aalto Startup Center continue to expand strongly despite a challenging economic environment. The combined revenue of the fastest-growing companies increased to EUR 56.95 million in 2024.
Fragmented phone use — not total screen time — is the main driver of information overload, study finds
Amid hot discussion on screen time, social media use and the impact of digital devices on our well-being, a seven-month study from Aalto University in Finland sheds new light on what overwhelms users the most –– and the results aren’t what you might think.