School of Science
Science for tomorrow’s technology, innovations and businesses
Scientific conclusions depend on who performs the analysis
Researchers reanalysed hundreds of previously published studies in the social and behavioural sciences. Only one in three re-analyses reached the same conclusions.
A new study published in Nature, "Investigating the analytic robustness of the social and behavioural sciences," finds that scientific conclusions can shift dramatically depending on who conducts the analysis.
More than 450 independent researchers from around the world conducted over 500 re-analyses of datasets from one hundred previously published studies in the social and behavioural sciences. All analysts received the same data and the same central research question, but they were free to carry out the analysis based on their own expert judgment.
The study revealed that scientific conclusions can change substantially depending on who performs the analysis.
‘All analysts reached the same conclusion as the original authors in about one third of cases,’ says data scientist Enrico Glerean from Aalto University.
The results come from a large-scale international collaboration led by Balázs Aczél and Barnabás Szászi from Eötvös Loránd University and Corvinus University, conducted as part of the Systematizing Confidence in Open Research and Evidence (SCORE) program with altogether 865 researchers.
‘It was truly a gigantic effort, with hundreds of people re-analysing published studies to assess how replicable their findings were,’ says Glerean.
Enrico GlereanAll analysts reached the same conclusion as the original authors in about one third of cases
Discrepancies are not due to lack of expertise
Over the past decade, the social and behavioural sciences have undergone substantial reforms aimed at making research more transparent, rigorous and reliable. One important question, however, is receiving increasing attention: to what extent do research findings depend on the specific way in which data are analysed?
‘Research often involves choosing a single analytic path, where one dataset is typically analysed by one researcher or one team. But there are also other reasonable ways to analyse the same data,’ says Aalto University Assistant Professor Christoph Huber.
While peer review assesses methodological acceptability, it rarely reveals what results might have emerged under alternative, yet equally defensible, statistical decisions.
‘Sometimes researchers disagree about how certain types of data should be analysed in the first place. At other times, the phenomena themselves do not permit a single, unequivocal interpretation,’ says Enrico Glerean.
Yet empirical research involves numerous decision points: how data are cleaned, how variables are defined, which statistical models or software are used, and how results are interpreted. Together, these choices constitute what is known as analytic variability — the flexibility that can fundamentally influence final conclusions.
‘Perhaps many findings rest on choices that don’t actually hold up under scrutiny. It’s worth having a closer look and rethinking how we’re doing science. Hopefully, this means that we will have more credible results in the future,’ Huber says.
The discrepancies were not due to a lack of expertise. Experienced researchers with strong statistical backgrounds were just as likely to arrive at divergent results as others. At the same time, observational studies proved less robust than experimental ones.
Christoph HuberResearch often involves choosing a single analytic path. But there are also other reasonable ways to analyse the same data
‘I personally reanalysed an experimental study and reached the same results. This may be because experimental data allow less room for analysis-dependent, open-ended conclusions,’ Huber notes.
We need humans more than machines
One way to increase the reliability of research findings is a so-called multiverse and multi-analysts analysis approach, in which the same dataset is analysed by different teams using different methodological choices, rather than relying on a single team and a single set of analytic decisions.
‘Multiverse and multi-analysts analyses can help show which results are robust regardless of the methods used. This also means that the social side of science becomes even more important: researchers should discuss different approaches and challenge one another,’ Glerean says.
Glerean also notes that popular generative AI tools are exerting an ever-greater influence on everything, and it is often trained on vast amounts of old material. This means it also draws on findings that have been shown to be irreproducible, misleading or simply wrong.
‘Science advances faster, and it self-corrects better than large language models trained on everything that was ever written. We will still need humans more than machines when it comes to understanding what counts as the best explanation of a phenomenon,’ Glerean says.
Original news article: Large-Scale Collaboration Releases New Findings on Research Credibility
Nature: Investigating the analytical robustness of the social and behavioural sciences
Contact information:
Department of Finance
The Department of Finance at Aalto University School of Business is one of the leading finance departments in Europe.
Read more news
The School of Business’s Ceremonial Conferment of degrees will bring festive atmosphere to the streets of Töölö in May
A dignified celebration is held every five years. The public can watch the spectacular conferment procession in Töölö on the afternoon of Friday 22 May
Researchers measure energy below a zeptojoule–enough for a red blood cell to move a nanometer
A new method for measuring incredibly miniscule amounts of energy – less than a trillionth of a billionth of a joule – could give quantum computing and the hunt for dark matter a boost, while paving the way for counting individual photons.
E-scooters are here to stay – first global study maps the state of shared micromobility
Shared e-scooters, city bikes and other small vehicles have evolved from niche experiments into a standard part of urban transport worldwide. Rather than treating micromobility as a passing trend, public authorities should take an active role in shaping its development, researchers say.