News

Chat AIs can role-play humans in surveys and pilot studies

Synthetic data from large language models can mimic human responses in interviews and questionnaires. Research data from popular crowdsourcing platforms may now contain fake responses that cannot be reliably detected, raising the risk of poisoned data
illustration of a green chat bubble against a yello background with yellow round objects in the middle portraying a "person is writing" prompt.
Image: Volodymyr Hryshchenko on Unsplash

Studying people in human-computer interaction (HCI) research can be slow. That’s why researchers at the Finnish Center for Artificial Intelligence (FCAI) recently harnessed the power of large language models (LLMs), specifically GPT-3, to generate open-ended answers to questions about video game player experience.

These AI-generated responses were often more convincing, as rated by humans, than real responses. These synthetic interviews offer a new approach to gathering data quickly and at low cost, which may help in fast iteration and initial testing of study designs and data analysis pipelines. Any findings based on AI-generated data, however, should also be confirmed with real data.

The researchers, based at Aalto University and the University of Helsinki, discovered some subtle differences in different versions of GPT-3 that affected the diversity of AI-generated responses. But a more discouraging implication is that data from popular crowdsourcing platforms may now automatically be suspect, as AI-generated responses are hard to distinguish from real ones. Amazon’s Mechanical Turk (MTurk), for example, can host surveys or research tasks for HCI, psychology, or related scientific areas and pay users for participation, but 'now that LLMs are so easy to access, any self-reported data from the internet cannot be trusted anymore. The economic incentives can drive malicious users to employ bots and LLMs to generate high-quality fake responses,' says Aalto University Associate Professor Perttu Hämäläinen.

The implications of synthetic data for anonymity, privacy and data protection in the medical field and similar domains are clear. However, in the realm of HCI, or science more widely, synthetic interviews and artificial experiments raise questions about the trustworthiness of crowdsourcing approaches that seeks to gather user data online. 'LLMs cannot and should not replace real participants, but synthetic data may be useful for initial exploration and piloting of research ideas,' suggests Hämäläinen. 'When real data is needed, it may be time to abandon crowdsourcing platforms like Mturk.'

“Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study” was awarded Best Paper at CHI, the Conference on Human Factors in Computing Systems in late April 2023. 

Reference: Hämäläinen, P., Tavast, M. and Kunnari, A. (2023). Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. doi: 10.1145/3544548.3580688
 

This article was originally published on FCAI's website

Mikke Tavast

FCAI

Finnish Center for Artificial Intelligence (external link)

The Finnish Center for Artificial Intelligence FCAI is a research hub initiated by Aalto University, the University of Helsinki, and the Technical Research Centre of Finland VTT. The goal of FCAI is to develop new types of artificial intelligence that can work with humans in complex environments, and help modernize Finnish industry. FCAI is one of the national flagships of the Academy of Finland.

A researcher in dark clothing presenting his work in front of a classroom, gesturing towards the whiteboard and talking.

Turbo-charging AI: Collaboration with NVIDIA renewed as joint tech center marks three years

NVIDIA AI Technology Center (NVAITC) Finland has accelerated research, training and computing power in over a dozen projects where high-performance computing meets AI

News
Christian Guckelsberger in front of graffiti

Ask a scientist: How will AI affect creativity?

The impact of creative AI is unfolding before our eyes, yet we struggle to understand it. It’s the perfect time to ask researchers what they see and think.

News
X-ray of hand with fingers making OK sign

AI is transforming healthcare: 5 things to know

Regulation, validation and trust are key when AI is used in medical services, say FCAI and Aalto University experts.

News
  • Published:
  • Updated:
Share
URL copied!

Read more news

Unite! White paper - a new university open science and innovation governance model and policy for a sustainable world
Cooperation, University Published:

Unite!’s pathway for the management of open science and innovation in the European Universities Alliances

Unite! presents a new university open science and innovation governance model and five policy recommendations.
The magnetic properties of a material can affect how it interacts with light.
Press releases, Research & Art Published:

A new optical metamaterial makes true one-way glass possible

Researchers have discovered how to make a new optical metamaterial that would underpin a variety of new technologies.
Mikko Alava ja kolme muuta tutkimusryhmän jäsentä kuvattuna teknisen laitteen äärellä.
Research & Art Published:

Significant grants for science from the Finnish Cultural Foundation

A total of thirty individuals and research groups from Aalto University received grants from the Finnish Cultural Foundation for science and art. Professors Harri Lipsanen and Zhipei Sun and their team, were awarded a large grant to develop electronic components mimicking the structure and function of the human brain. Among the grantees is also Professor Mikko Alava, whose team is developing water-resistant foams as substitutes for plastic using AI-based methods.
An illustration of planting a tree on a grave and the tree's growth
Awards and Recognition Published:

Landscape architect Anu Hakola receives Lappset prize

Hakola’s Master’s thesis examines memorial trees and their significance as part of the commemoration of the deceased.