News

New chatbot can explain apps and show you how they access hardware or data

Hey GUI helps you find information on apps and their user interfaces with a simple conversation rather than complex tools or code.
Hey GUI is a chatbot that can be used to find images or textual information about apps and their user interfaces using natural language conversations.
Hey GUI is a chatbot that can be used to find images or textual information about apps and their user interfaces using natural language conversations.

Chatbots have already become a part of our everyday lives with their quick and intuitive way to complete tasks like scheduling and finding information using natural language conversations. Researchers at Aalto University have now harnessed the power of chatbots to help designers and developers develop new apps and allow end users to find information on the apps on their devices. 

The chatbot — ‘Hey GUI’ (pronounced goo-ee), short for Graphical User Interface, which will be presented at ACM Designing Interactive Systems 2021 on 1 July — can answer questions by showing images and screenshots of apps, or through simple text phrases.

“Hey GUI eliminates the need for coding skills or technical expertise. Anyone can use it to search for information on, for example, which one of my apps is asking for permission to use the camera, or where is the search button on this screen”, says postdoctoral researcher Kashyap Todi of Aalto University.

Important findings for chatbot developers and researchers

Different kinds of applications and user interfaces contain vast amounts of information, which can be used for different needs and tasks. In their study, the researchers set out to find the most desirable chatbot features and how users would have conversations with them.

Three target groups were considered in the experiment: designers, software developers, and general users. Over 100 designers, developers, and ordinary users responded to the survey.

“Our goal was to get a good understanding of what capabilities and functions different kinds of users would find most beneficial when using this new type of chatbot, as well as how they would formulate questions — especially what kinds of words and phrases they would use. This is an important first step towards developing chatbots that can help users find information about apps using simple conversations”, says Kashyap.

As a result of the survey, 21 key features and more than 1300 of the most common questions put forward by users were collected and analysed to create Hey GUI, allowing users to find information within user interfaces using simple conversations, based on the most frequent expressions collected in the survey.

“Designers can get inspiration, for instance, by asking Hey GUI to find good online shop designs. The chatbot responds with several screenshots from existing apps in the catalogue. Software developers could ask a chatbot what typical colour codes are used for login buttons, and the chatbot responds with several options”, Kashyap adds.

In questions from ordinary users, there was a special emphasis on matters related to privacy and accessibility information, or getting help about their apps, such as asking which apps display maps.

The research prototype is currently under development. The common questions gathered from the survey are available at the website.

The study was conducted in collaboration with the universities of Luxembourg and Bayreuth.

For more information
See the project page.
Read the full article published at ACM DIS 2021.

Watch a short video with a demonstration of the chatbot
or
watch a long video with a demonstration of the chatbot.

Further information:
Kashyap Todi
Postdoctoral researcher, Aalto University, Department of Communications and Networking
[email protected]
+358 50 4723 177

Antti Oulasvirta
Professor, Aalto University, Department of Communications and Networking
[email protected]
+358 50 3841 561

Finnish Center for Artificial Intelligence

The Finnish Center for Artificial Intelligence FCAI is a research hub initiated by Aalto University, the University of Helsinki, and the Technical Research Centre of Finland VTT. The goal of FCAI is to develop new types of artificial intelligence that can work with humans in complex environments, and help modernize Finnish industry. FCAI is one of the national flagships of the Academy of Finland.

Read more
FCAI
  • Published:
  • Updated:
Share
URL copied!

Read more news

Portraits of Kamyar Hasanzadeh and Marika Tervahartiala over an image of the Makerspace along with event name and information.
Research & Art Published:

New event: Aalto Research Data Uncovered

Join us online or in-person at the Hybrid Stage!
Tauluja taidemuseon seinällä, kuvituskuva tutkimusuutiseen, joka paljastaa kuinka kuvataiteen katselu vaikuttaa tunteisiin
Research & Art Published:

Art evokes feelings in the body

A new study from the University of Turku and Aalto University in Finland reveals that art has a powerful effect on people's bodies and emotions.
Tummaan paitaan pukeutunut vaalea lyhyttukkainen nainen seisoo kädet ristissä rinnan edessä, taustalla korkeakoulun tiloja
Appointments, Research & Art, University Published:

Professor Arja Karhumaa: ‘We need to better understand how texts are viewed’

Introducing our tenured professors: professor Arja Karhumaa explores the relationship between language and the visual form of a text –how a text looks at any given moment, and how it affects us as human beings.
Konsta Laakso and Elsi Sloan act in the short film of Oasis of Radical Wellbeing
Research & Art Published:

The short film filmed in Aalto tells about a student's difficult choice

The shooting of the short film Myötätunto III, funded by Aalto University's Oasis of Radical Wellbeing project, began in Otaniemi on March 17th. The premier is on May 30th as a part of the Green Minds, Healthy Hearts – Social Sustainability Symposium in Aalto University's Harald Herlin Learning Center.