From Thought to Prompt: Cognitive Design Challenges in Human-LLM Interactions
A recording of the talk can be found here:
Event is hybrid. Participants may attend in person (T2, Computer Science Building, Aalto University) or online on Zoom.
Speaker: Hari Subramonyam
Research Assistant Professor
Graduate School of Education and Computer Science, Stanford University
Large language models (LLMs) exhibit dynamic capabilities and appear to comprehend complex and ambiguous natural language prompts. However, calibrating LLM interactions is challenging for HCI designers and end-users alike. A central issue is our limited grasp of how human cognitive processes begin with a goal and form intentions for executing interface actions, a blindspot even in established interaction models such as Norman's gulfs of execution and evaluation. In this talk, I theorize how end-users 'envision' translating their goals into clear intentions and craft prompts to obtain the desired LLM response. By mapping different interaction pathways to LLM-powered interfaces, I highlight three types of gaps in envisioning LLM interactions: (1) knowing whether LLMs can accomplish the task, (2) how to instruct the LLM to do the task, and (3) how to evaluate the success of the LLM's output in meeting the goal. I then present the design and evaluation of a specific LLM interface, Spellburst, for generative art composition and highlight design affordances to prompt-based interactions. Finally, I conclude by presenting high-level recommendations for the design and use of LLM interfaces.
Hari Subramonyam is a Research Assistant Professor at the Graduate School of Education and Computer Science (by courtesy) at Stanford University. Additionally, he holds the title of Ram and Vijay Shriram Faculty Fellow at the Institute for Human-Centered AI and is a core faculty member of Stanford HCI. Hari's research is uniquely positioned at the intersection of Human-Computer Interaction (HCI) and the Learning Sciences. He studies ways to enhance human learning via AI, focusing on (1) cognitively informed design practices, (2) collaborative design with learners and educators, and (3) the creation of transformative AI-enabled learning experiences. Through his research, Hari pioneers tools and methods emphasizing ethical considerations, responsible design, and human-centric values in AI development. Hari received his Ph.D. in Information from the University of Michigan School of Information.