Department of Computer Science: MSc Thesis Presentations

Improving the accuracy and efficiency of generative agents in external systems

Author: Antti Tiusanen
Supervisor: Juho Rousu

Abstract: Over the recent years the utilization of large language models (LLMs) has seen an increase in various applications. The use of pre-trained generative models has been increasing in both commercial and personal environments to a point where over a third of the general population in some countries use generative AI (GenAI) in their everyday lives. 

With the vast amount of generative AI adoption, the challenge of ensuring accuracy, efficiency, and security among other things arises. This issue becomes increasingly apparent in systems which use data that has not been used in the training of the GenAI model as the large language model that the GenAI uses is dependent on the data used in the training process. In these external systems the data is independent of the model training often due to security and privacy related reasons. This research provides a benchmarking structure for automatically evaluating the efficiency and accuracy of four generative agent system variants including word embedding, automated function calling, and multi-agent systems. The results focus on the implications and future improvements for benchmarking generative agent systems, with conclusions  drawn on definite evaluation metrics for a specific external system as well as LLM-as-a-judge evaluation.