Digital Cousins: Generative Multi-Environment Mixed Q-Learning

Abstract

Reinforcement learning is a popular strategy for designing policies in systems with unknown Markovian dynamics. Q-learning can be employed to solve a variety of optimization and control problems in a model-free way. Classical Q-learning suffers from high bias and estimation variance, training instability, slow convergence, and high sample complexity. We propose a new strategy for generating trajectories for policy optimization to combat these challenges.  We propose a novel on-line/in-real-time Q-learning methodology based on the creation of synthetic Markov Decision Processes (digital cousins) that run in parallel with the true system.  These cousins are statistically related, but distinct from the true system. The approach enables the ensemble learning of multiple policies which can be efficiently fused. The new learning method is distinctly different from more classical learning strategies that mix off-line collected data with real-time trajectory tracking. The proposed mixed strategy offers significantly improved convergence rates and performance.  In fact, the improvement in the learning rate is roughly proportional to the number of digital cousins. We provide theoretical results on convergence as well as the ranking of the informativeness of the synthetic environments via learning analyses based on coverage. We provide preliminary results on a partially decentralized multi-agent version of the learning approach.  This latter approach provides a good tradeoff between communication costs between the agents and overall performance. Numerical examples from wireless communication networks,  the cliff-walking game and random graphs are explored and validate the theoretical results.

Bio

Urbashi Mitra received the B.S. and the M.S. degrees from the University of California at Berkeley and her Ph.D. from Princeton University.  She began her academic career at Ohio State University.  Dr. Mitra is currently the Gordon S. Marshall Professor in Engineering at the University of Southern California with appointments in Electrical Engineering and Computer Science. Dr. Mitra is a Fellow of the IEEE, a foreign member of the Academia Europaea and a member of the USC chapter of Phi Kappa Phi.   She was the inaugural Editor-in-Chief for the IEEE Transactions on Molecular, Biological and Multi-scale Communications as well as multiple Associate editorships for IEEE transactions. She is active in service for the following IEEE societies: Signal Processing, Communications and Information Theory.  She is the recipient of: the 2025 Princeton ECE Department Distinguished Graduate Alumni Award, the 2024 IEEE Information Theory Society Aaron D. Wyner Distinguished Service Award, the 2021 USC Viterbi School of Engineering Senior Research Award, the 2017 IEEE Communications Society Women in Communications Engineering Technical Achievement Award, a 2016 UK Royal Academy of Engineering Distinguished Visiting Professorship, a 2016 US Fulbright Scholar Award, a 2016-2017 UK Leverhulme Trust Visiting Professorship,  IEEE Communications Society (2015-2016)  and Signal Processing Society (2024-2025) Distinguished Lectureships, 2012 Globecom Signal Processing for Communications Symposium Best Paper Award, 2012 US National Academy of Engineering Lillian Gilbreth Lectureship,  the 2009 DCOSS Applications & Systems Best Paper Award, 2002 Texas Instruments Visiting Professor, 2001 Okawa Foundation Award, 2000 OSU College of Engineering Lumley Award for Research, 1997 OSU College of Engineering MacQuigg Award for Teaching, and a 1996 National Science Foundation CAREER Award.   Dr. Mitra has held visiting appointments at: King’s College, London, Imperial College, the Delft University of Technology, Stanford University, Rice University, and the Eurecom Institute. Her research interests are in:  model-based machine learning, wireless communications, communication and sensor networks, biological communication systems, detection and estimation and the interface of communication, sensing and control.

Event organised in partnership with: