Publications

2024

1. arXiv

   

   POWQMIX: Weighted Value Factorization with Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning

   Chang Huang*, Shatong Zhu*, Junqiao Zhao, Hongtu Zhou, Chen Ye, Tiantian Feng, and Changjun Jiang

   In arXiv preprint arXiv:2405.08036. Under review, 2024

   Abs PDF

   Value function factorization methods are commonly used in cooperative multi-agent reinforcement learning, with QMIX receiving significant attention. Many QMIX-based methods introduce monotonicity constraints between the joint action value and individual action values to achieve decentralized execution. However, such constraints limit the representation capacity of value factorization, restricting the joint action values it can represent and hindering the learning of the optimal policy. To address this challenge, we propose the Potentially Optimal Joint Actions Weighted QMIX (POWQMIX) algorithm, which recognizes the potentially optimal joint actions and assigns higher weights to the corresponding losses of these joint actions during training. We theoretically prove that with such a weighted training approach the optimal policy is guaranteed to be recovered. Experiments in matrix games, difficulty-enhanced predator-prey, and StarCraft II Multi-Agent Challenge environments demonstrate that our algorithm outperforms the state-of-the-art value-based multi-agent reinforcement learning methods.

2. IEEE

   Coordinated Optimization and Configuration Optimization of Wind, Photovoltaics and Energy Storage based on Particle Swarm Optimization Algorithm

   Shatong Zhu, and Yifan Zhang

   In 2024 5th International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE), 2024

   Abs DOI

   We developed the optimal solutions for wind-solar hybrid systems, integrating generation, consumption, and storage strategies in specific industrial park scenarios, reducing electricity costs and minimizing generation waste, and improving power supply reliability and stability.

3. NeurIPS Spotlight

   

   Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning

   Lanqing Li*, Hai Zhang*, Xinyu Zhang, Shatong Zhu, Yang YU, Junqiao Zhao, and Pheng-Ann Heng

   In The 38th Annual Conference on Neural Information Processing Systems, 2024

   Spotlight Presentation [Top 1%]

   Awarded Abs PDF Code Poster

   Spotlight

   As a marriage between offline RL and meta-RL, the advent of offline meta-reinforcement learning (OMRL) has shown great promise in enabling RL agents to multi-task and quickly adapt while acquiring knowledge safely. Among which, context-based OMRL (COMRL) as a popular paradigm, aims to learn a universal policy conditioned on effective task representations. In this work, by examining several key milestones in the field of COMRL, we propose to integrate these seemingly independent methodologies into a unified framework. Most importantly, we show that the pre-existing COMRL algorithms are essentially optimizing the same mutual information objective between the task variable M and its latent representation Z by implementing various approximate bounds. Such theoretical insight offers ample design freedom for novel algorithms. As demonstrations, we propose a supervised and a self-supervised implementation of I(Z;M), and empirically show that the corresponding optimization algorithms exhibit remarkable generalization across a broad spectrum of RL benchmarks, context shift scenarios, data qualities and deep learning architectures. This work lays the information theoretic foundation for COMRL methods, leading to a better understanding of task representation learning in the context of reinforcement learning.