MARPO: A Reflective Policy Optimization for Multi-Agent Reinforcement Learning

Cuiling Wu; Yaozhong Gan; Junliang Xing; Ying Fu

doi:10.1609/aaai.v40i35.40219

MARPO: A Reflective Policy Optimization for Multi-Agent Reinforcement Learning

Cuiling Wu
, Yaozhong Gan
, Junliang Xing^*
, Ying Fu^*

^*Corresponding author for this work

Beijing Institute of Technology
Qiyuan Lab

Research output: Contribution to journal › Conference article › peer-review

Abstract

We propose Multi-Agent Reflective Policy Optimization (MARPO) to alleviate the issue of sample inefficiency in multi-agent reinforcement learning. MARPO consists of two key components: a reflection mechanism that leverages subsequent trajectories to enhance sample efficiency, and an asymmetric clipping mechanism that is derived from the KL divergence and dynamically adjusts the clipping range to improve training stability. We evaluate MARPO in classic multi-agent environments, where it consistently outperforms other methods.

Original language	English
Pages (from-to)	29740-29748
Number of pages	9
Journal	Proceedings of the AAAI Conference on Artificial Intelligence
Volume	40
Issue number	35
DOIs	https://doi.org/10.1609/aaai.v40i35.40219
Publication status	Published - 2026
Externally published	Yes
Event	40th AAAI Conference on Artificial Intelligence, AAAI 2026 - Singapore, Singapore Duration: 20 Jan 2026 → 27 Jan 2026

Access to Document

10.1609/aaai.v40i35.40219

Cite this

@article{b1bfd55a71174e08ab5d9278a54e659f,

title = "MARPO: A Reflective Policy Optimization for Multi-Agent Reinforcement Learning",

abstract = "We propose Multi-Agent Reflective Policy Optimization (MARPO) to alleviate the issue of sample inefficiency in multi-agent reinforcement learning. MARPO consists of two key components: a reflection mechanism that leverages subsequent trajectories to enhance sample efficiency, and an asymmetric clipping mechanism that is derived from the KL divergence and dynamically adjusts the clipping range to improve training stability. We evaluate MARPO in classic multi-agent environments, where it consistently outperforms other methods.",

author = "Cuiling Wu and Yaozhong Gan and Junliang Xing and Ying Fu",

note = "Publisher Copyright: {\textcopyright} 2026, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.; 40th AAAI Conference on Artificial Intelligence, AAAI 2026 ; Conference date: 20-01-2026 Through 27-01-2026",

year = "2026",

doi = "10.1609/aaai.v40i35.40219",

language = "English",

volume = "40",

pages = "29740--29748",

journal = "Proceedings of the AAAI Conference on Artificial Intelligence",

issn = "2159-5399",

publisher = "Association for the Advancement of Artificial Intelligence",

number = "35",

}

TY - JOUR

T1 - MARPO

T2 - 40th AAAI Conference on Artificial Intelligence, AAAI 2026

AU - Wu, Cuiling

AU - Gan, Yaozhong

AU - Xing, Junliang

AU - Fu, Ying

PY - 2026

Y1 - 2026

N2 - We propose Multi-Agent Reflective Policy Optimization (MARPO) to alleviate the issue of sample inefficiency in multi-agent reinforcement learning. MARPO consists of two key components: a reflection mechanism that leverages subsequent trajectories to enhance sample efficiency, and an asymmetric clipping mechanism that is derived from the KL divergence and dynamically adjusts the clipping range to improve training stability. We evaluate MARPO in classic multi-agent environments, where it consistently outperforms other methods.

AB - We propose Multi-Agent Reflective Policy Optimization (MARPO) to alleviate the issue of sample inefficiency in multi-agent reinforcement learning. MARPO consists of two key components: a reflection mechanism that leverages subsequent trajectories to enhance sample efficiency, and an asymmetric clipping mechanism that is derived from the KL divergence and dynamically adjusts the clipping range to improve training stability. We evaluate MARPO in classic multi-agent environments, where it consistently outperforms other methods.

UR - https://www.scopus.com/pages/publications/105034244630

U2 - 10.1609/aaai.v40i35.40219

DO - 10.1609/aaai.v40i35.40219

M3 - Conference article

AN - SCOPUS:105034244630

SN - 2159-5399

VL - 40

SP - 29740

EP - 29748

JO - Proceedings of the AAAI Conference on Artificial Intelligence

JF - Proceedings of the AAAI Conference on Artificial Intelligence

IS - 35

Y2 - 20 January 2026 through 27 January 2026

ER -

MARPO: A Reflective Policy Optimization for Multi-Agent Reinforcement Learning

Abstract

Access to Document

Other files and links

Fingerprint

Cite this