Opponent portrait for multiagent reinforcement learning in competitive environment

Existing investigations of opponent modeling and intention inferencing cannot make clear descriptions and practical explanations of the opponent's behaviors and intentions, which may inevitably limit the applicability of them. In this work, we propose a novel approach for opponent's policy explanation and intention inference based on the behavioral portrait of opponent. Specifically, we use the multiagent deep deterministic policy gradients (MADDPG) algorithm to train the agent and opponent in the competitive environment, and collect the behavioral data of opponent based on agent's observations. Then we perform pattern segmentation and extract the opponent's behavior events via Toeplitz inverse covariance-based clustering (TICC) algorithm; hence the opponent's behavior data can be encoded into a knowledge graph, named opponent's behavior knowledge graph (OKG). Based on this, we built a question-answer system (QA system) to query and match opponent historical information in OKG, so that the agent can obtain additional experience and gradually infer the intention of opponent with the episodes of iteration. We evaluate the proposed method on the competitive scenario in multiagent particle environment (MPE). Simulation results show that the agents are able to learn better policies with opponent portrait in competitive settings.