人机对抗中的博弈学习方法

Recent development in the field of human-computer gaming, one of the frontiers in artificial intelligence(AI), has witnessed a series of breakthroughs, such as AlphaGo and DeepStack beat professional human players in Go and heads-up no-limit Texas Hold'em, respectively. Such successes demonstrate synergistic interactions between game theory and machine learning. Game theory is a theoretical framework that deals with strategic interactions among multiple rational players. Combined with machine learning, it is well suited for modeling, analyzing, and solving decision-making problems in human-computer gaming tasks that often involve two or more decision-makers. Game theory based learning methods thus receive increasing attention in recent years. Besides the popular multi-agent reinforcement learning approaches, there are some other game theory based learning methods, i.e., game-theoretic learning methods, that are designed to converge to equilibria and can be dated back to the famous fictitious play proposed in 1951. In this paper, we give a selective overview of such game-theoretic learning methods in human-computer gaming. By analyzing key progresses in the field of human-computer gaming and game theory (including game-theoretic learning), we obtain a research framework for game-theoretic learning in human-computer gaming. In this framework, the role of game theory and machine learning each plays is identified: game theory provides models of strategic interactions and defines associated learning objectives (i.e., solution concepts) while machine learning helps give rise to stable, efficient, and scalable game solving algorithms. In detail, we first review important progresses in the field of human-computer gaming and game theory. Then, we introduce the definition of game-theoretic learning in human-computer gaming and compare it with traditional machine learning methods such as supervised learning and single-agent reinforcement learning. After that, we elaborate on its research framework. Intuitively, this research framework equivalently or approximately transforms the problem of achieving a good performance in a class of human-computer gaming tasks into the problem of solving a class of games. As we summarize, such transformation usually takes three basic steps: game model formulation, solution concept definition, and game solution computation. Employing this framework, we also analyze a recent game-theoretic learning algorithm that combines fictitious play and deep reinforcement learning called neural fictitious self-play, and also three milestones in the field of human-computer gaming, i.e., AlphaGo Zero, Libratus, and AlphaStar.At the end, we point out possible problems and challenges in the future research of game-theoretic learning in human-computer gaming, such as the definition of learning objectives in general-sum games, the interpretability of game-theoretic learning algorithms based on deep neural networks, the design of diverse environment suitable for game-theoretic learning, and the efficient solving of complex large-scale games that may exhibit non-transitive game behaviors. We believe that the research framework of game-theoretic learning in human-computer gaming offers guidance for the future development of human-computer gaming, and it also provides new perspectives on the development of artificial general intelligence.