Asynchronous Multi-agent Pareto Optimization for Diverse UAV Maneuver Strategy Generation

Tianze Zhou; Fubiao Zhang; Zhiwen Sun; Mingcheng Liu; Zhaoshun Wang

doi:10.1007/978-981-19-6613-2_120

Asynchronous Multi-agent Pareto Optimization for Diverse UAV Maneuver Strategy Generation

Tianze Zhou, Fubiao Zhang^*, Zhiwen Sun, Mingcheng Liu, Zhaoshun Wang

^*Corresponding author for this work

School of Aerospace Engineering

Beijing Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Recent advances have witnessed that Multi-Agent Reinforcement Learning (MARL) makes significant progress in Multi-UAV maneuver strategy generation. Difference from traditional MARL tasks, Multi-UAV combat scenarios are always in high dynamism and complexity, and exploring varying available maneuver strategies is necessary. In this paper, to extend the diverse maneuver strategy, we formalize the problem as the multi-objective optimization problem and propose an asynchronous Pareto-based multi-agent population optimization method. Besides, we propose the tolerance method to alleviate the Pareto front shock problem in the asynchronous Pareto optimization process. Finally, a 2V2 6-DOF UAV simulation environment is designed to evaluate the performance of the proposed methods. Experimental results show that our method can efficiently learn multiple maneuver strategies, such as counterattack and defense penetration.

Original language	English
Title of host publication	Advances in Guidance, Navigation and Control - Proceedings of 2022 International Conference on Guidance, Navigation and Control
Editors	Liang Yan, Haibin Duan, Yimin Deng, Liang Yan
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	1209-1218
Number of pages	10
ISBN (Print)	9789811966125
DOIs	https://doi.org/10.1007/978-981-19-6613-2_120
Publication status	Published - 2023
Event	International Conference on Guidance, Navigation and Control, ICGNC 2022 - Harbin, China Duration: 5 Aug 2022 → 7 Aug 2022

Publication series

Name	Lecture Notes in Electrical Engineering
Volume	845 LNEE
ISSN (Print)	1876-1100
ISSN (Electronic)	1876-1119

Conference

Conference	International Conference on Guidance, Navigation and Control, ICGNC 2022
Country/Territory	China
City	Harbin
Period	5/08/22 → 7/08/22

Keywords

Maneuver strategy generation
Multi-agent reinforcement learning
Pareto optimization

Access to Document

10.1007/978-981-19-6613-2_120

Cite this

Zhou, T., Zhang, F., Sun, Z., Liu, M., & Wang, Z. (2023). Asynchronous Multi-agent Pareto Optimization for Diverse UAV Maneuver Strategy Generation. In L. Yan, H. Duan, Y. Deng, & L. Yan (Eds.), Advances in Guidance, Navigation and Control - Proceedings of 2022 International Conference on Guidance, Navigation and Control (pp. 1209-1218). (Lecture Notes in Electrical Engineering; Vol. 845 LNEE). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-981-19-6613-2_120

Zhou, Tianze ; Zhang, Fubiao ; Sun, Zhiwen et al. / Asynchronous Multi-agent Pareto Optimization for Diverse UAV Maneuver Strategy Generation. Advances in Guidance, Navigation and Control - Proceedings of 2022 International Conference on Guidance, Navigation and Control. editor / Liang Yan ; Haibin Duan ; Yimin Deng ; Liang Yan. Springer Science and Business Media Deutschland GmbH, 2023. pp. 1209-1218 (Lecture Notes in Electrical Engineering).

@inproceedings{9dd56cadae304f48a35f050732b286c7,

title = "Asynchronous Multi-agent Pareto Optimization for Diverse UAV Maneuver Strategy Generation",

abstract = "Recent advances have witnessed that Multi-Agent Reinforcement Learning (MARL) makes significant progress in Multi-UAV maneuver strategy generation. Difference from traditional MARL tasks, Multi-UAV combat scenarios are always in high dynamism and complexity, and exploring varying available maneuver strategies is necessary. In this paper, to extend the diverse maneuver strategy, we formalize the problem as the multi-objective optimization problem and propose an asynchronous Pareto-based multi-agent population optimization method. Besides, we propose the tolerance method to alleviate the Pareto front shock problem in the asynchronous Pareto optimization process. Finally, a 2V2 6-DOF UAV simulation environment is designed to evaluate the performance of the proposed methods. Experimental results show that our method can efficiently learn multiple maneuver strategies, such as counterattack and defense penetration.",

keywords = "Maneuver strategy generation, Multi-agent reinforcement learning, Pareto optimization",

author = "Tianze Zhou and Fubiao Zhang and Zhiwen Sun and Mingcheng Liu and Zhaoshun Wang",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.; International Conference on Guidance, Navigation and Control, ICGNC 2022 ; Conference date: 05-08-2022 Through 07-08-2022",

year = "2023",

doi = "10.1007/978-981-19-6613-2_120",

language = "English",

isbn = "9789811966125",

series = "Lecture Notes in Electrical Engineering",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "1209--1218",

editor = "Liang Yan and Haibin Duan and Yimin Deng and Liang Yan",

booktitle = "Advances in Guidance, Navigation and Control - Proceedings of 2022 International Conference on Guidance, Navigation and Control",

address = "Germany",

}

Zhou, T, Zhang, F, Sun, Z, Liu, M & Wang, Z 2023, Asynchronous Multi-agent Pareto Optimization for Diverse UAV Maneuver Strategy Generation. in L Yan, H Duan, Y Deng & L Yan (eds), Advances in Guidance, Navigation and Control - Proceedings of 2022 International Conference on Guidance, Navigation and Control. Lecture Notes in Electrical Engineering, vol. 845 LNEE, Springer Science and Business Media Deutschland GmbH, pp. 1209-1218, International Conference on Guidance, Navigation and Control, ICGNC 2022, Harbin, China, 5/08/22. https://doi.org/10.1007/978-981-19-6613-2_120

Asynchronous Multi-agent Pareto Optimization for Diverse UAV Maneuver Strategy Generation. / Zhou, Tianze; Zhang, Fubiao; Sun, Zhiwen et al.
Advances in Guidance, Navigation and Control - Proceedings of 2022 International Conference on Guidance, Navigation and Control. ed. / Liang Yan; Haibin Duan; Yimin Deng; Liang Yan. Springer Science and Business Media Deutschland GmbH, 2023. p. 1209-1218 (Lecture Notes in Electrical Engineering; Vol. 845 LNEE).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Asynchronous Multi-agent Pareto Optimization for Diverse UAV Maneuver Strategy Generation

AU - Zhou, Tianze

AU - Zhang, Fubiao

AU - Sun, Zhiwen

AU - Liu, Mingcheng

AU - Wang, Zhaoshun

PY - 2023

Y1 - 2023

N2 - Recent advances have witnessed that Multi-Agent Reinforcement Learning (MARL) makes significant progress in Multi-UAV maneuver strategy generation. Difference from traditional MARL tasks, Multi-UAV combat scenarios are always in high dynamism and complexity, and exploring varying available maneuver strategies is necessary. In this paper, to extend the diverse maneuver strategy, we formalize the problem as the multi-objective optimization problem and propose an asynchronous Pareto-based multi-agent population optimization method. Besides, we propose the tolerance method to alleviate the Pareto front shock problem in the asynchronous Pareto optimization process. Finally, a 2V2 6-DOF UAV simulation environment is designed to evaluate the performance of the proposed methods. Experimental results show that our method can efficiently learn multiple maneuver strategies, such as counterattack and defense penetration.

AB - Recent advances have witnessed that Multi-Agent Reinforcement Learning (MARL) makes significant progress in Multi-UAV maneuver strategy generation. Difference from traditional MARL tasks, Multi-UAV combat scenarios are always in high dynamism and complexity, and exploring varying available maneuver strategies is necessary. In this paper, to extend the diverse maneuver strategy, we formalize the problem as the multi-objective optimization problem and propose an asynchronous Pareto-based multi-agent population optimization method. Besides, we propose the tolerance method to alleviate the Pareto front shock problem in the asynchronous Pareto optimization process. Finally, a 2V2 6-DOF UAV simulation environment is designed to evaluate the performance of the proposed methods. Experimental results show that our method can efficiently learn multiple maneuver strategies, such as counterattack and defense penetration.

KW - Maneuver strategy generation

KW - Multi-agent reinforcement learning

KW - Pareto optimization

UR - http://www.scopus.com/inward/record.url?scp=85151143767&partnerID=8YFLogxK

U2 - 10.1007/978-981-19-6613-2_120

DO - 10.1007/978-981-19-6613-2_120

M3 - Conference contribution

AN - SCOPUS:85151143767

SN - 9789811966125

T3 - Lecture Notes in Electrical Engineering

SP - 1209

EP - 1218

BT - Advances in Guidance, Navigation and Control - Proceedings of 2022 International Conference on Guidance, Navigation and Control

A2 - Yan, Liang

A2 - Duan, Haibin

A2 - Deng, Yimin

A2 - Yan, Liang

PB - Springer Science and Business Media Deutschland GmbH

T2 - International Conference on Guidance, Navigation and Control, ICGNC 2022

Y2 - 5 August 2022 through 7 August 2022

ER -

Zhou T, Zhang F, Sun Z, Liu M, Wang Z. Asynchronous Multi-agent Pareto Optimization for Diverse UAV Maneuver Strategy Generation. In Yan L, Duan H, Deng Y, Yan L, editors, Advances in Guidance, Navigation and Control - Proceedings of 2022 International Conference on Guidance, Navigation and Control. Springer Science and Business Media Deutschland GmbH. 2023. p. 1209-1218. (Lecture Notes in Electrical Engineering). doi: 10.1007/978-981-19-6613-2_120