Off-Policy Q-Learning for Infinite Horizon LQR Problem with Unknown Dynamics

Xinxing Li; Zhihong Peng; Li Liang

doi:10.1109/ISIE.2018.8433684

Off-Policy Q-Learning for Infinite Horizon LQR Problem with Unknown Dynamics

Xinxing Li, Zhihong Peng, Li Liang

School of Automation

Beijing Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Citations (Scopus)

Abstract

In this paper, a novel online Q-Iearning approach is proposed to solve the Infinite Horizon Linear Quadratic Regulator (IHLQR) problem for continuous-time (CT) linear time-invariant (LMI) systems. The proposed Q-Iearning algorithm employing off-policy reinforcement learning (RL) technology improves the exploration ability of Q-Iearning to the state space. During the learning process, the Q-Iearning algorithm can be implemented just using the data sets which just contains the information of the behavior policy and the corresponding system state, thus is data- driven. Moreover, the data sets can be used repeatedly, which is computationally efficient. A mild condition on probing noise is established to ensure the converge of the proposed Q-Iearning algorithm. Simulation results demonstrate the effectiveness of the developed algorithm.

Original language	English
Title of host publication	Proceedings - 2018 IEEE 27th International Symposium on Industrial Electronics, ISIE 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	258-263
Number of pages	6
ISBN (Print)	9781538637050
DOIs	https://doi.org/10.1109/ISIE.2018.8433684
Publication status	Published - 10 Aug 2018
Event	27th IEEE International Symposium on Industrial Electronics, ISIE 2018 - Cairns, Australia Duration: 13 Jun 2018 → 15 Jun 2018

Publication series

Name	IEEE International Symposium on Industrial Electronics
Volume	2018-June

Conference

Conference	27th IEEE International Symposium on Industrial Electronics, ISIE 2018
Country/Territory	Australia
City	Cairns
Period	13/06/18 → 15/06/18

Access to Document

10.1109/ISIE.2018.8433684

Cite this

Li, X., Peng, Z., & Liang, L. (2018). Off-Policy Q-Learning for Infinite Horizon LQR Problem with Unknown Dynamics. In Proceedings - 2018 IEEE 27th International Symposium on Industrial Electronics, ISIE 2018 (pp. 258-263). Article 8433684 (IEEE International Symposium on Industrial Electronics; Vol. 2018-June). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISIE.2018.8433684

@inproceedings{5a850979eb524660b526b1f105f27c21,

title = "Off-Policy Q-Learning for Infinite Horizon LQR Problem with Unknown Dynamics",

abstract = "In this paper, a novel online Q-Iearning approach is proposed to solve the Infinite Horizon Linear Quadratic Regulator (IHLQR) problem for continuous-time (CT) linear time-invariant (LMI) systems. The proposed Q-Iearning algorithm employing off-policy reinforcement learning (RL) technology improves the exploration ability of Q-Iearning to the state space. During the learning process, the Q-Iearning algorithm can be implemented just using the data sets which just contains the information of the behavior policy and the corresponding system state, thus is data- driven. Moreover, the data sets can be used repeatedly, which is computationally efficient. A mild condition on probing noise is established to ensure the converge of the proposed Q-Iearning algorithm. Simulation results demonstrate the effectiveness of the developed algorithm.",

author = "Xinxing Li and Zhihong Peng and Li Liang",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 27th IEEE International Symposium on Industrial Electronics, ISIE 2018 ; Conference date: 13-06-2018 Through 15-06-2018",

year = "2018",

month = aug,

day = "10",

doi = "10.1109/ISIE.2018.8433684",

language = "English",

isbn = "9781538637050",

series = "IEEE International Symposium on Industrial Electronics",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "258--263",

booktitle = "Proceedings - 2018 IEEE 27th International Symposium on Industrial Electronics, ISIE 2018",

address = "United States",

}

Li, X, Peng, Z & Liang, L 2018, Off-Policy Q-Learning for Infinite Horizon LQR Problem with Unknown Dynamics. in Proceedings - 2018 IEEE 27th International Symposium on Industrial Electronics, ISIE 2018., 8433684, IEEE International Symposium on Industrial Electronics, vol. 2018-June, Institute of Electrical and Electronics Engineers Inc., pp. 258-263, 27th IEEE International Symposium on Industrial Electronics, ISIE 2018, Cairns, Australia, 13/06/18. https://doi.org/10.1109/ISIE.2018.8433684

Off-Policy Q-Learning for Infinite Horizon LQR Problem with Unknown Dynamics. / Li, Xinxing; Peng, Zhihong; Liang, Li.
Proceedings - 2018 IEEE 27th International Symposium on Industrial Electronics, ISIE 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 258-263 8433684 (IEEE International Symposium on Industrial Electronics; Vol. 2018-June).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Off-Policy Q-Learning for Infinite Horizon LQR Problem with Unknown Dynamics

AU - Li, Xinxing

AU - Peng, Zhihong

AU - Liang, Li

PY - 2018/8/10

Y1 - 2018/8/10

N2 - In this paper, a novel online Q-Iearning approach is proposed to solve the Infinite Horizon Linear Quadratic Regulator (IHLQR) problem for continuous-time (CT) linear time-invariant (LMI) systems. The proposed Q-Iearning algorithm employing off-policy reinforcement learning (RL) technology improves the exploration ability of Q-Iearning to the state space. During the learning process, the Q-Iearning algorithm can be implemented just using the data sets which just contains the information of the behavior policy and the corresponding system state, thus is data- driven. Moreover, the data sets can be used repeatedly, which is computationally efficient. A mild condition on probing noise is established to ensure the converge of the proposed Q-Iearning algorithm. Simulation results demonstrate the effectiveness of the developed algorithm.

AB - In this paper, a novel online Q-Iearning approach is proposed to solve the Infinite Horizon Linear Quadratic Regulator (IHLQR) problem for continuous-time (CT) linear time-invariant (LMI) systems. The proposed Q-Iearning algorithm employing off-policy reinforcement learning (RL) technology improves the exploration ability of Q-Iearning to the state space. During the learning process, the Q-Iearning algorithm can be implemented just using the data sets which just contains the information of the behavior policy and the corresponding system state, thus is data- driven. Moreover, the data sets can be used repeatedly, which is computationally efficient. A mild condition on probing noise is established to ensure the converge of the proposed Q-Iearning algorithm. Simulation results demonstrate the effectiveness of the developed algorithm.

UR - http://www.scopus.com/inward/record.url?scp=85052365602&partnerID=8YFLogxK

U2 - 10.1109/ISIE.2018.8433684

DO - 10.1109/ISIE.2018.8433684

M3 - Conference contribution

AN - SCOPUS:85052365602

SN - 9781538637050

T3 - IEEE International Symposium on Industrial Electronics

SP - 258

EP - 263

BT - Proceedings - 2018 IEEE 27th International Symposium on Industrial Electronics, ISIE 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 27th IEEE International Symposium on Industrial Electronics, ISIE 2018

Y2 - 13 June 2018 through 15 June 2018

ER -

Off-Policy Q-Learning for Infinite Horizon LQR Problem with Unknown Dynamics

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this