Off-Policy Differentiable Logic Reinforcement Learning

Li Zhang; Xin Li; Mingzhong Wang; Andong Tian

doi:10.1007/978-3-030-86520-7_38

Off-Policy Differentiable Logic Reinforcement Learning

Li Zhang
, Xin Li^*
, Mingzhong Wang
, Andong Tian

^*Corresponding author for this work

Beijing Institute of Technology
University of the Sunshine Coast
Ubisoft China AI & Data Lab

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

9 Citations (Scopus)

Abstract

In this paper, we proposed an Off-Policy Differentiable Logic Reinforcement Learning (OPDLRL) framework to inherit the benefits of interpretability and generalization ability in Differentiable Inductive Logic Programming (DILP) and also resolves its weakness of execution efficiency, stability, and scalability. The key contributions include the use of approximate inference to significantly reduce the number of logic rules in the deduction process, an off-policy training method to enable approximate inference, and a distributed and hierarchical training framework. Extensive experiments, specifically playing real-time video games in Rabbids against human players, show that OPDLRL has better or similar performance as other DILP-based methods but far more practical in terms of sample efficiency and execution efficiency, making it applicable to complex and (near) real-time domains.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Proceedings
Editors	Nuria Oliver, Fernando Pérez-Cruz, Stefan Kramer, Jesse Read, Jose A. Lozano
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	617-632
Number of pages	16
ISBN (Print)	9783030865191
DOIs	https://doi.org/10.1007/978-3-030-86520-7_38
Publication status	Published - 2021
Externally published	Yes
Event	21st Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2021 - Virtual, Online Duration: 13 Sept 2021 → 17 Sept 2021

Publication series

Name	Lecture Notes in Computer Science
Volume	12976 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	21st Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2021
City	Virtual, Online
Period	13/09/21 → 17/09/21

Keywords

Deep reinforcement learning
Interpretable reinforcement learning
Neural-Symbolic AI

Access to Document

10.1007/978-3-030-86520-7_38

Cite this

Zhang, L., Li, X., Wang, M., & Tian, A. (2021). Off-Policy Differentiable Logic Reinforcement Learning. In N. Oliver, F. Pérez-Cruz, S. Kramer, J. Read, & J. A. Lozano (Eds.), Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Proceedings (pp. 617-632). (Lecture Notes in Computer Science; Vol. 12976 LNAI). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-86520-7_38

Zhang, Li ; Li, Xin ; Wang, Mingzhong et al. / Off-Policy Differentiable Logic Reinforcement Learning. Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Proceedings. editor / Nuria Oliver ; Fernando Pérez-Cruz ; Stefan Kramer ; Jesse Read ; Jose A. Lozano. Springer Science and Business Media Deutschland GmbH, 2021. pp. 617-632 (Lecture Notes in Computer Science).

@inproceedings{2b2915d23dc648f58b2813dc9b0a5ade,

title = "Off-Policy Differentiable Logic Reinforcement Learning",

abstract = "In this paper, we proposed an Off-Policy Differentiable Logic Reinforcement Learning (OPDLRL) framework to inherit the benefits of interpretability and generalization ability in Differentiable Inductive Logic Programming (DILP) and also resolves its weakness of execution efficiency, stability, and scalability. The key contributions include the use of approximate inference to significantly reduce the number of logic rules in the deduction process, an off-policy training method to enable approximate inference, and a distributed and hierarchical training framework. Extensive experiments, specifically playing real-time video games in Rabbids against human players, show that OPDLRL has better or similar performance as other DILP-based methods but far more practical in terms of sample efficiency and execution efficiency, making it applicable to complex and (near) real-time domains.",

keywords = "Deep reinforcement learning, Interpretable reinforcement learning, Neural-Symbolic AI",

author = "Li Zhang and Xin Li and Mingzhong Wang and Andong Tian",

note = "Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.; 21st Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2021 ; Conference date: 13-09-2021 Through 17-09-2021",

year = "2021",

doi = "10.1007/978-3-030-86520-7\_38",

language = "English",

isbn = "9783030865191",

series = "Lecture Notes in Computer Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "617--632",

editor = "Nuria Oliver and Fernando P{\'e}rez-Cruz and Stefan Kramer and Jesse Read and Lozano, \{Jose A.\}",

booktitle = "Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Proceedings",

address = "Germany",

}

Zhang, L, Li, X, Wang, M & Tian, A 2021, Off-Policy Differentiable Logic Reinforcement Learning. in N Oliver, F Pérez-Cruz, S Kramer, J Read & JA Lozano (eds), Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Proceedings. Lecture Notes in Computer Science, vol. 12976 LNAI, Springer Science and Business Media Deutschland GmbH, pp. 617-632, 21st Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2021, Virtual, Online, 13/09/21. https://doi.org/10.1007/978-3-030-86520-7_38

Off-Policy Differentiable Logic Reinforcement Learning. / Zhang, Li; Li, Xin; Wang, Mingzhong et al.
Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Proceedings. ed. / Nuria Oliver; Fernando Pérez-Cruz; Stefan Kramer; Jesse Read; Jose A. Lozano. Springer Science and Business Media Deutschland GmbH, 2021. p. 617-632 (Lecture Notes in Computer Science; Vol. 12976 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Off-Policy Differentiable Logic Reinforcement Learning

AU - Zhang, Li

AU - Li, Xin

AU - Wang, Mingzhong

AU - Tian, Andong

PY - 2021

Y1 - 2021

N2 - In this paper, we proposed an Off-Policy Differentiable Logic Reinforcement Learning (OPDLRL) framework to inherit the benefits of interpretability and generalization ability in Differentiable Inductive Logic Programming (DILP) and also resolves its weakness of execution efficiency, stability, and scalability. The key contributions include the use of approximate inference to significantly reduce the number of logic rules in the deduction process, an off-policy training method to enable approximate inference, and a distributed and hierarchical training framework. Extensive experiments, specifically playing real-time video games in Rabbids against human players, show that OPDLRL has better or similar performance as other DILP-based methods but far more practical in terms of sample efficiency and execution efficiency, making it applicable to complex and (near) real-time domains.

AB - In this paper, we proposed an Off-Policy Differentiable Logic Reinforcement Learning (OPDLRL) framework to inherit the benefits of interpretability and generalization ability in Differentiable Inductive Logic Programming (DILP) and also resolves its weakness of execution efficiency, stability, and scalability. The key contributions include the use of approximate inference to significantly reduce the number of logic rules in the deduction process, an off-policy training method to enable approximate inference, and a distributed and hierarchical training framework. Extensive experiments, specifically playing real-time video games in Rabbids against human players, show that OPDLRL has better or similar performance as other DILP-based methods but far more practical in terms of sample efficiency and execution efficiency, making it applicable to complex and (near) real-time domains.

KW - Deep reinforcement learning

KW - Interpretable reinforcement learning

KW - Neural-Symbolic AI

UR - https://www.scopus.com/pages/publications/85115693688

U2 - 10.1007/978-3-030-86520-7_38

DO - 10.1007/978-3-030-86520-7_38

M3 - Conference contribution

AN - SCOPUS:85115693688

SN - 9783030865191

T3 - Lecture Notes in Computer Science

SP - 617

EP - 632

BT - Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Proceedings

A2 - Oliver, Nuria

A2 - Pérez-Cruz, Fernando

A2 - Kramer, Stefan

A2 - Read, Jesse

A2 - Lozano, Jose A.

PB - Springer Science and Business Media Deutschland GmbH

T2 - 21st Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2021

Y2 - 13 September 2021 through 17 September 2021

ER -

Zhang L, Li X, Wang M, Tian A. Off-Policy Differentiable Logic Reinforcement Learning. In Oliver N, Pérez-Cruz F, Kramer S, Read J, Lozano JA, editors, Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Proceedings. Springer Science and Business Media Deutschland GmbH. 2021. p. 617-632. (Lecture Notes in Computer Science). doi: 10.1007/978-3-030-86520-7_38

Off-Policy Differentiable Logic Reinforcement Learning

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this