3D Contextual Transformer & Double Inception Network for Human Action Recognition

Enqi Liu; Kaoru Hirota; Chang Liu; Yaping Dai

doi:10.1109/CCDC58219.2023.10326469

3D Contextual Transformer & Double Inception Network for Human Action Recognition

Enqi Liu, Kaoru Hirota, Chang Liu, Yaping Dai^*

^*Corresponding author for this work

School of Automation

Beijing Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The 3D Contextual Transformer & Double Inception Network called CoTDIL-Net is proposed for human action recognition. The spatio-temporal enrichment module based on a 3D Contextual Transformer (CoT3D) is proposed for enhancing the features of adjacent frames. In addition, 3D Inception and 2D Inception are combined to form the feature extractor called DIFE for capturing short-term contextual features. Moreover, the LSTM is used to obtain long-term action change features, and a multi-stream input framework is introduced to obtain fuller contextual information. It aims to obtain multi-scale spatio-temporal features compared with single convolution methods, where CoT3D combines contextual action information, the DIFE captures short-term features while LSTM fuses long-term features. The experiments are carried out on a laptop with 32G RAM and a GeForce RTX3070 8G GPU by using the KTH dataset, and the results show a recognition accuracy of 97.2%. The obtained results indicate that the proposed CoTDIL-Net promote the convolutional structure understanding of human actions changes.

Original language	English
Title of host publication	Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1795-1800
Number of pages	6
ISBN (Electronic)	9798350334722
DOIs	https://doi.org/10.1109/CCDC58219.2023.10326469
Publication status	Published - 2023
Event	35th Chinese Control and Decision Conference, CCDC 2023 - Yichang, China Duration: 20 May 2023 → 22 May 2023

Publication series

Name	Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023

Conference

Conference	35th Chinese Control and Decision Conference, CCDC 2023
Country/Territory	China
City	Yichang
Period	20/05/23 → 22/05/23

Keywords

2D Inception
3D Inception
Contextual Transformer
Human action recognition
Long Short-Term Memory
multi-stream input

Access to Document

10.1109/CCDC58219.2023.10326469

Cite this

Liu, E., Hirota, K., Liu, C., & Dai, Y. (2023). 3D Contextual Transformer & Double Inception Network for Human Action Recognition. In Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023 (pp. 1795-1800). (Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CCDC58219.2023.10326469

@inproceedings{0928b0a0a60d42f48b0cdad4406b9ca5,

title = "3D Contextual Transformer & Double Inception Network for Human Action Recognition",

abstract = "The 3D Contextual Transformer & Double Inception Network called CoTDIL-Net is proposed for human action recognition. The spatio-temporal enrichment module based on a 3D Contextual Transformer (CoT3D) is proposed for enhancing the features of adjacent frames. In addition, 3D Inception and 2D Inception are combined to form the feature extractor called DIFE for capturing short-term contextual features. Moreover, the LSTM is used to obtain long-term action change features, and a multi-stream input framework is introduced to obtain fuller contextual information. It aims to obtain multi-scale spatio-temporal features compared with single convolution methods, where CoT3D combines contextual action information, the DIFE captures short-term features while LSTM fuses long-term features. The experiments are carried out on a laptop with 32G RAM and a GeForce RTX3070 8G GPU by using the KTH dataset, and the results show a recognition accuracy of 97.2%. The obtained results indicate that the proposed CoTDIL-Net promote the convolutional structure understanding of human actions changes.",

keywords = "2D Inception, 3D Inception, Contextual Transformer, Human action recognition, Long Short-Term Memory, multi-stream input",

author = "Enqi Liu and Kaoru Hirota and Chang Liu and Yaping Dai",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 35th Chinese Control and Decision Conference, CCDC 2023 ; Conference date: 20-05-2023 Through 22-05-2023",

year = "2023",

doi = "10.1109/CCDC58219.2023.10326469",

language = "English",

series = "Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1795--1800",

booktitle = "Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023",

address = "United States",

}

Liu, E, Hirota, K, Liu, C & Dai, Y 2023, 3D Contextual Transformer & Double Inception Network for Human Action Recognition. in Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023. Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023, Institute of Electrical and Electronics Engineers Inc., pp. 1795-1800, 35th Chinese Control and Decision Conference, CCDC 2023, Yichang, China, 20/05/23. https://doi.org/10.1109/CCDC58219.2023.10326469

3D Contextual Transformer & Double Inception Network for Human Action Recognition. / Liu, Enqi; Hirota, Kaoru; Liu, Chang et al.
Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023. Institute of Electrical and Electronics Engineers Inc., 2023. p. 1795-1800 (Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - 3D Contextual Transformer & Double Inception Network for Human Action Recognition

AU - Liu, Enqi

AU - Hirota, Kaoru

AU - Liu, Chang

AU - Dai, Yaping

PY - 2023

Y1 - 2023

N2 - The 3D Contextual Transformer & Double Inception Network called CoTDIL-Net is proposed for human action recognition. The spatio-temporal enrichment module based on a 3D Contextual Transformer (CoT3D) is proposed for enhancing the features of adjacent frames. In addition, 3D Inception and 2D Inception are combined to form the feature extractor called DIFE for capturing short-term contextual features. Moreover, the LSTM is used to obtain long-term action change features, and a multi-stream input framework is introduced to obtain fuller contextual information. It aims to obtain multi-scale spatio-temporal features compared with single convolution methods, where CoT3D combines contextual action information, the DIFE captures short-term features while LSTM fuses long-term features. The experiments are carried out on a laptop with 32G RAM and a GeForce RTX3070 8G GPU by using the KTH dataset, and the results show a recognition accuracy of 97.2%. The obtained results indicate that the proposed CoTDIL-Net promote the convolutional structure understanding of human actions changes.

AB - The 3D Contextual Transformer & Double Inception Network called CoTDIL-Net is proposed for human action recognition. The spatio-temporal enrichment module based on a 3D Contextual Transformer (CoT3D) is proposed for enhancing the features of adjacent frames. In addition, 3D Inception and 2D Inception are combined to form the feature extractor called DIFE for capturing short-term contextual features. Moreover, the LSTM is used to obtain long-term action change features, and a multi-stream input framework is introduced to obtain fuller contextual information. It aims to obtain multi-scale spatio-temporal features compared with single convolution methods, where CoT3D combines contextual action information, the DIFE captures short-term features while LSTM fuses long-term features. The experiments are carried out on a laptop with 32G RAM and a GeForce RTX3070 8G GPU by using the KTH dataset, and the results show a recognition accuracy of 97.2%. The obtained results indicate that the proposed CoTDIL-Net promote the convolutional structure understanding of human actions changes.

KW - 2D Inception

KW - 3D Inception

KW - Contextual Transformer

KW - Human action recognition

KW - Long Short-Term Memory

KW - multi-stream input

UR - http://www.scopus.com/inward/record.url?scp=85181824416&partnerID=8YFLogxK

U2 - 10.1109/CCDC58219.2023.10326469

DO - 10.1109/CCDC58219.2023.10326469

M3 - Conference contribution

AN - SCOPUS:85181824416

T3 - Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023

SP - 1795

EP - 1800

BT - Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 35th Chinese Control and Decision Conference, CCDC 2023

Y2 - 20 May 2023 through 22 May 2023

ER -

Liu E, Hirota K, Liu C, Dai Y. 3D Contextual Transformer & Double Inception Network for Human Action Recognition. In Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023. Institute of Electrical and Electronics Engineers Inc. 2023. p. 1795-1800. (Proceedings of the 35th Chinese Control and Decision Conference, CCDC 2023). doi: 10.1109/CCDC58219.2023.10326469

3D Contextual Transformer & Double Inception Network for Human Action Recognition

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this