Video Captioning with Semantic Information from the Knowledge Base

Dan Wang; Dandan Song

doi:10.1109/ICBK.2017.26

Video Captioning with Semantic Information from the Knowledge Base

Dan Wang, Dandan Song

School of Computer Science and Technology

Beijing Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Citations (Scopus)

Abstract

Generating video description is a very challenging task due to the complex spatiotemporal information. Recently, many methods have been proposed by utilizing LSTM to generate sentence for video. Inspired by recent work in machine translation and object detection, we propose a new approach for video captioning which aims to incorporate Knowledge Base information with frame features of the video. We compare and analyze our approach with prior work and show that the large volumes information is available to generate video description. We experiment with our ideas on the S2VT model, and we demonstrate that our method outperforms the state-of-the-art on video captioning benchmarks.

Original language	English
Title of host publication	Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017
Editors	Ruqian Lu, Xindong Wu, Tamer Ozsu, Xindong Wu, Jim Hendler
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	224-229
Number of pages	6
ISBN (Electronic)	9781538631195
DOIs	https://doi.org/10.1109/ICBK.2017.26
Publication status	Published - 30 Aug 2017
Event	2017 IEEE International Conference on Big Knowledge, ICBK 2017 - Hefei, China Duration: 9 Aug 2017 → 10 Aug 2017

Publication series

Name	Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017

Conference

Conference	2017 IEEE International Conference on Big Knowledge, ICBK 2017
Country/Territory	China
City	Hefei
Period	9/08/17 → 10/08/17

Keywords

Video description
knowledge base
object detection

Access to Document

10.1109/ICBK.2017.26

Cite this

Wang, D., & Song, D. (2017). Video Captioning with Semantic Information from the Knowledge Base. In R. Lu, X. Wu, T. Ozsu, X. Wu, & J. Hendler (Eds.), Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017 (pp. 224-229). Article 8023421 (Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICBK.2017.26

Wang, Dan ; Song, Dandan. / Video Captioning with Semantic Information from the Knowledge Base. Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017. editor / Ruqian Lu ; Xindong Wu ; Tamer Ozsu ; Xindong Wu ; Jim Hendler. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 224-229 (Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017).

@inproceedings{989d9815cece4a5daba1c9fe8f1b0564,

title = "Video Captioning with Semantic Information from the Knowledge Base",

abstract = "Generating video description is a very challenging task due to the complex spatiotemporal information. Recently, many methods have been proposed by utilizing LSTM to generate sentence for video. Inspired by recent work in machine translation and object detection, we propose a new approach for video captioning which aims to incorporate Knowledge Base information with frame features of the video. We compare and analyze our approach with prior work and show that the large volumes information is available to generate video description. We experiment with our ideas on the S2VT model, and we demonstrate that our method outperforms the state-of-the-art on video captioning benchmarks.",

keywords = "Video description, knowledge base, object detection",

author = "Dan Wang and Dandan Song",

note = "Publisher Copyright: {\textcopyright} 2017 IEEE.; 2017 IEEE International Conference on Big Knowledge, ICBK 2017 ; Conference date: 09-08-2017 Through 10-08-2017",

year = "2017",

month = aug,

day = "30",

doi = "10.1109/ICBK.2017.26",

language = "English",

series = "Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "224--229",

editor = "Ruqian Lu and Xindong Wu and Tamer Ozsu and Xindong Wu and Jim Hendler",

booktitle = "Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017",

address = "United States",

}

Wang, D & Song, D 2017, Video Captioning with Semantic Information from the Knowledge Base. in R Lu, X Wu, T Ozsu, X Wu & J Hendler (eds), Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017., 8023421, Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017, Institute of Electrical and Electronics Engineers Inc., pp. 224-229, 2017 IEEE International Conference on Big Knowledge, ICBK 2017, Hefei, China, 9/08/17. https://doi.org/10.1109/ICBK.2017.26

Video Captioning with Semantic Information from the Knowledge Base. / Wang, Dan; Song, Dandan.
Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017. ed. / Ruqian Lu; Xindong Wu; Tamer Ozsu; Xindong Wu; Jim Hendler. Institute of Electrical and Electronics Engineers Inc., 2017. p. 224-229 8023421 (Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Video Captioning with Semantic Information from the Knowledge Base

AU - Wang, Dan

AU - Song, Dandan

PY - 2017/8/30

Y1 - 2017/8/30

N2 - Generating video description is a very challenging task due to the complex spatiotemporal information. Recently, many methods have been proposed by utilizing LSTM to generate sentence for video. Inspired by recent work in machine translation and object detection, we propose a new approach for video captioning which aims to incorporate Knowledge Base information with frame features of the video. We compare and analyze our approach with prior work and show that the large volumes information is available to generate video description. We experiment with our ideas on the S2VT model, and we demonstrate that our method outperforms the state-of-the-art on video captioning benchmarks.

AB - Generating video description is a very challenging task due to the complex spatiotemporal information. Recently, many methods have been proposed by utilizing LSTM to generate sentence for video. Inspired by recent work in machine translation and object detection, we propose a new approach for video captioning which aims to incorporate Knowledge Base information with frame features of the video. We compare and analyze our approach with prior work and show that the large volumes information is available to generate video description. We experiment with our ideas on the S2VT model, and we demonstrate that our method outperforms the state-of-the-art on video captioning benchmarks.

KW - Video description

KW - knowledge base

KW - object detection

UR - http://www.scopus.com/inward/record.url?scp=85031748394&partnerID=8YFLogxK

U2 - 10.1109/ICBK.2017.26

DO - 10.1109/ICBK.2017.26

M3 - Conference contribution

AN - SCOPUS:85031748394

T3 - Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017

SP - 224

EP - 229

BT - Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017

A2 - Lu, Ruqian

A2 - Wu, Xindong

A2 - Ozsu, Tamer

A2 - Wu, Xindong

A2 - Hendler, Jim

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2017 IEEE International Conference on Big Knowledge, ICBK 2017

Y2 - 9 August 2017 through 10 August 2017

ER -

Wang D, Song D. Video Captioning with Semantic Information from the Knowledge Base. In Lu R, Wu X, Ozsu T, Wu X, Hendler J, editors, Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 224-229. 8023421. (Proceedings - 2017 IEEE International Conference on Big Knowledge, ICBK 2017). doi: 10.1109/ICBK.2017.26

Video Captioning with Semantic Information from the Knowledge Base

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this