Cached Embedding with Random Selection: Optimization Technique to Improve Training Speed of Character-Aware Embedding

Yaofei Yang; Hua Ping Zhang; Linfang Wu; Xin Liu; Yangsen Zhang

doi:10.1007/978-3-030-41964-6_5

Cached Embedding with Random Selection: Optimization Technique to Improve Training Speed of Character-Aware Embedding

Yaofei Yang, Hua Ping Zhang^*, Linfang Wu, Xin Liu, Yangsen Zhang

^*Corresponding author for this work

School of Computer Science and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Embedding is widely used in most natural language processing. e.g., neural machine translation, text classification, text abstraction and sentiment analysis etc. Word-based embedding is faster and character-based embedding performs better. In this paper, we explore a way to combine these two embeddings to bridge the gap between word-based and character-based embedding in speed and performance. In the experiments and analysis of Hybrid Embedding, we found it’s difficult to make these two different embeddings generate the same embedding vector, but we still obtain a comparable result. According to the results of analysis, we explore a form of character-based embedding called Cached Embedding that can achieve almost the same performance and reduce the extra training time by almost half compared to character-based embedding.

Original language	English
Title of host publication	Intelligent Information and Database Systems - 12th Asian Conference, ACIIDS 2020, Proceedings
Editors	Ngoc Thanh Nguyen, Bogdan Trawinski, Kietikul Jearanaitanakij, Suphamit Chittayasothorn, Ali Selamat
Publisher	Springer
Pages	51-62
Number of pages	12
ISBN (Print)	9783030419639
DOIs	https://doi.org/10.1007/978-3-030-41964-6_5
Publication status	Published - 2020
Event	12th Asian Conference on Intelligent Information and Database Systems, ACIIDS 2020 - Phuket, Thailand Duration: 23 Mar 2020 → 26 Mar 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12033 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	12th Asian Conference on Intelligent Information and Database Systems, ACIIDS 2020
Country/Territory	Thailand
City	Phuket
Period	23/03/20 → 26/03/20

Keywords

Cached Embedding
Char-aware embedding
Linguist
Natural language processing
Time reduction
Training speed
Word embedding

Access to Document

10.1007/978-3-030-41964-6_5

Cite this

Yang, Y., Zhang, H. P., Wu, L., Liu, X., & Zhang, Y. (2020). Cached Embedding with Random Selection: Optimization Technique to Improve Training Speed of Character-Aware Embedding. In N. T. Nguyen, B. Trawinski, K. Jearanaitanakij, S. Chittayasothorn, & A. Selamat (Eds.), Intelligent Information and Database Systems - 12th Asian Conference, ACIIDS 2020, Proceedings (pp. 51-62). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12033 LNAI). Springer. https://doi.org/10.1007/978-3-030-41964-6_5

Yang, Yaofei ; Zhang, Hua Ping ; Wu, Linfang et al. / Cached Embedding with Random Selection : Optimization Technique to Improve Training Speed of Character-Aware Embedding. Intelligent Information and Database Systems - 12th Asian Conference, ACIIDS 2020, Proceedings. editor / Ngoc Thanh Nguyen ; Bogdan Trawinski ; Kietikul Jearanaitanakij ; Suphamit Chittayasothorn ; Ali Selamat. Springer, 2020. pp. 51-62 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{afd587ab739241dda0b4cd57d78b723c,

title = "Cached Embedding with Random Selection: Optimization Technique to Improve Training Speed of Character-Aware Embedding",

abstract = "Embedding is widely used in most natural language processing. e.g., neural machine translation, text classification, text abstraction and sentiment analysis etc. Word-based embedding is faster and character-based embedding performs better. In this paper, we explore a way to combine these two embeddings to bridge the gap between word-based and character-based embedding in speed and performance. In the experiments and analysis of Hybrid Embedding, we found it{\textquoteright}s difficult to make these two different embeddings generate the same embedding vector, but we still obtain a comparable result. According to the results of analysis, we explore a form of character-based embedding called Cached Embedding that can achieve almost the same performance and reduce the extra training time by almost half compared to character-based embedding.",

keywords = "Cached Embedding, Char-aware embedding, Linguist, Natural language processing, Time reduction, Training speed, Word embedding",

author = "Yaofei Yang and Zhang, {Hua Ping} and Linfang Wu and Xin Liu and Yangsen Zhang",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 12th Asian Conference on Intelligent Information and Database Systems, ACIIDS 2020 ; Conference date: 23-03-2020 Through 26-03-2020",

year = "2020",

doi = "10.1007/978-3-030-41964-6_5",

language = "English",

isbn = "9783030419639",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "51--62",

editor = "Nguyen, {Ngoc Thanh} and Bogdan Trawinski and Kietikul Jearanaitanakij and Suphamit Chittayasothorn and Ali Selamat",

booktitle = "Intelligent Information and Database Systems - 12th Asian Conference, ACIIDS 2020, Proceedings",

address = "Germany",

}

Yang, Y, Zhang, HP, Wu, L, Liu, X & Zhang, Y 2020, Cached Embedding with Random Selection: Optimization Technique to Improve Training Speed of Character-Aware Embedding. in NT Nguyen, B Trawinski, K Jearanaitanakij, S Chittayasothorn & A Selamat (eds), Intelligent Information and Database Systems - 12th Asian Conference, ACIIDS 2020, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12033 LNAI, Springer, pp. 51-62, 12th Asian Conference on Intelligent Information and Database Systems, ACIIDS 2020, Phuket, Thailand, 23/03/20. https://doi.org/10.1007/978-3-030-41964-6_5

Cached Embedding with Random Selection: Optimization Technique to Improve Training Speed of Character-Aware Embedding. / Yang, Yaofei; Zhang, Hua Ping; Wu, Linfang et al.
Intelligent Information and Database Systems - 12th Asian Conference, ACIIDS 2020, Proceedings. ed. / Ngoc Thanh Nguyen; Bogdan Trawinski; Kietikul Jearanaitanakij; Suphamit Chittayasothorn; Ali Selamat. Springer, 2020. p. 51-62 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12033 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Cached Embedding with Random Selection

T2 - 12th Asian Conference on Intelligent Information and Database Systems, ACIIDS 2020

AU - Yang, Yaofei

AU - Zhang, Hua Ping

AU - Wu, Linfang

AU - Liu, Xin

AU - Zhang, Yangsen

PY - 2020

Y1 - 2020

N2 - Embedding is widely used in most natural language processing. e.g., neural machine translation, text classification, text abstraction and sentiment analysis etc. Word-based embedding is faster and character-based embedding performs better. In this paper, we explore a way to combine these two embeddings to bridge the gap between word-based and character-based embedding in speed and performance. In the experiments and analysis of Hybrid Embedding, we found it’s difficult to make these two different embeddings generate the same embedding vector, but we still obtain a comparable result. According to the results of analysis, we explore a form of character-based embedding called Cached Embedding that can achieve almost the same performance and reduce the extra training time by almost half compared to character-based embedding.

AB - Embedding is widely used in most natural language processing. e.g., neural machine translation, text classification, text abstraction and sentiment analysis etc. Word-based embedding is faster and character-based embedding performs better. In this paper, we explore a way to combine these two embeddings to bridge the gap between word-based and character-based embedding in speed and performance. In the experiments and analysis of Hybrid Embedding, we found it’s difficult to make these two different embeddings generate the same embedding vector, but we still obtain a comparable result. According to the results of analysis, we explore a form of character-based embedding called Cached Embedding that can achieve almost the same performance and reduce the extra training time by almost half compared to character-based embedding.

KW - Cached Embedding

KW - Char-aware embedding

KW - Linguist

KW - Natural language processing

KW - Time reduction

KW - Training speed

KW - Word embedding

UR - http://www.scopus.com/inward/record.url?scp=85082295056&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-41964-6_5

DO - 10.1007/978-3-030-41964-6_5

M3 - Conference contribution

AN - SCOPUS:85082295056

SN - 9783030419639

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 51

EP - 62

BT - Intelligent Information and Database Systems - 12th Asian Conference, ACIIDS 2020, Proceedings

A2 - Nguyen, Ngoc Thanh

A2 - Trawinski, Bogdan

A2 - Jearanaitanakij, Kietikul

A2 - Chittayasothorn, Suphamit

A2 - Selamat, Ali

PB - Springer

Y2 - 23 March 2020 through 26 March 2020

ER -

Yang Y, Zhang HP, Wu L, Liu X, Zhang Y. Cached Embedding with Random Selection: Optimization Technique to Improve Training Speed of Character-Aware Embedding. In Nguyen NT, Trawinski B, Jearanaitanakij K, Chittayasothorn S, Selamat A, editors, Intelligent Information and Database Systems - 12th Asian Conference, ACIIDS 2020, Proceedings. Springer. 2020. p. 51-62. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-41964-6_5

Cached Embedding with Random Selection: Optimization Technique to Improve Training Speed of Character-Aware Embedding

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this