Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings

Yuanyuan Qiu; Hongzheng Li; Shen Li; Yingdi Jiang; Renfen Hu; Lijiao Yang

doi:10.1007/978-3-030-01716-3_18

Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings

Yuanyuan Qiu, Hongzheng Li, Shen Li, Yingdi Jiang, Renfen Hu^*, Lijiao Yang

^*Corresponding author for this work

School of Foreign Languages

Beijing Normal University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

55 Citations (Scopus)

Abstract

The evaluation of word embeddings has received a considerable amount of attention in recent years, but there have been some debates about whether intrinsic measures can predict the performance of downstream tasks. To investigate this question, this paper presents the first study on the correlation between results of intrinsic evaluation and extrinsic evaluation with Chinese word embeddings. We use word similarity and word analogy as the intrinsic tasks, Named Entity Recognition and Sentiment Classification as the extrinsic tasks. A variety of Chinese word embeddings trained with different corpora and context features are used in the experiments. From the data analysis, we reach some interesting conclusions: there are strong correlations between intrinsic and extrinsic evaluations, and the performance of different tasks can be affected by training corpora and context features to varying degrees.

Original language	English
Title of host publication	Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data - 17th China National Conference, CCL 2018, and 6th International Symposium, NLP-NABD 2018, Proceedings
Editors	Xiaojie Wang, Ting Liu, Maosong Sun, Zhiyuan Liu, Yang Liu
Publisher	Springer Verlag
Pages	209-221
Number of pages	13
ISBN (Print)	9783030017156
DOIs	https://doi.org/10.1007/978-3-030-01716-3_18
Publication status	Published - 2018
Event	17th China National Conference on Computational Linguistics, CCL 2018 and 6th International Symposium on Natural Language Processing Based on Naturally Annotated Big Data, NLP-NABD 2018 - Changsha, China Duration: 19 Oct 2018 → 21 Oct 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11221 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th China National Conference on Computational Linguistics, CCL 2018 and 6th International Symposium on Natural Language Processing Based on Naturally Annotated Big Data, NLP-NABD 2018
Country/Territory	China
City	Changsha
Period	19/10/18 → 21/10/18

Keywords

Extrinsic evaluation
Intrinsic evaluation
Word embedding

Access to Document

10.1007/978-3-030-01716-3_18

Cite this

Qiu, Y., Li, H., Li, S., Jiang, Y., Hu, R., & Yang, L. (2018). Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings. In X. Wang, T. Liu, M. Sun, Z. Liu, & Y. Liu (Eds.), Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data - 17th China National Conference, CCL 2018, and 6th International Symposium, NLP-NABD 2018, Proceedings (pp. 209-221). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11221 LNAI). Springer Verlag. https://doi.org/10.1007/978-3-030-01716-3_18

Qiu, Yuanyuan ; Li, Hongzheng ; Li, Shen et al. / Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings. Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data - 17th China National Conference, CCL 2018, and 6th International Symposium, NLP-NABD 2018, Proceedings. editor / Xiaojie Wang ; Ting Liu ; Maosong Sun ; Zhiyuan Liu ; Yang Liu. Springer Verlag, 2018. pp. 209-221 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{81f82a758ae34b91a42260923f6ee68c,

title = "Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings",

abstract = "The evaluation of word embeddings has received a considerable amount of attention in recent years, but there have been some debates about whether intrinsic measures can predict the performance of downstream tasks. To investigate this question, this paper presents the first study on the correlation between results of intrinsic evaluation and extrinsic evaluation with Chinese word embeddings. We use word similarity and word analogy as the intrinsic tasks, Named Entity Recognition and Sentiment Classification as the extrinsic tasks. A variety of Chinese word embeddings trained with different corpora and context features are used in the experiments. From the data analysis, we reach some interesting conclusions: there are strong correlations between intrinsic and extrinsic evaluations, and the performance of different tasks can be affected by training corpora and context features to varying degrees.",

keywords = "Extrinsic evaluation, Intrinsic evaluation, Word embedding",

author = "Yuanyuan Qiu and Hongzheng Li and Shen Li and Yingdi Jiang and Renfen Hu and Lijiao Yang",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2018.; 17th China National Conference on Computational Linguistics, CCL 2018 and 6th International Symposium on Natural Language Processing Based on Naturally Annotated Big Data, NLP-NABD 2018 ; Conference date: 19-10-2018 Through 21-10-2018",

year = "2018",

doi = "10.1007/978-3-030-01716-3_18",

language = "English",

isbn = "9783030017156",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "209--221",

editor = "Xiaojie Wang and Ting Liu and Maosong Sun and Zhiyuan Liu and Yang Liu",

booktitle = "Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data - 17th China National Conference, CCL 2018, and 6th International Symposium, NLP-NABD 2018, Proceedings",

address = "Germany",

}

Qiu, Y, Li, H, Li, S, Jiang, Y, Hu, R & Yang, L 2018, Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings. in X Wang, T Liu, M Sun, Z Liu & Y Liu (eds), Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data - 17th China National Conference, CCL 2018, and 6th International Symposium, NLP-NABD 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11221 LNAI, Springer Verlag, pp. 209-221, 17th China National Conference on Computational Linguistics, CCL 2018 and 6th International Symposium on Natural Language Processing Based on Naturally Annotated Big Data, NLP-NABD 2018, Changsha, China, 19/10/18. https://doi.org/10.1007/978-3-030-01716-3_18

Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings. / Qiu, Yuanyuan; Li, Hongzheng; Li, Shen et al.
Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data - 17th China National Conference, CCL 2018, and 6th International Symposium, NLP-NABD 2018, Proceedings. ed. / Xiaojie Wang; Ting Liu; Maosong Sun; Zhiyuan Liu; Yang Liu. Springer Verlag, 2018. p. 209-221 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11221 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings

AU - Qiu, Yuanyuan

AU - Li, Hongzheng

AU - Li, Shen

AU - Jiang, Yingdi

AU - Hu, Renfen

AU - Yang, Lijiao

N1 - Publisher Copyright: © Springer Nature Switzerland AG 2018.

PY - 2018

Y1 - 2018

N2 - The evaluation of word embeddings has received a considerable amount of attention in recent years, but there have been some debates about whether intrinsic measures can predict the performance of downstream tasks. To investigate this question, this paper presents the first study on the correlation between results of intrinsic evaluation and extrinsic evaluation with Chinese word embeddings. We use word similarity and word analogy as the intrinsic tasks, Named Entity Recognition and Sentiment Classification as the extrinsic tasks. A variety of Chinese word embeddings trained with different corpora and context features are used in the experiments. From the data analysis, we reach some interesting conclusions: there are strong correlations between intrinsic and extrinsic evaluations, and the performance of different tasks can be affected by training corpora and context features to varying degrees.

AB - The evaluation of word embeddings has received a considerable amount of attention in recent years, but there have been some debates about whether intrinsic measures can predict the performance of downstream tasks. To investigate this question, this paper presents the first study on the correlation between results of intrinsic evaluation and extrinsic evaluation with Chinese word embeddings. We use word similarity and word analogy as the intrinsic tasks, Named Entity Recognition and Sentiment Classification as the extrinsic tasks. A variety of Chinese word embeddings trained with different corpora and context features are used in the experiments. From the data analysis, we reach some interesting conclusions: there are strong correlations between intrinsic and extrinsic evaluations, and the performance of different tasks can be affected by training corpora and context features to varying degrees.

KW - Extrinsic evaluation

KW - Intrinsic evaluation

KW - Word embedding

UR - http://www.scopus.com/inward/record.url?scp=85055443505&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01716-3_18

DO - 10.1007/978-3-030-01716-3_18

M3 - Conference contribution

AN - SCOPUS:85055443505

SN - 9783030017156

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 209

EP - 221

BT - Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data - 17th China National Conference, CCL 2018, and 6th International Symposium, NLP-NABD 2018, Proceedings

A2 - Wang, Xiaojie

A2 - Liu, Ting

A2 - Sun, Maosong

A2 - Liu, Zhiyuan

A2 - Liu, Yang

PB - Springer Verlag

T2 - 17th China National Conference on Computational Linguistics, CCL 2018 and 6th International Symposium on Natural Language Processing Based on Naturally Annotated Big Data, NLP-NABD 2018

Y2 - 19 October 2018 through 21 October 2018

ER -

Qiu Y, Li H, Li S, Jiang Y, Hu R, Yang L. Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings. In Wang X, Liu T, Sun M, Liu Z, Liu Y, editors, Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data - 17th China National Conference, CCL 2018, and 6th International Symposium, NLP-NABD 2018, Proceedings. Springer Verlag. 2018. p. 209-221. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-01716-3_18

Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this