Pseudo Label based Contrastive Sampling for Long Text Retrieval

Le Zhu; Shumin Shi; Heyan Huang

doi:10.1109/IALP54817.2021.9675219

Pseudo Label based Contrastive Sampling for Long Text Retrieval

Le Zhu, Shumin Shi^*, Heyan Huang

^*Corresponding author for this work

School of Computer Science and Technology

Beijing Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Applying BERT to text retrieval brings great success recently, however, the internal limit of input length downgrades the performance on longer texts when using BERT. To address this issue, we split the long text into paragraphs as basic retrieval units. Then we explore several ways to calculate the pseudo labels for each query-paragraph pair: Inherit, BM25 and Vector inner product. With the annotated pseudo labels, contrastive sampling will be adopted to distinguish positive/negative examples to feed BERT for evaluating the relevance. Experiments show that our approach is effective on TREC 2020.

Original language	English
Title of host publication	2021 International Conference on Asian Language Processing, IALP 2021
Editors	Deyi Xiong, Ridong Jiang, Yanfeng Lu, Minghui Dong, Haizhou Li
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	95-98
Number of pages	4
ISBN (Electronic)	9781665483117
DOIs	https://doi.org/10.1109/IALP54817.2021.9675219
Publication status	Published - 2021
Event	2021 International Conference on Asian Language Processing, IALP 2021 - Singapore, Singapore Duration: 11 Dec 2021 → 13 Dec 2021

Publication series

Name	2021 International Conference on Asian Language Processing, IALP 2021

Conference

Conference	2021 International Conference on Asian Language Processing, IALP 2021
Country/Territory	Singapore
City	Singapore
Period	11/12/21 → 13/12/21

Keywords

BERT
Contrastive Sampling
Long Text Retrieval
Pretrained Language Model
Pseudo Label

Access to Document

10.1109/IALP54817.2021.9675219

Cite this

Zhu, L., Shi, S., & Huang, H. (2021). Pseudo Label based Contrastive Sampling for Long Text Retrieval. In D. Xiong, R. Jiang, Y. Lu, M. Dong, & H. Li (Eds.), 2021 International Conference on Asian Language Processing, IALP 2021 (pp. 95-98). (2021 International Conference on Asian Language Processing, IALP 2021). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IALP54817.2021.9675219

Zhu, Le ; Shi, Shumin ; Huang, Heyan. / Pseudo Label based Contrastive Sampling for Long Text Retrieval. 2021 International Conference on Asian Language Processing, IALP 2021. editor / Deyi Xiong ; Ridong Jiang ; Yanfeng Lu ; Minghui Dong ; Haizhou Li. Institute of Electrical and Electronics Engineers Inc., 2021. pp. 95-98 (2021 International Conference on Asian Language Processing, IALP 2021).

@inproceedings{9db0096dfc9945628252203a2220e2c5,

title = "Pseudo Label based Contrastive Sampling for Long Text Retrieval",

abstract = "Applying BERT to text retrieval brings great success recently, however, the internal limit of input length downgrades the performance on longer texts when using BERT. To address this issue, we split the long text into paragraphs as basic retrieval units. Then we explore several ways to calculate the pseudo labels for each query-paragraph pair: Inherit, BM25 and Vector inner product. With the annotated pseudo labels, contrastive sampling will be adopted to distinguish positive/negative examples to feed BERT for evaluating the relevance. Experiments show that our approach is effective on TREC 2020.",

keywords = "BERT, Contrastive Sampling, Long Text Retrieval, Pretrained Language Model, Pseudo Label",

author = "Le Zhu and Shumin Shi and Heyan Huang",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE; 2021 International Conference on Asian Language Processing, IALP 2021 ; Conference date: 11-12-2021 Through 13-12-2021",

year = "2021",

doi = "10.1109/IALP54817.2021.9675219",

language = "English",

series = "2021 International Conference on Asian Language Processing, IALP 2021",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "95--98",

editor = "Deyi Xiong and Ridong Jiang and Yanfeng Lu and Minghui Dong and Haizhou Li",

booktitle = "2021 International Conference on Asian Language Processing, IALP 2021",

address = "United States",

}

Zhu, L, Shi, S & Huang, H 2021, Pseudo Label based Contrastive Sampling for Long Text Retrieval. in D Xiong, R Jiang, Y Lu, M Dong & H Li (eds), 2021 International Conference on Asian Language Processing, IALP 2021. 2021 International Conference on Asian Language Processing, IALP 2021, Institute of Electrical and Electronics Engineers Inc., pp. 95-98, 2021 International Conference on Asian Language Processing, IALP 2021, Singapore, Singapore, 11/12/21. https://doi.org/10.1109/IALP54817.2021.9675219

Pseudo Label based Contrastive Sampling for Long Text Retrieval. / Zhu, Le; Shi, Shumin; Huang, Heyan.
2021 International Conference on Asian Language Processing, IALP 2021. ed. / Deyi Xiong; Ridong Jiang; Yanfeng Lu; Minghui Dong; Haizhou Li. Institute of Electrical and Electronics Engineers Inc., 2021. p. 95-98 (2021 International Conference on Asian Language Processing, IALP 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Pseudo Label based Contrastive Sampling for Long Text Retrieval

AU - Zhu, Le

AU - Shi, Shumin

AU - Huang, Heyan

PY - 2021

Y1 - 2021

N2 - Applying BERT to text retrieval brings great success recently, however, the internal limit of input length downgrades the performance on longer texts when using BERT. To address this issue, we split the long text into paragraphs as basic retrieval units. Then we explore several ways to calculate the pseudo labels for each query-paragraph pair: Inherit, BM25 and Vector inner product. With the annotated pseudo labels, contrastive sampling will be adopted to distinguish positive/negative examples to feed BERT for evaluating the relevance. Experiments show that our approach is effective on TREC 2020.

AB - Applying BERT to text retrieval brings great success recently, however, the internal limit of input length downgrades the performance on longer texts when using BERT. To address this issue, we split the long text into paragraphs as basic retrieval units. Then we explore several ways to calculate the pseudo labels for each query-paragraph pair: Inherit, BM25 and Vector inner product. With the annotated pseudo labels, contrastive sampling will be adopted to distinguish positive/negative examples to feed BERT for evaluating the relevance. Experiments show that our approach is effective on TREC 2020.

KW - BERT

KW - Contrastive Sampling

KW - Long Text Retrieval

KW - Pretrained Language Model

KW - Pseudo Label

UR - http://www.scopus.com/inward/record.url?scp=85125194836&partnerID=8YFLogxK

U2 - 10.1109/IALP54817.2021.9675219

DO - 10.1109/IALP54817.2021.9675219

M3 - Conference contribution

AN - SCOPUS:85125194836

T3 - 2021 International Conference on Asian Language Processing, IALP 2021

SP - 95

EP - 98

BT - 2021 International Conference on Asian Language Processing, IALP 2021

A2 - Xiong, Deyi

A2 - Jiang, Ridong

A2 - Lu, Yanfeng

A2 - Dong, Minghui

A2 - Li, Haizhou

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2021 International Conference on Asian Language Processing, IALP 2021

Y2 - 11 December 2021 through 13 December 2021

ER -

Zhu L, Shi S, Huang H. Pseudo Label based Contrastive Sampling for Long Text Retrieval. In Xiong D, Jiang R, Lu Y, Dong M, Li H, editors, 2021 International Conference on Asian Language Processing, IALP 2021. Institute of Electrical and Electronics Engineers Inc. 2021. p. 95-98. (2021 International Conference on Asian Language Processing, IALP 2021). doi: 10.1109/IALP54817.2021.9675219

Pseudo Label based Contrastive Sampling for Long Text Retrieval

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this