Evaluation on ChatGPT for Chinese Language Understanding

Linhan Li; Huaping Zhang; Chunjin Li; Haowen You; Wenyao Cui

doi:10.1162/dint_a_00232

Evaluation on ChatGPT for Chinese Language Understanding

Linhan Li, Huaping Zhang^*, Chunjin Li, Haowen You, Wenyao Cui

^*Corresponding author for this work

School of Computer Science and Technology

Beijing Institute of Technology

Research output: Contribution to journal › Article › peer-review

7 Citations (Scopus)

Abstract

ChatGPT has attracted extension attention of academia and industry. This paper aims to evaluate ChatGPT in Chinese language understanding capability on 6 tasks using 11 datasets. Experiments indicate that ChatGPT achieved competitive results in sentiment analysis, summary, and reading comprehension in Chinese, while it is prone to factual errors in closed-book QA. Further, on two more difficult Chinese understanding tasks, that is, idiom fill-in-the-blank and cants understanding, we found that a simple chain-of-thought prompt can improve the accuracy of ChatGPT in complex reasoning. This paper further analyses the possible risks of using ChatGPT based on the results. Finally, we briefly describe the research and development progress of our ChatBIT.

Original language	English
Pages (from-to)	885-903
Number of pages	19
Journal	Data Intelligence
Volume	5
Issue number	4
DOIs	https://doi.org/10.1162/dint_a_00232
Publication status	Published - 1 Sept 2023

Keywords

Artificial intelligence
ChatBIT
ChatGPT
Chinese Language Understanding
Language Model

Access to Document

10.1162/dint_a_00232

Cite this

@article{67b242a924cb46bf9de079034f7e41f3,

title = "Evaluation on ChatGPT for Chinese Language Understanding",

abstract = "ChatGPT has attracted extension attention of academia and industry. This paper aims to evaluate ChatGPT in Chinese language understanding capability on 6 tasks using 11 datasets. Experiments indicate that ChatGPT achieved competitive results in sentiment analysis, summary, and reading comprehension in Chinese, while it is prone to factual errors in closed-book QA. Further, on two more difficult Chinese understanding tasks, that is, idiom fill-in-the-blank and cants understanding, we found that a simple chain-of-thought prompt can improve the accuracy of ChatGPT in complex reasoning. This paper further analyses the possible risks of using ChatGPT based on the results. Finally, we briefly describe the research and development progress of our ChatBIT.",

keywords = "Artificial intelligence, ChatBIT, ChatGPT, Chinese Language Understanding, Language Model",

author = "Linhan Li and Huaping Zhang and Chunjin Li and Haowen You and Wenyao Cui",

year = "2023",

month = sep,

day = "1",

doi = "10.1162/dint_a_00232",

language = "English",

volume = "5",

pages = "885--903",

journal = "Data Intelligence",

issn = "2096-7004",

publisher = "China National Publications Import and Export (Group) Corporation",

number = "4",

}

TY - JOUR

T1 - Evaluation on ChatGPT for Chinese Language Understanding

AU - Li, Linhan

AU - Zhang, Huaping

AU - Li, Chunjin

AU - You, Haowen

AU - Cui, Wenyao

PY - 2023/9/1

Y1 - 2023/9/1

N2 - ChatGPT has attracted extension attention of academia and industry. This paper aims to evaluate ChatGPT in Chinese language understanding capability on 6 tasks using 11 datasets. Experiments indicate that ChatGPT achieved competitive results in sentiment analysis, summary, and reading comprehension in Chinese, while it is prone to factual errors in closed-book QA. Further, on two more difficult Chinese understanding tasks, that is, idiom fill-in-the-blank and cants understanding, we found that a simple chain-of-thought prompt can improve the accuracy of ChatGPT in complex reasoning. This paper further analyses the possible risks of using ChatGPT based on the results. Finally, we briefly describe the research and development progress of our ChatBIT.

AB - ChatGPT has attracted extension attention of academia and industry. This paper aims to evaluate ChatGPT in Chinese language understanding capability on 6 tasks using 11 datasets. Experiments indicate that ChatGPT achieved competitive results in sentiment analysis, summary, and reading comprehension in Chinese, while it is prone to factual errors in closed-book QA. Further, on two more difficult Chinese understanding tasks, that is, idiom fill-in-the-blank and cants understanding, we found that a simple chain-of-thought prompt can improve the accuracy of ChatGPT in complex reasoning. This paper further analyses the possible risks of using ChatGPT based on the results. Finally, we briefly describe the research and development progress of our ChatBIT.

KW - Artificial intelligence

KW - ChatBIT

KW - ChatGPT

KW - Chinese Language Understanding

KW - Language Model

UR - http://www.scopus.com/inward/record.url?scp=85179658086&partnerID=8YFLogxK

U2 - 10.1162/dint_a_00232

DO - 10.1162/dint_a_00232

M3 - Article

AN - SCOPUS:85179658086

SN - 2096-7004

VL - 5

SP - 885

EP - 903

JO - Data Intelligence

JF - Data Intelligence

IS - 4

ER -

Evaluation on ChatGPT for Chinese Language Understanding

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this