Evaluation on ChatGPT for Chinese Language Understanding

Linhan Li, Huaping Zhang*, Chunjin Li, Haowen You, Wenyao Cui

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

7 Citations (Scopus)

Abstract

ChatGPT has attracted extension attention of academia and industry. This paper aims to evaluate ChatGPT in Chinese language understanding capability on 6 tasks using 11 datasets. Experiments indicate that ChatGPT achieved competitive results in sentiment analysis, summary, and reading comprehension in Chinese, while it is prone to factual errors in closed-book QA. Further, on two more difficult Chinese understanding tasks, that is, idiom fill-in-the-blank and cants understanding, we found that a simple chain-of-thought prompt can improve the accuracy of ChatGPT in complex reasoning. This paper further analyses the possible risks of using ChatGPT based on the results. Finally, we briefly describe the research and development progress of our ChatBIT.

Original languageEnglish
Pages (from-to)885-903
Number of pages19
JournalData Intelligence
Volume5
Issue number4
DOIs
Publication statusPublished - 1 Sept 2023

Keywords

  • Artificial intelligence
  • ChatBIT
  • ChatGPT
  • Chinese Language Understanding
  • Language Model

Fingerprint

Dive into the research topics of 'Evaluation on ChatGPT for Chinese Language Understanding'. Together they form a unique fingerprint.

Cite this