Chinese lexical analysis using cascaded hidden Markov model

Qun Liu*, Hua Ping Zhang, Hong Kui Yu, Xue Qi Cheng

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

89 Citations (Scopus)

Abstract

This paper presents an approach for Chinese lexical analysis using cascaded hidden Markov model (CHMM), which aims to incorporate Chinese word segmentation, part-of-speech tagging, disambiguation and unknown words recognition into an integrated theoretical frame. A class-based HMM is applied in word segmentation, and in this model, unknown words are treated in the same way as common words listed in the lexicon. Unknown words are recognized with reliability on roles sequence tagged using Viterbi algorithm in roles HMM. As for disambiguation, the authors bring forth an n-shortest-path strategy that, in the early stage, reserves the top N segmentation results as candidates and covers more ambiguity. Various experiments show that each level in the CHMM contributes to Chinese lexical analysis. A CHMM-based system ICTCLAS is accomplished. The system ranked top in the official open evaluation, which was held by the 973 project in 2002. And ICTCLAS achieved 2 first ranks and 1 second rank in the first international word segmentation bakeoff held by SIGHAN (the ACL Special Interest Group on Chinese Language Processing) in 2003. It indicates that ICTCLAS is one of the best Chinese lexical analyzers. In a word, CHMM is effective for Chinese lexical analysis.

Original languageEnglish
Pages (from-to)1421-1429
Number of pages9
JournalJisuanji Yanjiu yu Fazhan/Computer Research and Development
Volume41
Issue number8
Publication statusPublished - Aug 2004
Externally publishedYes

Keywords

  • Cascaded hidden Markov model
  • Chinese lexical analysis
  • ICTCLAS
  • POS tagging
  • Unknown words recognition
  • Word segmentation

Fingerprint

Dive into the research topics of 'Chinese lexical analysis using cascaded hidden Markov model'. Together they form a unique fingerprint.

Cite this