基于注意力机制和微分跟踪器的宽度学习系统

Translated title of the contribution: Broad learning system based on attention mechanism and tracking differentiator

Lüchao Liao, Weidong Zou*, Jialong Yang, Huihuang Lu, Yuanqing Xia, Jianlei Gao

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Broad learning system (BLS) has advantages such as a simple model structure, high training efficiency, and easy interpretability. However, it also has drawbacks such as insufficient feature learning capability and unstable generalization performance. To alleviate these problems, broad learning system based on attention mechanism and tracking differentiator (TD), abbreviated as A-TD-BLS, was proposed. In terms of model structure, ATD-BLS introduced self-attention mechanism to the original BLS, and further fused and transformed the extracted features through attention weighting to improve the feature learning ability. In terms of model training methods, a weight optimization algorithm based on tracking differentiator was designed. This method effectively alleviates the overfitting phenomenon of the original BLS by limiting the size of the weight values, significantly reduces the influence of the number of hidden layer nodes on model performance and makes the generalization performance more stable. Moreover, the training algorithm was extended to the BLS incremental learning framework, so that the model can improve performance by dynamically adding hidden layer nodes. Multiple experiments conducted on some benchmark datasets show that compared to the original BLS, the classification accuracy of A-TD-BLS is increased by 1. 27% on average on classification datasets and the root mean square error of A-TD-BLS is reduced by 0. 53 on average on regression datasets. Besides, A-TD-BLS is less affected by the number of hidden layer nodes and has more stable generalization performance. Based on the above experimental results, it can be concluded that A-TD-BLS enhances the stability of generalization performance of the original BLS model, reduces the sensitivity of the model's generalization performance to hyperparameters, and effectively suppresses the phenomenon of overfitting.

Translated title of the contributionBroad learning system based on attention mechanism and tracking differentiator
Original languageChinese (Traditional)
Pages (from-to)583-593
Number of pages11
JournalShenzhen Daxue Xuebao (Ligong Ban)/Journal of Shenzhen University Science and Engineering
Volume41
Issue number5
DOIs
Publication statusPublished - Sept 2024

Fingerprint

Dive into the research topics of 'Broad learning system based on attention mechanism and tracking differentiator'. Together they form a unique fingerprint.

Cite this