TY - JOUR
T1 - 基于注意力机制和微分跟踪器的宽度学习系统
AU - Liao, Lüchao
AU - Zou, Weidong
AU - Yang, Jialong
AU - Lu, Huihuang
AU - Xia, Yuanqing
AU - Gao, Jianlei
N1 - Publisher Copyright:
© 2024 Editorial Office of Journal of Shenzhen University. All rights reserved.
PY - 2024/9
Y1 - 2024/9
N2 - Broad learning system (BLS) has advantages such as a simple model structure, high training efficiency, and easy interpretability. However, it also has drawbacks such as insufficient feature learning capability and unstable generalization performance. To alleviate these problems, broad learning system based on attention mechanism and tracking differentiator (TD), abbreviated as A-TD-BLS, was proposed. In terms of model structure, ATD-BLS introduced self-attention mechanism to the original BLS, and further fused and transformed the extracted features through attention weighting to improve the feature learning ability. In terms of model training methods, a weight optimization algorithm based on tracking differentiator was designed. This method effectively alleviates the overfitting phenomenon of the original BLS by limiting the size of the weight values, significantly reduces the influence of the number of hidden layer nodes on model performance and makes the generalization performance more stable. Moreover, the training algorithm was extended to the BLS incremental learning framework, so that the model can improve performance by dynamically adding hidden layer nodes. Multiple experiments conducted on some benchmark datasets show that compared to the original BLS, the classification accuracy of A-TD-BLS is increased by 1. 27% on average on classification datasets and the root mean square error of A-TD-BLS is reduced by 0. 53 on average on regression datasets. Besides, A-TD-BLS is less affected by the number of hidden layer nodes and has more stable generalization performance. Based on the above experimental results, it can be concluded that A-TD-BLS enhances the stability of generalization performance of the original BLS model, reduces the sensitivity of the model's generalization performance to hyperparameters, and effectively suppresses the phenomenon of overfitting.
AB - Broad learning system (BLS) has advantages such as a simple model structure, high training efficiency, and easy interpretability. However, it also has drawbacks such as insufficient feature learning capability and unstable generalization performance. To alleviate these problems, broad learning system based on attention mechanism and tracking differentiator (TD), abbreviated as A-TD-BLS, was proposed. In terms of model structure, ATD-BLS introduced self-attention mechanism to the original BLS, and further fused and transformed the extracted features through attention weighting to improve the feature learning ability. In terms of model training methods, a weight optimization algorithm based on tracking differentiator was designed. This method effectively alleviates the overfitting phenomenon of the original BLS by limiting the size of the weight values, significantly reduces the influence of the number of hidden layer nodes on model performance and makes the generalization performance more stable. Moreover, the training algorithm was extended to the BLS incremental learning framework, so that the model can improve performance by dynamically adding hidden layer nodes. Multiple experiments conducted on some benchmark datasets show that compared to the original BLS, the classification accuracy of A-TD-BLS is increased by 1. 27% on average on classification datasets and the root mean square error of A-TD-BLS is reduced by 0. 53 on average on regression datasets. Besides, A-TD-BLS is less affected by the number of hidden layer nodes and has more stable generalization performance. Based on the above experimental results, it can be concluded that A-TD-BLS enhances the stability of generalization performance of the original BLS model, reduces the sensitivity of the model's generalization performance to hyperparameters, and effectively suppresses the phenomenon of overfitting.
KW - artificial intelligence
KW - broad learning system
KW - feature extraction
KW - incremental learning
KW - self-attention mechanism
KW - tracking differentiator
UR - http://www.scopus.com/inward/record.url?scp=85204962620&partnerID=8YFLogxK
U2 - 10.3724/SP.J.1249.2024.05583
DO - 10.3724/SP.J.1249.2024.05583
M3 - 文章
AN - SCOPUS:85204962620
SN - 1000-2618
VL - 41
SP - 583
EP - 593
JO - Shenzhen Daxue Xuebao (Ligong Ban)/Journal of Shenzhen University Science and Engineering
JF - Shenzhen Daxue Xuebao (Ligong Ban)/Journal of Shenzhen University Science and Engineering
IS - 5
ER -