Hierarchical text-label integrated attention network for document classification

Recurrent neural networks (RNN) and convolutional neural networks (CNN) have been extensively used on text classification to capture the local and long-range dependencies. Recent work has demonstrated the superiority of self-attention networks (SAN) owing to their highly parallelizable computation and excellent performance. However, SAN has difficulty capturing meaningful semantic relationships over very long sequences, and the memory requirement grows rapidly in line with the sequence length. To solve these limitations of SAN in processing long document sequence, this paper proposes four novel ideas and build a hierarchical text-label integrated attention network(HLAN). Firstly, a hierarchical architecture is introduced to map the hierarchy of document, which effectively shortens the sequence length of each process. Secondly, the attention weights are calculated in the joint embedding space of text and label. Thirdly, a multi-head soft attention is proposed to compress the sequence encoded by self-attention into a single vector. Finally, a loss term called class loss is given and combined with cross entropy loss. HLAN achieves competitive results over the compared strong baseline methods on 4 out of 5 benchmark datasets, which verifies the effectiveness of HLAN for document classification, in terms of both accuracy and memory requirement.