Enhancing consistency with the fusion of paralleled decoders for text generation

Generating coherent and consistent long text is an important but challenging task. Despite the recent success of planning-based methods in maintaining consistency and modeling long-distance coherence for text, the existing generative model still suffers from the inconsistency problem among prompt, plan, and target text. In this paper, we propose a novel generative model MDFUT, which leverages an autoregressive model to do content planning and surface realization simultaneously. To alleviate error accumulation and performance compromising for the pre-trained language model, we introduce a novel paralleled dual decoder architecture to improve generation form. Moreover, we propose a bridging objective to minimize the bidirectional KL divergence between the distributions of the dual decoder to enhance the consistency between the plan and text. We use BART as the backbone model and extend the typical transformer to dual decoder architecture. Extensive experiments are conducted on four datasets: Wikiplots and ROCStories for long- and short-form story generation task, CMV for argument generation task, and CNNNews for news generation task. The results show that our model achieves a significant improvement in both automatic and human evaluations and can generate more consistent texts than the baselines.