LOW-RESOURCE ABSTRACTIVE TEXT SUMMARIZATION VIA CONTRASTIVE UNSUPERVISED LEARNING WITH A ROUGE-ORIENTED LOSS FUNCTION
Abstract
The relevance of this work is motivated by the fact that in specialized domains and low-resource languages obtaining a sufficient number of “document-summary” pairs for effective abstractive text summarization is expensive and often practically infeasible, whereas modern deep learning models require large labeled corpora and large volumes of text remain unused. The aim of this study is to develop a new training method for abstractive summarization models in low-resource (10-shot and 100-shot) settings that improves quality by fine-tuning the model on unlabeled data using unsupervised contrastive learning with input augmentation. The research tasks include designing a new contrastive loss function and comparing the proposed approach with existing methods of low-resource abstractive summarization. The methods and approaches comprise a new loss function for fine-tuning a transformer that includes a contrastive generative component based on a differentiable approximation of the ROUGE-3 metric. Two variants of the method are proposed – the sequential “DiffROUGE-seq” and the semi-supervised “DiffROUGE-sim”. BART-large is used as the base model, while input augmentations are generated by FLAN-T5-large in a zero-shot regime. Experiments are conducted on popular datasets AESLC, Gigaword, XSum, and Reddit. The proposed method achieves substantial improvements in ROUGE scores in successful cases, with average gains of 2.69 in the 10-shot setting and 2.13 in the 100-shot setting. In summary, the newly proposed low-resource abstractive summarization method that leverages both labeled and unlabeled data is significantly more effective in terms of ROUGE than existing approaches
References
1. Vaswani A., Shazeer N., Parmar N., Uszkoreit J. Jones L., Gomez A. N., Kaiser Ł., Polosukhin I. At-tention is all you need, Proceedings of the 31st International Conference on Neural Information Pro-cessing Systems, 2017, pp. 6000-6010. DOI: 10.5555/3295222.3295349.
2. Lewis M., Liu Y., Goyal N., Ghazvininejad M., Mohamed, A., Levy O., Stoyanov V., Zettlemoyer L. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, Proceedings of the 58th Annual Meeting of the Association for Computational Lin-guistics, 2020, pp. 7871-7880. DOI: 10.18653/v1/2020.acl-main.703.
3. Zhang J., Zhao Y., Saleh M., Liu P. J. PEGASUS: pre-training with extracted gap-sentences for abstrac-tive summarization, Proceedings of the 37th International Conference on Machine Learning, 2020, pp. 7871-7880. DOI: 10.5555/3524938.3525989.
4. Raffel C., Shazeer N., Roberts A., Lee K., Narang S., Matena M., Zhou Y., Li W., Liu P. J. Exploring the limits of transfer learning with a unified text-to-text transformer, The Journal of Machine Learning Re-search, 2020, Vol. 21, No. 1, pp. 5485-5551. DOI: 10.5555/3455716.3455856.
5. Lin C.-Y. ROUGE: A Package for Automatic Evaluation of Summaries, Text Summarization Branches Out. Association for Computational Linguistics, 2004, pp. 74-81.
6. Tsvigun A., Lysenko I., Sedashov D., Lazichny I., Damirov E., Karlov V., Belousov A., Sanochkin L., Panov M., Panchenko A., Burtsev M., Shelmanov A. Active Learning for Abstractive Text Summariza-tion, Findings of the Association for Computational Linguistics: EMNLP 2022. Association for Compu-tational Linguistics, 2022, pp. 5128-5152. DOI: 10.18653/v1/2022.findings-emnlp.377.
7. Wan D., Bansal M. FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguis-tics, 2022, pp. 1010-1028. DOI: 10.18653/v1/2022.naacl-main.74.
8. Chen, Y.-S., Shuai, H.-H. Meta-Transfer Learning for Low-Resource Abstractive Summarization, Pro-ceedings of the AAAI Conference on Artificial Intelligence, 2021, Vol. 35, No.14, pp. 12692-12700. DOI: 10.1609/aaai.v35i14.17503.
9. Chen Y., Liu Y., Xu R., Yang Z., Zhu C., Zeng M., Zhang Y. UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers). Association for Computational Lin-guistics, 2023, pp. 12833-12855. DOI: 10.18653/v1/2023.acl-long.718.
10. Ravaut M., Joty S., Chen N. Towards Summary Candidates Fusion, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2022, pp. 8488-8504. DOI: 10.18653/v1/2022.emnlp-main.581.
11. Zhang R., Tetreault J. This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2019, pp. 446-456. DOI: 10.18653/v1/P19-1043.
12. Napoles C., Gormley M., Durme B.V. Annotated Gigaword, Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX). As-sociation for Computational Linguistics, 2012, pp. 95-100.
13. Narayan S., Cohen S. B., Lapata M. Don't Give Me the Details, Just the Summary! Topic-Aware Con-volutional Neural Networks for Extreme Summarization, Proceedings of the 2018 Conference on Empir-ical Methods in Natural Language Processing. Association for Computational Linguistics, 2018, pp. 1797-1807. DOI: 10.18653/v1/D18-1206.
14. Kim B., Kim H., Kim G. Abstractive Summarization of Reddit Posts with Multi-level Memory Net-works, Proceedings of the 2019 Conference of the North American Chapter of the Association for Com-putational Linguistics: Human Language Technologies, Vol. 1 (Long and Short Papers). Association for Computational Linguistics, 2019, pp. 2519-2531. DOI: 10.18653/v1/N19-1260.
15. Xie Q. Dai Z. Hovy E. Luong M.-T., Le Q. V. Unsupervised data augmentation for consistency training, Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020. DOI: 10.5555/3495724.3496249.
16. Shao C., Chen X., Feng Y. Greedy Search with Probabilistic N-gram Matching for Neural Machine Translation, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro-cessing. Association for Computational Linguistics, 2018, pp. 4778-4784. DOI: 10.18653/v1/D18-1510.
17. Papineni K., Roukos S., Ward T., Zhu W.-J. BLEU: a method for automatic evaluation of machine trans-lation, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Associa-tion for Computational Linguistics, 2002, pp. 311-318.
18. Chung H.W., Hou L., Longpre S., Zoph B., Tai Y., Fedus W., Li Y., Wang X., Dehghani M., Brahma S., Webson A., Gu S.S., Dai Z., Suzgun M., Chen X., Chowdhery A., Castro-Ros A., Pellat M., Robinson K., Valter D., Narang S., Mishra G., Yu A., Zhao V., Huang Y., Dai A., Yu H., Petrov S., Chi E.H., Dean J., Devlin J., Roberts A., Zhou D., Le Q.V., Wei J. Scaling instruction-finetuned language models, The Journal of Machine Learning Research, 2024, Vol. 25, No. 1, pp. 3381-3433. DOI: 10.5555/3722577.3722647.
19. Tianyi Z., Varsha K., Felix W., Kilian Q. W., Yoav A. BERTScore: Evaluating Text Generation with BERT, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Associa-tion for Computational Linguistics, 2020, pp. 311-318.
20. Yuan W., Neubig G., Liu P. BARTSCORE: evaluating generated text as text generation, Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021. DOI: 10.5555/3540261.3542349.
21. Wolf T., Debut L., Sanh V., Chaumond J., Delangue C., Moi A., Cistac P., Rault T., Louf R., Funtowicz M., Davison J., Shleifer S., von Platen P., Ma C., Jernite Y., Plu J., Xu C., Le Scao T., Gugger S., Drame M., Lhoest Q., Rush A. Transformers: State-of-the-Art Natural Language Processing, Proceed-ings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demon-strations. Association for Computational Linguistics, 2020, pp. 38-45. DOI: 10.18653/v1/2020.emnlp-demos.6.
22. Ansel J., Yang E., He H., Gimelshein N., Jain A., Voznesensky M., Bao B., Bell P., Berard D., Burovski E., Chauhan G., Chourdia A., Constable W., Desmaison A., DeVito Z., Ellison E., Feng W., Gong J., Gschwind M., Hirsh B., Huang S., Kalambarkar K., Kirsch, L., Lazos M., Lezcano M., Liang Y.,
Liang J., Lu Y., Luk CK, Maher B., Pan Y., Puhrsch C., Reso M., Saroufim M., Siraichi M. Y., Suk H., Suo M., Tillet P., Wang E., Wang X., Wen W., Zhang S., Zhao X., Zhou K., Zou R., Mathews A., Chan-an G., Wu P., Chintala S. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation, 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vol. 2 (ASPLOS '24). ACM, 2024. DOI: 10.1145/3620665.3640366.
23. Lhoest Q., Villanova del Moral A., Jernite Y., Thakur A., von Platen P., Patil S., Chaumond J.,
Drame M., Plu J., Tunstall L., Davison J., Šaško M., Chhablani G., Malik B., Brandeis S., Le Sca, Sanh V., Xu C., Patry N., McMillan-Majo, ., Schmid P., Gugger S., Delangue C., Matussière T.,
Debut L., Bekman S., Cistac P., Goehringer T., Mustar V., Lagunas F., Rush A., Wolf T. Datasets:
A Community Library for Natural Language Processing, Proceedings of the 2021 Conference on Empir-ical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, 2021, pp. 175-184. DOI: 10.18653/v1/2021.emnlp-demo.21.
24. Boyd S., Vandenberghe L. Convex optimization. Cambridge university press, 2004.
