A NEURAL NETWORK-BASED METHOD FOR EXTRACTING CAUSAL RELATIONS USING COMPARATIVE ANALYSIS OF SYNTHETIC AND OPEN CORPORA
Abstract
The automatic extraction of causal relations is critically important for decision support systems, but its development is hindered by the scarcity of annotated corpora. The aim of this study is to comparatively analyze the effectiveness of three types of training data: open expert corpora, a manually annotated political corpus, and synthetic data generated by a large language model. Experiments were conducted using the DistilBERT architecture with token-level BIOES tagging. Synthetic data were generated with the Llama-3 LLM by encapsulating causes and effects in XML markers, followed by deterministic conversion into the BIOES format. For comparison, two expert corpora (EventStoryLine and SemEval-2010 Task 8), the expert-annotated political corpus PolitiCAUSE, and a synthetic sample of 1,100 balanced examples were used. The model trained on synthetic data achieved a Macro F1 score of 0.736, which is 6.7% higher than the result obtained from training on the expert corpora (0.690). Training on the political corpus PolitiCAUSE (24,417 examples) yielded a substantially lower result, with a Macro F1 score of 0.367. This discrepancy is attributed to class imbalance (approximately 15% causal tokens versus 50% in the synthetic sample) and to differences in task formulation: PolitiCAUSE is oriented toward binary sentence-level classification of causal presence, whereas the present study addresses token-level span labeling. The hypothesis that annotation quality and class balance are more important than data volume is confirmed. The proposed LLM-generation pipeline makes it possible to create training datasets that outperform expert corpora without direct manual annotation costs. The method is recommended for rapid deployment of causal relation extraction systems in new domains.
References
1. Cheng Q., Zeng Z., Hu X. et al. A Survey of Event Causality Identification: Taxonomy, Challenges, Assessment, and Prospects, ACM Computing Surveys, 2025, Vol. 58, No. 3, Art. 59. DOI: 10.1145/3698571.
2. Tan C., De Young I., Rosenfeld A. et al. Causal Reasoning in Natural Language Processing: A Survey, Transactions of the Association for Computational Linguistics, 2023, Vol. 11, pp. 1-25. DOI: 10.1162/tacl_a_00567.
3. Devlin J., Chang M.-W., Lee K., Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Proceedings of NAACL-HLT, 2019, pp. 4171-4186. DOI: 10.18653/v1/N19-1423.
4. Sanh V., Debut L., Chaumond J., Wolf T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, arXiv preprint arXiv:1910.01108, 2019. DOI: 10.48550/arXiv.1910.01108.
5. Hendrickx I., Kim S.N., Kozareva Z. et al. SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals, Proceedings of the 5th International Workshop on Semantic Evaluation, 2010, pp. 33-38. DOI: 10.3115/1615957.1615963.
6. Tan F.A., Hürriyetoğlu A., Caselli T. et al. The Causal News Corpus: Annotating Causal Relations in Event Sentences from News, Proceedings of LREC, 2022, pp. 2298-2310. DOI: 10.18653/v1/2022.lrec-1.246.
7. Søgaard A., Ebert S., Bastings J., Filippova K. We need to talk about random splits, Proceedings of EACL, 2021, pp. 1823-1832. DOI: 10.18653/v1/2021.eacl-main.156.
8. Josifoski M., Sakota M., Peyrard M., West R. Exploiting Asymmetry for Synthetic Training Data Gener-ation: SynthIE and the Case of Information Extraction, Proceedings of EMNLP, 2023, pp. 1555-1574. DOI: 10.18653/v1/2023.emnlp-main.96.
9. Meta AI. The Llama 3 Herd of Models, arXiv preprint arXiv:2407.21783, 2024. DOI: 10.48550/arXiv.2407.21783.
10. Lample G., Ballesteros M., Subramanian S., Kawakami K., Dyer C. Neural Architectures for Named Entity Recognition, Proceedings of NAACL-HLT, 2016, pp. 260-270. DOI: 10.18653/v1/N16-1030.
11. Wei J., Zou K. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classifica-tion Tasks, Proceedings of EMNLP, 2019, pp. 6382-6388. DOI: 10.18653/v1/D19-1670.
12. Anaby-Tavor A., Carmeli B., Goldbraich E. et al. Do Not Have Enough Data? Deep Learning to the Rescue!, Proceedings of AAAI, 2020, Vol. 34, No. 05, pp. 7383-7390. DOI: 10.1609/aaai.v34i05.6234.
13. Lugovskoy D.V. Metody mashinnogo obucheniya dlya analiza i klassifikatsii tekstov na estestvennom yazyke: dis. ... kand. fiz.-mat. nauk [Machine learning methods for analyzing and classifying natural lan-guage texts: cand. of phys. and math. sc. diss.]: 05.13.17. Saint Petersburg, 2022, 142 p.
14. Presnyakov M.V. Metody i algoritmy avtomaticheskogo izvlecheniya faktograficheskoy informatsii iz tekstov na estestvennom yazyke: dis. ... kand. tekhn. nauk [Methods and algorithms for automatically ex-tracting factual information from natural language texts: cand. ofeng. sc. diss.]: 05.13.11. Moscow, 2021, 165 p.
15. Meng Y., Li X., Zhang Y. et al. GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Models, arXiv preprint arXiv:2310.04742, 2023. DOI: 10.48550/arXiv.2310.04742.
16. McCoy R.T., Pavlick E., Linzen T. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference, Proceedings of ACL, 2019, pp. 3428-3448. DOI: 10.18653/v1/P19-1334.
17. Sang E.F., De Meulder F. Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition, Proceedings of HLT-NAACL, 2003, pp. 142-147. DOI: 10.3115/1119176.1119195.
18. Powers D.M. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation, arXiv preprint arXiv:2010.16061, 2020. DOI: 10.48550/arXiv.2010.16061.
19. Caselli T., Vossen P. The Event StoryLine Corpus: A New Benchmark for Causal and Temporal Rela-tion Extraction, Proceedings of the Workshop on Events and Stories in the News, 2017, pp. 77-86. DOI: 10.18653/v1/W17-2711.
20. Garcia Corral P., Bechara H., Zhang R., Jankin S. PolitiCause: An Annotation Scheme and Corpus for Causality in Political Texts, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 12836-12845. Available at: https://aclanthology.org/2024.lrec-main.1124.
