Introduction
This research explores how Large Language Models (LLMs) can improve the automated identification of rhetorical strategies in academic writing, with a particular focus on the Engagement system of the Appraisal framework. While existing NLP approaches can identify these discourse features, they often struggle with the highly imbalanced data found in academic corpora, where common categories are well represented but nuanced rhetorical moves are relatively rare. Building on previous sequence-tagging approaches, this study investigates two strategies for addressing this challenge: using a generative LLM for direct rhetorical span extraction with few-shot prompting and a Context Anchoring algorithm, and using LLM-generated and corpus-derived data to augment minority classes for training a RoBERTa-based sequence tagger. The research ultimately examines whether improving the quality and quantity of training data can be more effective than increasing model complexity when computationally analyzing complex rhetorical features in academic writing.
Interface
Engagement Labels
The study utilized the Engagement Discourse Treebank (EDT) oversampled reference corpus. The main experiment employed a Data-Augmented RoBERTa (DA-RoBERTa) sequence tagger using a Span Categorizer (spancat) architecture, which can handle overlapping pragmatic layers without structural contradiction. To overcome the class imbalance of context-dependent minority classes (JUSTIFYING, ENDOPHORIC, SOURCES, CITATION), the training dataset was augmented to 2,000 instances per engagement category. This augmentation was performed by using a teacher model (RoBERTa + LSTM) to pseudo-label unannotated sentences mined from an academic corpus, creating a Silver-Standard Dataset.
| Category | Pre-Augmentation (Base Train) | Synthetic Sentences (Few-Shot) | Silver-Standard Dataset | Spans Added |
|---|---|---|---|---|
| ATTRIBUTION | 6,720 | 9,353 | 9,080 | |
| CITATION | 2,595 | 4,924 | 4,683 | +2,000 |
| COUNTER | 3,721 | 3,773 | 4,804 | |
| DENY | 2,819 | 2,851 | 3,829 | |
| ENDOPHORIC | 1,670 | 3,782 | 3,670 | +2,000 |
| ENTERTAIN | 8,272 | 9,101 | 10,789 | |
| JUSTIFYING | 2,917 | 4,962 | 5,515 | +2,000 |
| MONOGLOSS | 6,987 | 11,157 | 9,949 | |
| PROCLAIM | 3,048 | 3,767 | 3,537 | |
| SOURCES | 4,368 | 7,850 | 6,449 | +2,000 |
Table 1. Pre- and Post-Augmentation Span Distribution Across Base and Augmented Datasets.
Note: The Synthetic Sentences Dataset represents the few-shot prompt LLM-generated sentences, which was used for the preliminary experiment, not in the main experiment. To see the full details, click Full Paper to access the full research paper.
Results
The data-augmented DA-RoBERTa model utilizing the Silver-Standard Dataset achieved the highest global performance with a Macro F1 of 0.7217 and a Weighted F1 of 0.7369. This DA-RoBERTa model outperformed the RoBERTa + LSTM benchmark in 7 out of 10 specific engagement categories, including ATTRIBUTION, DENY, ENDOPHORIC, JUSTIFYING, MONOGLOSS, PROCLAIM, and SOURCES.
| Main Experiment | Eguchi & Kyle (2023) | ||||||
|---|---|---|---|---|---|---|---|
| Category / Architecture | RoBERTa-base | DA-RoBERTa (zero-shot) | DA-RoBERTa (few-shot) | DA-RoBERTa (Silver-Standard) | RoBERTa + LSTM | ||
| ATTRIBUTION | .6634 | .6720 | .6691 | .6920 | .6728 | ||
| CITATION | .6806 | .6809 | .6897 | .6980 | .9168 | ||
| COUNTER | .6775 | .7031 | .6815 | .6974 | .8224 | ||
| DENY | .7706 | .7404 | .7690 | .8080 | .7719 | ||
| ENDOPHORIC | .4967 | .5949 | .6161 | .6600 | .5955 | ||
| ENTERTAIN | .7571 | .7748 | .7671 | .7701 | .7912 | ||
| JUSTIFYING | .5300 | .6469 | .6568 | .6676 | .5920 | ||
| MONOGLOSS | .7706 | .7734 | .7782 | .7967 | .7638 | ||
| PROCLAIM | .6255 | .6617 | .6782 | .7104 | .6716 | ||
| SOURCES | .6565 | .6516 | .6669 | .7165 | .6624 | ||
| F1-Score (Macro) | .6629 | .6900 | .6973 | .7217 | .7260 | ||
| F1-Score (Weighted) | .6914 | .7106 | .7159 | .7369 | .7367 | ||
Findings
- Model Architecture: The results demonstrated that augmenting the training data with a Silver-Standard dataset effectively compensates for the simpler model architecture of DA-RoBERTa, which lacks additional recurrent LSTM layers.
- Semantic Distribution: A t-SNE semantic distribution analysis showed that LLM-generated synthetic text lacked organic structural diversity, whereas the Silver-Standard dataset retained the natural, overlapping density of human-written academic prose.
Limitations
- Synthetic data limitations: LLM-generated training data was constrained by repetitive syntactic patterns and lacked the linguistic diversity of authentic academic writing, resulting in lower performance than Silver-Standard Dataset augmentation (Macro F1: 0.6973 vs. 0.7217).
- Pseudo-labelling noise: The augmentation approach depended on the quality of the teacher model, meaning classification and boundary errors were propagated into the training data and could negatively affect the final DA-RoBERTa model.