Fine-Tuned DistilBERT for Automated User Story Quality Assessment

Source abstract

User stories are one of the most widely used mechanisms for requirements specification in agile methodologies; however, their narrative nature can introduce structural and pragmatic ambiguities and inconsistencies that affect the quality of the development process. This article proposes an approach for the automatic detection of violations of individual quality criteria within the Quality User Story (QUS) framework using DistilBERT. DistilBERT-based models, as well as BERT and RoBERTa as baselines, were trained for eight individual criteria using cross-validation on a refined dataset of 960 user stories. State-of-the-art generative models, namely GPT-3.5 Turbo, GPT-4 Turbo, and GPT-4o, were used as reference models for comparison. The results show that the specialized models consistently outperform the reference generative models, achieving high F1-scores and lower experimental variability, with DistilBERT exhibiting the best performance. Statistical analysis confirms significant differences between the two approaches.