LOONA: An LLM Assistant for Spanish Language User Story Quality Assurance in Agile Development

Source abstract

User stories are central to agile requirements engineering, but in practice they often exhibit ambiguity, missing information, and weak or absent acceptance criteria, thereby increasing rework and reducing verifiability. This paper presents LOONA, a mathematically formulated NLP/LLM decision-support pipeline for Spanish user story quality assurance. We model the task as supervised binary classification followed by conditional text generation: a fine-tuned Spanish BERT classifier estimates whether a story requires improvement, and a generative model is triggered only when the estimated class exceeds a decision threshold. The study uses an anonymized reconstruction of 100 Spanish-language user stories from an administrative software backlog, labeled by three agile requirements experts through majority vote. The reconstructed annotation table yields Fleiss’ κ=0.76, indicating substantial agreement. We report stratified 5-fold classification results, lexical baselines, rule-based baselines, automatic rewrite-similarity metrics, and paired expert review of held-out outputs. LOONA achieved Accuracy =0.91, Precision =0.90, Recall =0.92, F1-score =0.91, and MCC =0.82, outperforming TF-IDF logistic regression, TF-IDF support vector machines, and a rule-based baseline. In paired expert review (n=30), 70% of generated rewrites improved over the original story, 20% were unchanged, and 10% degraded; generated acceptance criteria were judged correct in 86.7% of cases and free of hallucinated information in 93.3%. The contribution is framed as a formal and empirical proof of concept for model-level LLM-assisted requirements quality assurance, with explicit limitations regarding dataset size, confidentiality, and human oversight.