Beyond Predictive Performance: SHAP-Based Explanations and Alignment with Literature-Based Indicators in Code Smell Detection

Source abstract

Machine learning-based code smell detection relies on structural metrics, but predictive performance alone does not establish whether the information used by classifiers corresponds to the characteristics reported in the literature. This study examines the explanatory behavior of Support Vector Machine (SVM) models by integrating SHAP-based feature importance, point-biserial associations, and literature-based code smell indicators. The analysis reconstructs 120 SVM models across three open-source Java systems and four code smells, using ten folds per system–smell combination. Feature importance, persistence across folds, and consistency across software systems are examined alongside SHAP–correlation concordance, indicator coverage, and directional compatibility. The results show that influential metrics vary across code smells and software systems, while high average importance does not necessarily imply persistence across folds. Agreement between SHAP-based rankings and statistical associations is partial. Indicator coverage and directional compatibility provide distinct forms of evidence, and some literature-based operationalizations cannot be evaluated using the available metrics. A complementary sensitivity analysis shows high agreement among repeated SHAP realizations under the tested conditions. These findings indicate that explanatory relevance depends on the experimental context and that model contributions, statistical associations, and literature-based indicators provide complementary evidence for interpreting code smell predictions.