Compressing student history in mathematics knowledge tracing

Source abstract

This paper investigates how complicated a representation of a student’s past performance has to be to predict what the student will do next. The data used were 1,927,265 mathematics responses from 8,996 Finnish students. Evaluation was conducted on students entirely excluded from model fitting. Predictions based on item identity alone achieved an AUC of .8362. Adding one dynamically updated measure of recent performance raised AUC to .8669. This simple learner state recovered 93.5% of the AUC improvement between the item-only baseline and a query-conditioned transformer (.8690). Increasing the explicit learner state to 50 dimensions raised this proportion to 97.1%, while a 128-dimensional transformer using a common summary of student history achieved an AUC of .8687. The same ordering was observed for log loss, although differences among the richer models were somewhat larger. Thus knowing a student’s recent performance relative to item difficulty captured most of the improvement obtainable from more complex representations of the response sequence.