<b>Representation Graphs and Mathematical Reasoning: A Comparative Case Study of Human and Large Language Model Solution Trajectories in Algebra and Geometry</b>

Source abstract

In this work, representation graphs are explored and mathematical reasoning is analyzed by comparing the solution trajectories of human learners and large language models (LLM) in both algebra and geometry. The comparative case study design was suggested to examine the interpretation and representation of mathematical problems, as well as the ways in which the problems were solved with equations, calculation, diagrams, and verbal explanations. The proposed sample was composed of 60 students from University and three LLMs. There were 20 mathematical problems (10 algebraic and 10 geometric tasks), resulting in up to 1260 attempts in total to solve the problems. Mathematical steps and transitions were represented graphically, and the accuracy of answers, representational diversity, validity of transitions, and errors were quantitatively analyzed. The preliminary results for the human participants were 75.0%, with LLM 1 (85.0%), LLM 2 (80.0%) and LLM 3 (75.0%). Human accuracy made 80.0% and 70.0% in algebra and geometry tasks, respectively, while the LLM 1 achieved accuracy of 90.0% and 80.0% in algebra and geometry tasks, respectively. It was also illustrated that there were differences in the representation frequency, the complexity of the solution and the distribution of errors. The above numerical results presented in this report were not the actual experimental results but were examples of the analytical framework proposed. The study provided the process approach to compare mathematical reasoning other than final answer. It can be used to provide a framework for mathematics instruction, for identifying errors, and for assessing answers generated by Artificial Intelligence. To be a reliable and widely applicable representation graph further empirical research with valid data is required.