Beyond Access: Guided LLM Scaffolding for Independent Learning in Undergraduate Statistics

Abstract

Large language models (LLMs) are increasingly entering students' learning practices, but their educational value may depend on whether they are used to support reasoning or to complete tasks without engaging in the underlying reasoning. This study examines guided LLM use in an undergraduate Probability and Statistics course, focusing on the distinction between assigned LLM access and the quality of students' actual interaction with the model. In a four-week quasi-experimental summer program, students were organized into three balanced conditions: no LLM access, unrestricted LLM access, and guided LLM access. The guided condition used the same LLM platform as the unrestricted condition, but students received explicit training and rules intended to promote reasoning-focused help-seeking, stepwise hints, verification, and ethical use. All quizzes and the delayed final exam were completed without LLM or external assistance, allowing us to separate AI-supported practice performance from independent learning. Results show that guided use was associated with a clearer learning-oriented interaction pattern than unrestricted access, especially in prioritizing reasoning over final answers and requesting stepwise support. In behavior-defined analyses, Guided-LLM students showed a promising pattern of stronger no-help quiz performance, while practice scores showed no consistent Guided-LLM advantage. Available time measures did not support a simple duration-based explanation, and self-assessment calibration suggested better alignment between perceived and demonstrated understanding in Guided-LLM. These findings suggest that access alone may not reliably distinguish independent performance; instead, the quality of interaction and reasoning-focused scaffolds warrant further study.