Exploring Generative AI assisted feedback writing for students' written responses to a physics conceptual question with prompt engineering and few-shot learning

Tong Wan; Zhongzhou Chen

doi:10.1103/physrevphyseducres.20.010152

Home/Archive/arXiv

arXiv · 2311.06180

Exploring Generative AI assisted feedback writing for students' written responses to a physics conceptual question with prompt engineering and few-shot learning

Tong Wan·Zhongzhou Chen

View original source ↗Read source document ↗Export metadata ↓

Abstract

Instructor's feedback plays a critical role in students' development of conceptual understanding and reasoning skills. However, grading student written responses and providing personalized feedback can take a substantial amount of time. In this study, we explore using GPT-3.5 to write feedback to student written responses to conceptual questions with prompt engineering and few-shot learning techniques. In stage one, we used a small portion (n=20) of the student responses on one conceptual question to iteratively train GPT. Four of the responses paired with human-written feedback were included in the prompt as examples for GPT. We tasked GPT to generate feedback to the other 16 responses, and we refined the prompt after several iterations. In stage two, we gave four student researchers the 16 responses as well as two versions of feedback, one written by the authors and the other by GPT. Students were asked to rate the correctness and usefulness of each feedback, and to indicate which one was generated by GPT. The results showed that students tended to rate the feedback by human and GPT equally on correctness, but they all rated the feedback by GPT as more useful. Additionally, the successful rates of identifying GPT's feedback were low, ranging from 0.1 to 0.6. In stage three, we tasked GPT to generate feedback to the rest of the student responses (n=65). The feedback was rated by four instructors based on the extent of modification needed if they were to give the feedback to students. All the instructors rated approximately 70% of the feedback statements needing only minor or no modification. This study demonstrated the feasibility of using Generative AI as an assistant to generating feedback for student written responses with only a relatively small number of examples. An AI assistance can be one of the solutions to substantially reduce time spent on grading student written responses.

Explore related subjects

physics.ed-ph

Keep this discovery

BibTeX RIS

Tong Wan, Zhongzhou Chen. 2023-11-10. Exploring Generative AI assisted feedback writing for students' written responses to a physics conceptual question with prompt engineering and few-shot learning. https://doi.org/10.1103/physrevphyseducres.20.010152

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Questioning your brilliance in physics: Differential shifts in fixed mindsets by grade and gender

Students' domain-specific mindsets and their beliefs about their capacity to improve through effort play a crucial role in shaping their experiences and decisions to persist in STEM disciplines. Physics is generally seen as a field requiring innate brilliance, which can reinforce fixed mindsets, particularly after initial setbacks in performance that are common in introductory university courses. In this study, we examine changes in fixed mindsets and potential gender differences in an introductory calculus-based physics course. Our sample consisted of 508 students with an average age of 18, predominantly White, with men comprising the majority. Based upon survey response distributions, three distinct mindset categories were identified: Hesitant, Hopeful, and Confident, describing how strongly students rejected a fixed mindset in physics. The results suggested large gender differences in distributions at the high and low-end groups. We also found an overall decline toward fixed mindsets across the course, and logistic regressions controlling for initial mindsets showed that women were significantly more likely than men to shift away from the Confident category. While the majority of men tended to stay within the Confident category, the majority of women moved away from it. Particularly, this differential shift was seen among students receiving Bs or Cs, the most commonly awarded grades in this course. Furthermore, there were relatively small differences in the probability of change within men as a function of grades received, whereas women showed marked declines toward fixed beliefs with either a B or C. Our findings provide empirical evidence for the dynamic, grade-sensitive nature of students' mindsets in a calculus-based physics course.

physics.ed-ph↗

arXiv2026

Mechanics Cognitive Diagnostic: Testing Fine-Grained Learning Objectives in Introductory Physics

Physics courses use research-based assessments (RBAs) such as the Force Concept Inventory (FCI), Force and Motion Conceptual Evaluation (FMCE), and Energy and Momentum Conceptual Survey (EMCS) to measure learning in introductory mechanics, but their fixed-length, pretest-posttest design makes them retrospective: posttest scores summarize completed instruction and arrive after a course ends. We are developing the Mechanics Cognitive Diagnostic (MCD), a cognitive diagnostic computerized adaptive test that reports students' mastery of fine-grained learning objectives (LOs) throughout instruction. Using evidence-centered design, we defined 14 LOs from introductory mechanics textbooks and AP Physics standards, mapped FCI, FMCE, and EMCS items onto them with a Q-matrix, and refined the mapping with the deterministic inputs, noisy "and" gate (DINA) model, using posttest responses from 24,394 students in 807 courses across 79 institutions through LASSO. The FCI and EMCS achieved good DINA model fit; the FMCE showed marginal fit. Classification accuracy for most LOs met or exceeded benchmarks for low-stakes formative assessment. RBA items, though not developed for LO-level diagnosis, support it reliably, giving the MCD a working 14-LO item bank built from RBAs that physics courses already use. As data accumulate, we can revise or retire weak LOs and items and add new items through online calibration without interrupting testing. We plan to expand the MCD to 35 LOs, two per week, to cover a typical introductory mechanics course.

physics.ed-ph↗

arXiv2026

GW Explorer: A Beginner's Guide -- Developing a Computational Gravitational-Wave Outreach Curriculum for High School Students

We present GW Explorer: A Beginner's Guide, an outreach curriculum designed to introduce high school students to gravitational-wave (GW) astrophysics through interactive Python Jupyter notebooks. Most existing GW resources target beginning audiences and advanced students, leaving a gap at the pre-college level that we directly address. The curriculum integrates foundational physics with hands-on computation implemented through both self-directed and workshop-based instructional formats. In the self-directed format, students completed the curriculum independently on cloud-based platforms such as Google Colab. During the workshop format, students worked through the same activities under the guidance of University of Nevada, Las Vegas graduate student mentors. Topics span gravity, spacetime, GW sources, interferometric detection, and data analysis. An implementation in local high school classrooms informed the content and pacing, and survey results demonstrate gains in conceptual understanding and coding confidence. GW Explorer offers a scalable, open-access framework for authentic astrophysics research in the high school classroom.

physics.ed-ph↗