arXiv subjects

N. Sanjay Rebello

Publications and source records attributed to N. Sanjay Rebello.

33 records · Page 2Linked to original sources

arXiv2025

Investigation of the Inter-Rater Reliability between Large Language Models and Human Raters in Qualitative Analysis

Qualitative analysis is typically limited to small datasets because it is time-intensive. Moreover, a second human rater is required to ensure reliable findings. Artificial intelligence tools may replace human raters if we demonstrate high reliability compared to human ratings. We investigated the inter-rater reliability of state-of-the-art Large Language Models (LLMs), ChatGPT-4o and ChatGPT-4.5-preview, in rating audio transcripts coded manually. We explored prompts and hyperparameters to optimize model performance. The participants were 14 undergraduate student groups from a university in the midwestern United States who discussed problem-solving strategies for a project. We prompted an LLM to replicate manual coding, and calculated Cohen's Kappa for inter-rater reliability. After optimizing model hyperparameters and prompts, the results showed substantial agreement ($κ>0.6$) for three themes and moderate agreement on one. Our findings demonstrate the potential of GPT-4o and GPT-4.5 for efficient, scalable qualitative analysis in physics education and identify their limitations in rating domain-general constructs.

physics.ed-ph↗

arXiv2025

AI Reasoning Models for Problem Solving in Physics

Reasoning models are the new generation of Large Language Models (LLMs) capable of complex problem solving. Their reliability in solving introductory physics problems was tested by evaluating a sample of n = 5 solutions generated by one such model -- OpenAI's o3-mini -- per each problem from 20 chapters of a standard undergraduate textbook. In total, N = 408 problems were given to the model and N x n = 2,040 generated solutions examined. The model successfully solved 94% of the problems posed, excelling at the beginning topics in mechanics but struggling with the later ones such as waves and thermodynamics.

physics.ed-ph↗

arXiv2025

Sense of Belonging and Intent to Persist: Mediating Role of Motivation and Moderating Role of Gender in Physics and Astronomy Graduate Students

This study investigates how graduate students' sense of belonging (SB) influences their intent to persist (IP) in physics and astronomy programs, and how this relationship is shaped by the basic psychological needs that drive motivation-autonomy, competence, and relatedness-as well as gender. Grounded in self-determination theory, the analysis treats these three needs as mediators and gender as a moderator. A quantitative survey was administered to graduate students in the Department of Physics and Astronomy at a large public land-grant R1 Midwestern university in the USA. Using probit regressions, we found that SB significantly predicts IP. Autonomy may play a compensatory role when SB is high, competence amplifies the effect of SB on IP, and relatedness buffers against low SB. Gender moderates the relationship: women report lower IP at low levels of SB but exceed men when SB is strong. These findings underscore the importance of fostering a sense of belonging, academic confidence, and social connection-particularly for women in male-dominated STEM fields.

physics.ed-ph↗

arXiv2025

Students' Perceptions to a Large Language Model's Generated Feedback and Scores of Argumentation Essays

Students in introductory physics courses often rely on ineffective strategies, focusing on final answers rather than understanding underlying principles. Integrating scientific argumentation into problem-solving fosters critical thinking and links conceptual knowledge with practical application. By facilitating learners to articulate their scientific arguments for solving problems, and by providing real-time feedback on students' strategies, we aim to enable students to develop superior problem-solving skills. Providing timely, individualized feedback to students in large-enrollment physics courses remains a challenge. Recent advances in Artificial Intelligence (AI) offer promising solutions. This study investigates the potential of AI-generated feedback on students' written scientific arguments in an introductory physics class. Using Open AI's GPT-4o, we provided delayed feedback on student written scientific arguments and surveyed them about the perceived usefulness and accuracy of this feedback. Our findings offer insights into the viability of implementing real-time AI feedback to enhance students' problem-solving and metacognitive skills in large-enrollment classrooms.

physics.ed-ph↗

arXiv2025

Dual-Role Dynamics in Prompting: Elementary Pre-service Teachers' AI Prompting Strategies for Representational Choices

Pre-service teachers play a unique dual role as they straddle between the roles of students and future teachers. This dual role requires them to adopt both the learner's and the instructor's perspectives while engaging with pedagogical and content knowledge. The current study investigates how pre-service elementary teachers taking a physical science course prompt AI to generate representations that effectively communicate conceptual ideas to two distinct audiences. The context involves participants interacting with AI to generate appropriate representations that explain the concepts of wave velocity to their elementary students (while casting themselves as teachers) and the Ideal Gas Law to their English teachers (while casting themselves as students). Emergent coding of the AI prompts highlight that, when acting as teachers, participants were more explicit in specifying the target audience, predetermining the type of representation, and producing a broader variety of representations compared to when they acted as students. Implications of the observed 'exploratory' and 'prescriptive' prompting trends across the two roles on pre-service teachers' education and their professional development are discussed.

physics.ed-ph↗

arXiv2025

Analyzing Undergraduate Problem-Solving in Physics Through Interaction With an AI Chatbot

Providing individualized scaffolding for physics problem solving at scale remains an instructional challenge. We investigate (1) students' perceptions of a Socratic Artificial Intelligence (AI) chatbot's impact on problem-solving skills and confidence and (2) how the specificity of students' questions during tutoring relates to performance. We deployed a custom Socratic AI chatbot in a large-enrollment introductory mechanics course at a Midwestern public university, logging full dialogue transcripts from 150 first-year STEM majors. Post-interaction surveys revealed median ratings of 4.0/5 for knowledge-based skills and 3.4/5 for overall effectiveness. Transcript analysis showed question specificity rose from approximately 10-15% in the first turn to 100% by the final turn, and specificity correlated positively with self reported expected course grade (Pearson r = 0.43). These findings demonstrate that AI-driven Socratic dialogue not only fosters expert-like reasoning but also generates fine-grained analytics for physics education research, establishing a scalable dual-purpose tool for instruction and learning analytics.

physics.ed-ph↗

arXiv2025

Using an LLM to Investigate Students' Explanations on Conceptual Physics Questions

Analyzing students' written solutions to physics questions is a major area in PER. However, gauging student understanding in college courses is bottlenecked by large class sizes, which limits assessments to a multiple-choice (MC) format for ease of grading. Although sufficient in quantifying scientifically correct conceptions, MC assessments do not uncover students' deeper ways of understanding physics. Large language models (LLMs) offer a promising approach for assessing students' written responses at scale. Our study used an LLM, validated by human graders, to classify students' written explanations to three questions on the Energy and Momentum Conceptual Survey as correct or incorrect, and organized students' incorrect explanations into emergent categories. We found that the LLM (GPT-4o) can fairly assess students' explanations, comparable to human graders (0-3% discrepancy). Furthermore, the categories of incorrect explanations were different from corresponding MC distractors, allowing for different and deeper conceptions to become accessible to educators.

physics.ed-ph↗

arXiv2025

From Self-Crafted to Engineered Prompts: Student Evaluations of AI-Generated Feedback in Introductory Physics

The abilities of Generative-Artificial Intelligence (AI) to produce real-time, sophisticated responses across diverse contexts has promised a huge potential in physics education, particularly in providing customized feedback. In this study, we investigate around 1200 introductory students' preferences about AI-feedback generated from three distinct prompt types: (a) self-crafted, (b) entailing foundational prompt-engineering techniques, and (c) entailing foundational prompt-engineering techniques along with principles of effective-feedback. The results highlight an overwhelming fraction of students preferring feedback generated using structured prompts, with those entailing combined features of prompt engineering and effective feedback to be favored most. However, the popular choice also elicited stronger preferences with students either liking or disliking the feedback. Students also ranked the feedback generated using their self-crafted prompts as the least preferred choice. Students' second preferences given their first choice and implications of the results such as the need to incorporate prompt engineering in introductory courses are discussed.

physics.ed-ph↗

arXiv2025

Exploring the Feasibility of Employing a Hybrid Machine Learning Method to Unpack Student Reasoning Patterns in Physics Essays

We propose a novel clustering pipeline that combines two classic clustering algorithms to better understand student problem-solving strategies. This unsupervised machine learning method helps uncover patterns in reasoning without pre-defined labels. We applied it to essays written for an online multiple-choice quiz, the resulting clusters showed strong statistical alignment with students' selected answers. We also report on the resulting clusters of the hybrid pipeline compared to that of K-Means (MacQueen, 1967) and Hierarchal Density-Based Spatial Clustering of Application with Noise (HDBSCAN) (McInnes, Healy, and Astels, 2017) by analyzing the Scatter Plots, Silhouette Scores (Rousseeuw, 1987), and Davies Bouldin Index (Davies and Bouldin, 1979).

physics.ed-ph↗

arXiv2025

Applying a STEM Ways of Thinking Framework for Student-generated Engineering Design-based Physics Problems

This second paper in a multi-part series builds on the first, which introduced the Ways of Thinking for Engineering Design-based Physics (WoT4EDP) framework for STEM education in an introductory undergraduate physics course. Here, we apply the framework to analyze transcripts of group discussions and written reports from 14 student teams as they engaged in a self-generated engineering design (ED) problem in an introductory physics laboratory. We qualitatively examine: (i) the aspects students address in their problem statements; (ii) how they engage in design-based, science-based, and mathematics-based thinking, as well as metacognitive reflection, while developing solutions; and (iii) how they incorporate computational thinking through Python coding. Key findings highlight the need for: (i) increased guidance for iterative problem framing; (ii) structured support for assessing design limitations, engaging in a feasibility study, adopting a systematic approach to applying physics and mathematics in their iterations, and making specific metacognitive reflections; and (iii) integration of Python-based activities into laboratory tasks with appropriate scaffolding. We present our findings through a detailed qualitative analysis, drawing extensively from the qualitative methods literature. We outline our analytical approach, present coding charts, and employ qualitative methods such as thematic analysis and thick description to convey our findings. In doing so, we contribute to ongoing efforts to enhance the rigor of qualitative analysis in physics education research (PER). Based on our analysis, we provide valuable insight for educators and researchers in designing physics-based engineering design tasks and promoting interdisciplinary problem-solving in STEM education.

physics.ed-ph↗

arXiv2025

Presenting a STEM Ways of Thinking Framework for Engineering Design-based Physics Problems

Investigating students' thinking in classroom tasks, particularly in science and engineering, is essential for improving educational practices and advancing student learning. In this context, the notion of Ways of Thinking (WoT) has gained traction in STEM education, offering a framework to explore how students approach and solve interdisciplinary problems. Building on our earlier studies and contributing to ongoing discussions on WoT frameworks, this paper introduces a new WoT framework: Ways of Thinking in Engineering Design based Physics (WoT4EDP). WoT4EDP integrates five key elements: design, science, mathematics, metacognitive reflection, and computational thinking within an undergraduate introductory physics laboratory. This framework offers a novel perspective by emphasizing how these interconnected elements work together to foster deeper learning and holistic problem-solving in Engineering Design based projects. A key takeaway is that this framework serves as a practical tool for educators and researchers to design, implement, and analyze interdisciplinary STEM activities in physics classrooms. We describe the development of WoT4EDP, situate it within the broader landscape of undergraduate STEM education, and provide detailed characterizations of its components. Additionally, we compare WoT4EDP with two contemporary frameworks: Dalal et al. (2021) and English (2023), to glean insights that enhance its application and promote interdisciplinary thinking. This paper is the first of a two-part series. In the upcoming second part, we will demonstrate the application of the WoT4EDP framework, showcasing how it can be used to analyze student thinking in real-world, ED-based physics projects.

physics.ed-ph↗

arXiv2024

Investigating the Design-Science Connection in a multi-week Engineering Design (ED)-based introductory physics laboratory task

Reform documents advocate for innovative pedagogical strategies to enhance student learning. A key innovation is the integration of science and engineering practices through Engineering Design (ED)-based physics laboratory tasks, where students tackle engineering design problems by applying physics principles. While this approach has its benefits, research shows that students do not always effectively apply scientific concepts, but instead rely on trial-and-error approaches, and end up 'gadgeteering' their way to a solution. This leads to what is commonly referred to as the "design-science gap" -- that students do not always consciously apply science concepts while solving a design problem. However, as obvious as the notion of a `gap' may appear, there seems to exist no consensus on the definitions of `design' and `science', further complicating the understanding of this `gap'. This qualitative study addresses the notion of the design-science gap by examining student-groups' discussions and written lab reports from a multi-week ED-based undergraduate introductory physics laboratory task. Building on our earlier studies, we developed and employed a nuanced, multi-layered coding scheme inspired by the Gioia Framework to characterize `design thinking' and `science thinking'. We discuss how student-groups engage in various aspects of design and how they apply concepts physics principles to solve the problem. In the process, we demonstrate the interconnectedness of students' design thinking and science thinking. We advocate for the usage of the term "design-science connection" as opposed to "design-science gap" to deepen both design and scientific thinking. Our findings offer valuable insights for educators in design-based science education.

physics.ed-ph↗

arXiv2018

Use of Eye-Tracking Technology to Investigate Cognitive Load Theory

Cognitive load theory (CLT) provides us guiding principles in the design of learning materials. CLT differentiates three different kinds of cognitive load -- intrinsic, extraneous and germane load. Intrinsic load is related to the learning goal, extraneous load costs cognitive resources but does not contribute to learning. Germane load can foster learning. Objective methods, such as eye movement measures and EEG have been used measure the total cognitive load. Very few research studies, if any, have been completed to measure the three kinds of load separately with physiological methods in a continuous manner. In this current study, we will show how several eye-tracking based parameters are related to the three kinds of load by having explicit manipulation of the three loads independently. Participants having low prior knowledge regarding the learning material participated in the study. Working memory capacity was also measured by an operation memory span task.

physics.ed-ph↗

arXiv2017

Effect of visual cues and video solutions on conceptual tasks

Transfer of learning is an important objective of education. However, students usually have difficulties in solving physics transfer tasks even after having solved similar problems previously. We investigated if instruction provided using videos containing detailed explanations of previously solved problems will improve students' performance in tackling near and far transfer tasks. We also investigated whether the combination of visual cues followed by video solutions yields further enhancement of students' performance. N=33 students in an algebra-based physics class participated in an interview containing two problem sets each with one initial task, a training session, and a near and far transfer task. For the training, students either received visual cues, visual cues and a video or only videos, depending on the condition. We compare students' correctness rate on near and far transfer tasks in the three conditions.

physics.ed-ph↗

arXiv2003

Students' Models of Newton's Second Law in Mechanics and Electromagnetism

We investigated students' use of Newton's second law in mechanics and electromagnetism contexts by interviewing students in a two-semester calculus-based physics course. We observed that students' responses are consistent with three mental models. These models appeard in mechanics contexts and were transferred to electromagnetism contexts. We developed an inventory to help instructors identify these models and direct students towards the correct one.

physics.ed-ph↗