arXiv subjects

Ralf Widenhorn

Publications and source records attributed to Ralf Widenhorn.

3 recordsLinked to original sources

arXiv2026

Feedback Indices to Evaluate LLM Responses to Rebuttals for Multiple Choice Type Questions

We present a systematic framework of indices designed to characterize Large Language Model (LLM) responses when challenged with rebuttals during a chat. Assessing how LLMs respond to user dissent is crucial for understanding their reliability and behavior patterns, yet the complexity of human-LLM interactions makes systematic evaluation challenging. Our approach employs a fictitious-response rebuttal method that quantifies LLM behavior when presented with multiple-choice questions followed by deliberate challenges to their fictitious previous response. The indices are specifically designed to detect and measure what could be characterized as sycophantic behavior (excessive agreement with user challenges) or stubborn responses (rigid adherence to the fictitious response in the chat history) from LLMs. These metrics allow investigation of the relationships between sycophancy, stubbornness, and the model's actual mastery of the subject matter. We demonstrate the utility of these indices using two physics problems as test scenarios with various OpenAI models. The framework is intentionally generalizable to any multiple-choice format question, including on topics without universally accepted correct answers. Our results reveal measurable differences across OpenAI model generations, with trends indicating that newer models and those employing greater "Reasoning Effort" exhibit reduced sycophantic behavior. The FR pairing method combined with our proposed indices provides a practical, adaptable toolkit for systematically comparing LLM dialogue behaviors across different models and contexts.

physics.ed-ph↗

arXiv2025

Translating the Force Concept Inventory in the age of AI

We present a study that translates the Force Concept Inventory (FCI) using OpenAI GPT-4o and assess the specific difficulties of translating a scientific-focused topic using Large Language Models (LLMs). The FCI is a physics exam meant to evaluate outcomes of a student cohort before and after instruction in Newtonian physics. We examine the problem-solving ability of the LLM in both the translated document and the translation back into English, detailing the language-dependent issues that complicate the translation. While ChatGPT performs remarkably well on answering the questions in both the translated language as well as the back-translation into English, problems arise with language-specific nuances and formatting. Pitfalls include words or phrases that lack one-to-one matching terms in another language, especially discipline-specific scientific terms, or outright mistranslations. Depending on the context, these translations can result in a critical change in the physical meaning of the problem. Additionally, issues with question numbering and lettering are found in some languages. The issues around the translations of numbering and lettering provide insight into the abilities of the LLM and suggest that it is not simply relying upon FCI questions that may have been part of the LLM training data to provide answers. These findings underscore that while LLMs can accelerate multilingual access to educational tools, careful review is still needed to ensure fidelity and clarity in translated assessments. LLMs provide a new opportunity to expand educational tools and assessments. At the same time, there are unique challenges using LLMs to facilitate translations that this case study examines in detail.

physics.ed-ph↗

arXiv2025

Multilingual Performance of a Multimodal Artificial Intelligence System on Multisubject Physics Concept Inventories

We investigate the multilingual and multimodal performance of a large language model-based artificial intelligence (AI) system, GPT-4o, using a diverse set of physics concept inventories spanning multiple languages and subject categories. The inventories, sourced from the PhysPort website, cover classical physics topics such as mechanics, electromagnetism, optics, and thermodynamics, as well as relativity, quantum mechanics, astronomy, mathematics, and laboratory skills. Unlike previous text-only studies, we uploaded the inventories as images to reflect what a student would see on paper, thereby assessing the system's multimodal functionality. Our results indicate variation in performance across subjects, with laboratory skills standing out as the weakest. We also observe differences across languages, with English and European languages showing the strongest performance. Notably, the relative difficulty of an inventory item is largely independent of the language of the survey. When comparing AI results to existing literature on student performance, we find that the AI system outperforms average post-instruction undergraduate students in all subject categories except laboratory skills. Furthermore, the AI performs worse on items requiring visual interpretation of images than on those that are purely text-based. While our exploratory findings show GPT-4o's potential usefulness in physics education, they highlight the critical need for instructors to foster students' ability to critically evaluate AI outputs, adapt curricula thoughtfully in response to AI advancements, and address equity concerns associated with AI integration.

physics.ed-ph↗