Designing an AI Agent That Does Not Tutor: Design Rules for Socratic Probing in Introductory Physics

Abstract

Instructors need to know whether a student understands a concept, not only whether the student can produce a correct answer, and the two have come apart: a student can paste a physics question into a chatbot and submit a fluent, correct answer without understanding it. We built a conversational AI agent for introductory mechanics that only probes. It is designed as an instrument that a correct answer alone should not pass. Its key measure is cave detection: identifying a correct first idea that the student abandons when a single variable of the scenario is changed. A single right-or-wrong item cannot see this. The agent asks a student to commit to a prediction about an everyday situation, explain it without physics vocabulary, and defend that explanation against a changed case built from the student's own words. It does not teach, hint, confirm, or correct, and it reports nothing until the end. Such an instrument works only if its replies give nothing away. LLM agents tend to leak evaluation: praise, word choice, reply length, and pacing reveal whether an answer was right, and a student who knows stops reasoning. We report design rules that suppress this leakage, derived from eight versions of the agent and more than 40 scripted simulation runs. The obvious fixes often made leakage worse: listing forbidden praise phrases increased praise, and a single worked example in the prompt spread into unrelated conversations. Each changed case had to alter exactly one variable; otherwise a changed answer could not be traced to a single idea. The evidence is simulation evidence only: no real students have used the agent, and the author wrote both the personas and the scoring rules. This paper reports design findings, not learning outcomes.