Searcharxiv⌕ Search

arXiv subjects

Fareya Ikram

Publications and source records attributed to Fareya Ikram.

6 recordsLinked to original sources

Principal Trait Analysis: Identifying User Behavioral Traits in Human-AI Collaboration

Human-artificial intelligence (AI) collaboration takes many forms and occur in both professional and educational scenarios. It is important to identify how human users interact with AI and how their behavior affects task success and learning outcomes. Existing approaches often rely on predefined behavioral categories or derive patterns without establishing their relationship to outcomes. In this work, we propose Principal Trait Analysis (PTA), a data-driven method for identifying interpretable behavioral traits from human-AI collaboration dialogues. Our method analyzes large corpora of human-AI collaborative session traces, derive common traits across the dataset, scores each human collaborator on how well their interaction aligns with these traits, and selects a set that covers the most variance across users. While unsupervised and data-driven, our method also enables domain expertise to be injected during trait identification. We evaluate PTA on two human-AI collaborative coding datasets and investigate how the traits predict future task success, learning outcomes, and evolve over time.

cs.CL↗

Many Paths to the Same Mistake: Generating Diverse Reasoning Paths Behind the Same Student Mistake

Realistic evaluation of AI systems requires user simulators that capture not only cooperative behavior, but also the errors and variability that arise in real interactions. In educational technology, simulating step-by-step flawed student reasoning is essential, yet datasets mapping these pathways are scarce, and Large Language Models (LLMs) often auto-correct rather than sustain simulated mistakes. To address this, we present a generate-and-filter pipeline that produces diverse and realistic student error traces. Our approach utilizes forward consistency, a metric verifying if a flawed reasoning trace can be used to recover the student's actual incorrect answer, paired with an LLM-as-a-judge to ensure realism. Our evaluation suggests that while many LLMs struggle to maintain incorrect reasoning mid-trace, smaller supervised fine-tuned models successfully capture these divergent paths. This work provides a framework for high-fidelity error trace generation, advancing realistic, non-cooperative user simulation.

cs.CL↗

Mathematics Teachers Interactions with a Multi-Agent System for Personalized Problem Generation

Large language models can increasingly adapt educational tasks to learners characteristics. In the present study, we examine a multi-agent teacher-in-the-loop system for personalizing middle school math problems. The teacher enters a base problem and desired topic, the LLM generates the problem, and then four AI agents evaluate the problem using criteria that each specializes in (mathematical accuracy, authenticity, readability, and realism). Eight middle school mathematics teachers created 212 problems in ASSISTments using the system and assigned these problems to their students. We find that both teachers and students wanted to modify the fine-grained personalized elements of the real-world context of the problems, signaling issues with authenticity and fit. Although the agents detected many issues with realism as the problems were being written, there were few realism issues noted by teachers and students in the final versions. Issues with readability and mathematical hallucinations were also somewhat rare. Implications for multi-agent systems for personalization that support teacher control are given.

cs.AI↗

A Multi-Agent Approach to Validate and Refine LLM-Generated Personalized Math Problems

Students benefit from math problems contextualized to their interests. Large language models (LLMs) offer promise for efficient personalization at scale. However, LLM-generated personalized problems may often have problems such as unrealistic quantities and contexts, poor readability, limited authenticity with respect to students' experiences, and occasional mathematical inconsistencies. To alleviate these problems, we propose a multi-agent framework that formalizes personalization as an iterative generate--validate--revise process; we use four specialized validator agents targeting the criteria of solvability, realism, readability, and authenticity, respectively. We evaluate our framework on 600 problems drawn from a popular online mathematics homework platform, ASSISTments, personalizing each problem to a fixed set of 20 student interest topics. We compare three refinement strategies that differ in how validation feedback is coordinated into revisions. Results show that authenticity and realism are the most frequent failure modes in initial LLM-personalized problems, but that a single refinement iteration substantially reduces these failures. We further find that different refinement strategies have different strengths on different criteria. We also assess validator reliability via human evaluation. Results show that reliability is highest on realism and lowest on authenticity, highlighting the need for better evaluation protocols that consider teachers' and students' personal characteristics.

cs.CY↗

The StudyChat Dataset: Analyzing Student Dialogues With ChatGPT in an Artificial Intelligence Course

The widespread availability of large language models (LLMs), such as ChatGPT, has significantly impacted education, raising both opportunities and challenges. Students can frequently interact with LLM-powered, interactive learning tools, but their usage patterns need to be observed and understood. We introduce StudyChat, a publicly available dataset capturing real-world student interactions with an LLM-powered tutoring chatbot in a semester-long, university-level artificial intelligence (AI) course. We deploy a web application that replicates ChatGPT's core functionalities, and use it to log student interactions with the LLM while working on programming assignments. We collect 16,851 interactions, which we annotate using a dialogue act labeling schema inspired by observed interaction patterns and prior research. We analyze these interactions, highlight usage trends, and analyze how specific student behavior correlates with their course outcome. We find that students who prompt LLMs for conceptual understanding and coding help tend to perform better on assignments and exams. Moreover, students who use LLMs to write reports and circumvent assignment learning objectives have lower outcomes on exams than others. StudyChat serves as a shared resource to facilitate further research on the evolving role of LLMs in education.

cs.AI↗

Exploring LLMs for Predicting Tutor Strategy and Student Outcomes in Dialogues

Tutoring dialogues have gained significant attention in recent years, given the prominence of online learning and the emerging tutoring abilities of artificial intelligence (AI) agents powered by large language models (LLMs). Recent studies have shown that the strategies used by tutors can have significant effects on student outcomes, necessitating methods to predict how tutors will behave and how their actions impact students. However, few works have studied predicting tutor strategy in dialogues. Therefore, in this work we investigate the ability of modern LLMs, particularly Llama 3 and GPT-4o, to predict both future tutor moves and student outcomes in dialogues, using two math tutoring dialogue datasets. We find that even state-of-the-art LLMs struggle to predict future tutor strategy while tutor strategy is highly indicative of student outcomes, outlining a need for more powerful methods to approach this task.

cs.CL↗