arXiv subjects

Ryan Baker

Publications and source records attributed to Ryan Baker.

8 recordsLinked to original sources

arXiv2026

When Rubrics Change: Cross-Rubric Generalization for Critical Thinking Essay Scoring

Automated essay scoring (AES) research has largely focused on cross-prompt generalization, where essays from unseen prompts are scored while the scoring criteria are typically held constant. In practice, however, educators may revise or even introduce new rubrics in their scoring task, to evaluate different aspects of essays. We study cross-rubric generalization: training on essays labeled under one set of rubrics and evaluating on previously unseen rubrics, which target different aspects of the essay. We use a Large Language Model (LLM) fine-tuning framework with two components: rubric-agnostic intermediate representations, called traits, and target-essay supervision under seen rubrics during training. On an AES dataset augmented with multiple rubric-defined labels of student critical thinking skills, we find that traits improve macro F1 by 5.0% over a baseline without traits in the hardest setting, where both target rubrics and target essays are unseen during training. We further find that increasing target-essay supervision improves performance, with our best fine-tuned open-source Llama-based model outperforming GPT-5-mini prompting by 2.1% macro F1 and trailing GPT-5 by 1.9%. These results show that trait-based intermediate structure and controlled supervision improve generalization to unseen rubrics.

cs.CL↗

arXiv2025

What do you say? A pilot study investigating student responses in Data Driven Classroom Interviews

Data that contextualizes student interactions with online learning systems can be challenging to obtain. This study looks at the rhetorical strategies of a novel method for conducting in-the-moment Data-Driven Classroom Interviews (DDCIs). By using Ordered Network Analysis (ONA) to reanalyze data from Wei et al.'s (2025) Epistemic Network Analysis, we better account for the sequences in which these rhetorical strategies emerge during the interview process. Specifically, we examine how five rhetorical strategies by interviewers relate to five possible rhetorical strategies used in student responses. As with the previous study, results demonstrate minor differences in how students with high and low situational interest respond. Namely, whereas students with high situational interest show moderately higher levels of enthusiasm, students with low situational interest are more likely to respond to interviewers with an explanation. However, overall this study confirms that there are few interviewer-driven differences in these interviews, and it documents that interviewers are following guidelines to rely upon open-ended questions

cs.HC↗

arXiv2025

Toward LLM-Supported Automated Assessment of Critical Thinking Subskills

As the world becomes increasingly saturated with AI-generated content, disinformation, and algorithmic persuasion, critical thinking - the capacity to evaluate evidence, detect unreliable claims, and exercise independent judgment - is becoming a defining human skill. Developing critical thinking skills through timely assessment and feedback is crucial; however, there has not been extensive work in educational data mining on defining, measuring, and supporting critical thinking. In this paper, we investigate the feasibility of measuring "subskills" that underlie critical thinking. We ground our work in an authentic task where students operationalize critical thinking by writing argumentative essays. We developed a coding rubric based on an established skills progression and completed human coding for a corpus of student essays. We then evaluated three distinct approaches to automated scoring: zero-shot prompting, few-shot prompting, and supervised fine-tuning, implemented across three large language models (GPT-5, Llama 3.1 8B, and ModernBERT). Fine-tuning Llama 3.1 8B achieved the best results and demonstrated particular strength on subskills with highly separable proficiency levels with balanced labels across levels, while lower performance was observed for subskills that required detection of subtle distinctions between proficiency levels or imbalanced labels. Our exploratory work represents an initial step toward scalable assessment of critical thinking skills across authentic educational contexts. Future research should continue to combine automated critical thinking assessment with human validation to more accurately detect and measure dynamic, higher-order thinking skills.

cs.CY↗

arXiv2022

Vehicle Guidance and Tracking Systems

Our application of command and control is the Aegis Combat System. Major components of this system include missile guidance and missile tracking. To look further into some of the aspects of these systems, an extremely simplified model of the Aegis Combat System will be designed. In this simplified model, a small-scale car will autonomously follow a small-scale remote-controlled car. There will be three major components of this system: the controller and the two small-scale cars. Through this model, the team can demonstrate the real-world application of certain aspects of C2 such as command, communication, and sensor data fusion. Figure 1 shows a picture of the Aegis Combat System.

eess.SY↗

arXiv2022

Analyzing Adaptive Scaffolds that Help Students Develop Self-Regulated Learning Behaviors

Providing adaptive scaffolds to help learners develop self-regulated learning (SRL) processes has been an important goal for intelligent learning environments. Adaptive scaffolding is especially important in open-ended learning environments (OELE), where novice learners often face difficulties in completing their learning tasks. This paper presents a systematic framework for adaptive scaffolding in Betty's Brain, a learning-by-teaching OELE for middle school science, where students construct a causal model to teach a virtual agent, generically named Betty. We evaluate the adaptive scaffolding framework and discuss its implications on the development of more effective scaffolds for SRL in OELEs. We detect key cognitive/metacognitive inflection points, i.e., instances where students' behaviors and performance change as they work on their learning tasks. At such inflection points, Mr. Davis (a mentor agent) or Betty (the teachable agent) provide conversational feedback, focused on strategies to help students become productive learners. We conduct a classroom study with 98 middle schoolers to analyze the impact of adaptive scaffolds on students' learning behaviors and performance. Adaptive scaffolding produced mixed results, with some scaffolds (viz., strategic hints that supported debugging and assessment of causal models) being generally more useful to students than others (viz., encouragement prompts). We also note differences in learning behaviors of High and Low performers after receiving scaffolds. Overall, our findings suggest how adaptive scaffolding in OELEs like Betty's Brain can be further improved to narrow the gap between High and Low performers.

cs.CY↗

arXiv2019

The effect of habitats and fitness on species coexistence in systems with cyclic dominance

Cyclic dominance between species may yield spiral waves that are known to provide a mechanism enabling persistent species coexistence. This observation holds true even in presence of spatial heterogeneity in the form of quenched disorder. In this work we study the effects on spatio-temporal patterns and species coexistence of structured spatial heterogeneity in the form of habitats that locally provide one of the species with an advantage. Performing extensive numerical simulations of systems with three and six species we show that these structured habitats destabilize spiral waves. Analyzing extinction events, we find that species extinction probabilities display a succession of maxima as function of time, that indicate a periodically enhanced probability for species extinction. Analysis of the mean extinction time reveals that as a function of the parameter governing the advantage of one of the species a transition between stable coexistence and unstable coexistence takes place. We also investigate how efficiency as a predator or a prey affects species coexistence.

q-bio.PE↗

arXiv2018

Enabling End-To-End Machine Learning Replicability: A Case Study in Educational Data Mining

The use of machine learning techniques has expanded in education research, driven by the rich data from digital learning environments and institutional data warehouses. However, replication of machine learned models in the domain of the learning sciences is particularly challenging due to a confluence of experimental, methodological, and data barriers. We discuss the challenges of end-to-end machine learning replication in this context, and present an open-source software toolkit, the MOOC Replication Framework (MORF), to address them. We demonstrate the use of MORF by conducting a replication at scale, and provide a complete executable container, with unique DOIs documenting the configurations of each individual trial, for replication or future extension at https://github.com/educational-technology-collective/fy2015-replication. This work demonstrates an approach to end-to-end machine learning replication which is relevant to any domain with large, complex or multi-format, privacy-protected data with a consistent schema.

cs.CY↗

arXiv2018

MORF: A Framework for Predictive Modeling and Replication At Scale With Privacy-Restricted MOOC Data

Big data repositories from online learning platforms such as Massive Open Online Courses (MOOCs) represent an unprecedented opportunity to advance research on education at scale and impact a global population of learners. To date, such research has been hindered by poor reproducibility and a lack of replication, largely due to three types of barriers: experimental, inferential, and data. We present a novel system for large-scale computational research, the MOOC Replication Framework (MORF), to jointly address these barriers. We discuss MORF's architecture, an open-source platform-as-a-service (PaaS) which includes a simple, flexible software API providing for multiple modes of research (predictive modeling or production rule analysis) integrated with a high-performance computing environment. All experiments conducted on MORF use executable Docker containers which ensure complete reproducibility while allowing for the use of any software or language which can be installed in the linux-based Docker container. Each experimental artifact is assigned a DOI and made publicly available. MORF has the potential to accelerate and democratize research on its massive data repository, which currently includes over 200 MOOCs, as demonstrated by initial research conducted on the platform. We also highlight ways in which MORF represents a solution template to a more general class of problems faced by computational researchers in other domains.

cs.SE↗