Searcharxiv⌕ Search

arXiv subjects

Geoffrey Challen

Publications and source records attributed to Geoffrey Challen.

7 recordsLinked to original sources

Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline

Oral exams are compelling because (1) students can be asked to provide their reasoning for an answer and (2) their interactive nature permits follow-up questions to resolve ambiguous answers. But oral exams scale poorly. In this paper, we present scalable conversational assessment, an LLM-based system that conducts interactive text-based conversations with students to assess their ability to make and support an argument. We use a two-agent architecture to separate concerns and reduce the likelihood that answers are leaked to the student: the interviewer interacts directly with the student, but only receives guidance about the next question to ask the student; the hidden evaluator knows the goals of the whole activity, tracks the student's progress using a rubric, and guides the interviewer on when to follow-up on the current question or proceed to the next question. The system was first tested using a collection of simulated students with LLM personas of a particular ability level or adversarial tactic (e.g., answer-extractors, prompt-injectors, confident fabricators). A novel element of this evaluation was that it required the use of a fictional assessment topic to prevent personas that shouldn't have information from using knowledge present in the LLM's pre-training. The system was also deployed in a small, elective AI literacy course in both formative and summative contexts. Through manual and post-facto AI review of the transcripts we identified opportunities to improve the follow-up question generation and ensure that grading only credited ideas that the student (and not the interviewer) contributed.

cs.CY↗

AI Can Do Your Homework. Now What? Report from an Online Workshop on Computing Assessment in the Age of Generative AI

On 28 July 2026, the SIGCSE Virtual 2026 Working Group on computing assessment and generative AI held an open online workshop attended by 73 computing educators. The first hour ran seven strategy rooms, one per approach to adapting (or deliberately preserving) computing assessment in the AI era: open-ended and authentic task design; ambitious, AI-leveraged projects; evaluating and fixing AI-produced work; controlled and AI-free assessment; process evidence and effort signals; rubric and grading redesign; and oral and interactive assessment. The second hour ran question rooms seeded from those strategies, plus two cross-cutting rooms on fairness and trust and student motivation. This report records what was discussed: the approaches participants have tried, the results they reported, and the questions every room left open. It is the first public artifact of the working group, whose taxonomy of computing assessments that accommodate generative AI use will follow.

cs.CY↗

Implementation of Split Deadlines in a Large CS1 Course

Office hour utilization in computer science courses can spike near deadlines, producing long wait times, frustrated students, and overworked staff. To address this problem, a large CS1 course implemented a split deadlines policy. Students were randomly divided into two groups with staggered release and due dates. Each group had the same amount of time to complete assignments, but the number of students with each due date was reduced by half. Our study evaluates the effectiveness of this policy. We measure office hour utilization and staff efficiency near deadlines, examine the policy's impact on student performance, and investigate student perception of the policy's fairness and effectiveness. Overall we found that the split deadline policy increased office hour efficiency, resulted in no significant difference in performance between groups, and was considered fair and effective by most students. Our experience report includes reflections and student feedback indicating how to implement and further improve similar policies.

cs.CY↗

Investigating the Presence and Development of Student Instructor Preferences in a Large-Scale CS1 Course

Prior research has established the importance of student instructor preferences and identified various influencing factors. However, the dynamics of how student instructor preferences develop and change are less well understood, due to the limitations of common course structures and reliance on one-time measurements. To bridge this gap, we utilize data from a novel learning platform that provides students with access to instructional content created by multiple instructors. This platform enables the quantification of preference emergence and evolution throughout an entire semester, as students repeatedly select content from different instructors. Examining both initial and final student instructor preferences suggests that preference is a dynamic construct continually shaped by experiences. Furthermore, our analysis of the associations between preferences and student characteristics reveals a nuanced picture: while student attributes did not significantly correlate with initial preferences, substantial differences emerged in final preferences across genders and self-reported prior programming experience. This analysis contributes to the existing body of knowledge by expanding our understanding of student instructor preferences and student-instructor relationships in computer science education. We also provide practical insights that institutions and instructors can draw on when multiple instructors collaborate on a course.

cs.CY↗

Accelerating Accurate Assignment Authoring Using Solution-Generated Autograders

Students learning to program benefit from access to large numbers of practice problems. Autograders are commonly used to support programming questions by providing quick feedback on submissions. But authoring accurate autograders remains challenging. Autograders are frequently created by enumerating test cases--a tedious process that can produce inaccurate autograders that fail to correctly classify submissions. When authoring accurate autograders is slow, it is difficult to create large banks of practice problems to support beginning programmers. We present solution-generated autograding: a faster, more accurate, and more enjoyable way to create autograders. Our approach leverages a key difference between software testing and autograding: The question author can provide a solution. By starting with a solution, we can eliminate the need to manually enumerate test cases, validate the autograder's accuracy, and evaluate other aspects of submission code quality beyond behavioral correctness. We describe Questioner, an implementation of solution-generated autograding for Java and Kotlin, and share experiences from four years using Questioner to support a large CS1 course: authoring nearly 800 programming questions used by thousands of students to evaluate millions of submissions.

cs.CY↗

Lessons from Four Years of PHONELAB Experimentation

Over the last four years we have operated a public smartphone platform testbed called PHONELAB. PHONELAB consists of up to several-hundred participants who run an experimental platform image on their primary smartphone. The experimental platform consists of both instrumentation and experimental changes to platform components, including core Android services and Linux. This paper describes the design of the testbed, the process of conducting PHONELAB experiments, and some of the research the testbed has supported. We also offer many lessons learned along the way, almost all of which have been learned the hard way--through trial and a lot of error. We expect our experiences will help those contemplating operating large user-facing testbeds, anyone conducting experiments on smartphones, and many mobile systems researchers.

cs.NI↗

VeriFi: Model-Driven Runtime Verification Framework for Wireless Protocol Implementations

Validating wireless protocol implementations is challenging. Today's approaches require labor-intensive experimental setup and manual trace investigation, but produce poor coverage and inaccurate and irreproducible results. We present VERIFI, the first systematic sniffer-based, model-guided runtime verification framework for wireless protocol implementations. VERIFI takes a formal model of the protocol being verified as input. To achieve good coverage, it first applies state reachability analysis by applying model checking techniques. It then uses a new PACKETSNIPER component to selectively trigger packet losses required to quickly investigate all reachable protocol states. Our results show that the selective packet jamming allows VERIFI to significantly improve the coverage of protocol states. Finally, VERIFI accommodates uncertainty caused by the sniffer when validating traces, allowing it to provide accurate and reproducible results. By modeling uncertainty, VERIFI highlights likely protocol violations for developers to examine.

cs.NI↗