arXiv subjects

Viraj Kumar

Publications and source records attributed to Viraj Kumar.

8 recordsLinked to original sources

arXiv2026

Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications

As AI code tools become integrated into programming environments, students increasingly describe intended behavior in natural language and rely on these tools to generate code, shifting emphasis from code writing to specification. Yet little is known about the comments students write as specifications in AI-assisted programming tasks. We analyze a four-year dataset of undergraduate programming submissions and reflections from tasks in which students wrote comments to guide code generation and refined solutions using test-case feedback. We introduce a taxonomy spanning three dimensions: comment type, code expression level, and code construct. Using automated classification, we examine how these dimensions vary across attempts and how students describe the process in their reflections. Our findings show that students mostly wrote natural-language What comments, shifted toward How comments for more procedural constructs, and focused more on verifying generated code than on repeatedly rewriting comments.

cs.SE↗

arXiv2026

Reshaping Undergraduate Computer Science Education in the Generative AI Era

Generative AI represents a turning point for Computer Science (CS) education. In recent decades, post-secondary CS education has largely focused on what has been seen as practical software engineering skills: implementation-level programming, debugging, testing, and software design, analysis, and documentation. However, this framing is becoming less tenable as generative AI automates many of these tasks, challenging their centrality in CS education. To keep pace with advances in AI technology, CS curricula should consider a shift toward understanding and verifying AI-generated artifacts. This white paper outlines the findings of two international NUS-Google Workshops in Singapore, where we convened faculty members, industry practitioners, and students, and proposes a strategic response to reshape how CS should be taught at the undergraduate level. Based on the findings, we identify critical skills that must be preserved and those that are becoming less important. By incorporating these skills as "breadcrumbs," we can provide helpful nudges and engaging exercises within the current curriculum, enhancing learning experiences for everyone. We believe that to effectively prepare future computer science graduates, capable of creating, solving problems, and managing, as well as co-creating, artifacts with AI. It is important to consider a shift in curricula. Emphasizing system design, abstraction, and critical evaluation could greatly enhance their education and readiness for the challenges ahead. We propose prerequisites for solutions to reform CS education by fostering AI-native competencies, re-centering fundamental education, enhancing advanced pathways, embracing new pedagogies, and shifting institutional support.

cs.CY↗

arXiv2025

Ambiguity Resolution with Human Feedback for Code Writing Tasks

Specifications for code writing tasks are usually expressed in natural language and may be ambiguous. Programmers must therefore develop the ability to recognize ambiguities in task specifications and resolve them by asking clarifying questions. We present and evaluate a prototype system, based on a novel technique (ARHF: Ambiguity Resolution with Human Feedback), that (1) suggests specific inputs on which a given task specification may be ambiguous, (2) seeks limited human feedback about the code's desired behavior on those inputs, and (3) uses this feedback to generate code that resolves these ambiguities. We evaluate the efficacy of our prototype, and we discuss the implications of such assistive systems on Computer Science education.

cs.SE↗

arXiv2025

Probing the Unknown: Exploring Student Interactions with Probeable Problems at Scale in Introductory Programming

Introductory programming courses often rely on small code-writing exercises that have clearly specified problem statements. This limits opportunities for students to practice how to clarify ambiguous requirements -- a critical skill in real-world programming. In addition, the emerging capabilities of large language models (LLMs) to produce code from well-defined specifications may harm student engagement with traditional programming exercises. This study explores the use of ``Probeable Problems'', automatically gradable tasks that have deliberately vague or incomplete specifications. Such problems require students to submit test inputs, or `probes', to clarify requirements before implementation. Through analysis of over 40,000 probes in an introductory course, we identify patterns linking probing behaviors to task success. Systematic strategies, such as thoroughly exploring expected behavior before coding, resulted in fewer incorrect code submissions and correlated with course success. Feedback from nearly 1,000 participants highlighted the challenges and real-world relevance of these tasks, as well as benefits to critical thinking and metacognitive skills. Probeable Problems are easy to set up and deploy at scale, and help students recognize and resolve uncertainties in programming problems.

cs.HC↗

arXiv2024

Explain in Plain Language Questions with Indic Languages: Drawbacks, Affordances, and Opportunities

Background: Introductory computer science courses use ``Explain in Plain English'' (EiPE) activities to develop and assess students' code comprehension skills, but creating effective autograders for these questions is challenging and limited to English. This is a particular challenge in linguistically diverse countries like India where students may have limited proficiency in English. Methods: We evaluate the efficacy of a recently introduced approach called Code Generation Based Grading (CGBG) in enabling language agnostic ``Explain in Plain Language'' (EiPL) activities. Here students' EiPL responses generate code that is tested for functional equivalence to the original which was being described. Objectives: We initially evaluate the correctness of code generated from correct EiPL responses provided in 10 of India's most commonly spoken languages. To evaluate the effectiveness of the approach in practice, we assess student success and perceptions of EiPL questions in a NPTEL (National Programme on Technology Enhanced Learning) course. Results: We find promising results for the correctness of code generated from translations of correct EiPL responses, with most languages achieving a correctness rate of 75% or higher. However, in practice, many students preferred to respond in English due to greater familiarity with English as a technical language, difficulties writing in their native language, and perceptions of the grader being less capable of generating code from prompts in their mother tongue.

cs.CY↗

arXiv2024

Probeable Problems for Beginner-level Programming-with-AI Contests

To broaden participation, competitive programming contests may include beginner-level problems that do not require knowledge of advanced Computer Science concepts (e.g., algorithms and data structures). However, since most participants have easy access to AI code-generation tools, these problems often become trivial to solve. For beginner-friendly programming contests that do not prohibit the use of AI tools, we propose Probeable Problems: code writing tasks that provide (1) a problem specification that deliberately omits certain details, and (2) a mechanism to probe for these details by asking clarifying questions and receiving immediate feedback. To evaluate our proposal, we conducted a 2-hour programming contest for undergraduate Computer Science students from multiple institutions, where each student was an active member of their institution's computing club. The contest comprised of six Probeable Problems for which a popular code-generation tool (GitHub Copilot) was unable to generate accurate solutions due to the absence of details. Students were permitted to work individually or in groups, and were free to use AI tools. We obtained consent from 26 groups (67 students) to use their submissions for research. We analyze the extent to which the code submitted by these groups identifies missing details and identify ways in which Probeable Problems can support learning in formal and informal CS educational contexts.

cs.CY↗

arXiv2023

GuardRails: Automated Suggestions for Clarifying Ambiguous Purpose Statements

Before implementing a function, programmers are encouraged to write a purpose statement i.e., a short, natural-language explanation of what the function computes. A purpose statement may be ambiguous i.e., it may fail to specify the intended behaviour when two or more inequivalent computations are plausible on certain inputs. Our paper makes four contributions. First, we propose a novel heuristic that suggests such inputs using Large Language Models (LLMs). Using these suggestions, the programmer may choose to clarify the purpose statement (e.g., by providing a functional example that specifies the intended behaviour on such an input). Second, to assess the quality of inputs suggested by our heuristic, and to facilitate future research, we create an open dataset of purpose statements with known ambiguities. Third, we compare our heuristic against GitHub Copilot's Chat feature, which can suggest similar inputs when prompted to generate unit tests. Fourth, we provide an open-source implementation of our heuristic as an extension to Visual Studio Code for the Python programming language, where purpose statements and functional examples are specified as docstrings and doctests respectively. We believe that this tool will be particularly helpful to novice programmers and instructors.

cs.SE↗

arXiv2022

Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language

GitHub Copilot is an artificial intelligence model for automatically generating source code from natural language problem descriptions. Since June 2022, Copilot has officially been available for free to all students as a plug-in to development environments like Visual Studio Code. Prior work exploring OpenAI Codex, the underlying model that powers Copilot, has shown it performs well on typical CS1 problems thus raising concerns about the impact it will have on how introductory programming courses are taught. However, little is known about the types of problems for which Copilot does not perform well, or about the natural language interactions that a student might have with Copilot when resolving errors. We explore these questions by evaluating the performance of Copilot on a publicly available dataset of 166 programming problems. We find that it successfully solves around half of these problems on its very first attempt, and that it solves 60\% of the remaining problems using only natural language changes to the problem description. We argue that this type of prompt engineering, which we believe will become a standard interaction between human and Copilot when it initially fails, is a potentially useful learning activity that promotes computational thinking skills, and is likely to change the nature of code writing skill development.

cs.HC↗