Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.19368

OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration

Qirui Jin·Kenan Li·Liao Zhu·Lingching Tung·Yikai Zhang·Yijia Wu·Zijian Jin·Pengfei Gao·Yu Kang·Qingwei Lin·Wenke Lee
View original source ↗Read source document ↗Export metadata ↓

Abstract

Language-model (LM) agents are increasingly evaluated on repository-level software engineering (SWE) benchmarks derived from GitHub issues and pull requests. Although these benchmarks cover a broad range of development tasks, they provide limited insight into a common yet technically distinct challenge: migrating an existing implementation from one operating system (OS) to another while preserving behavior on both. We introduce OS-Bench, an executable benchmark for cross-OS code migration between Linux and Windows. Motivated by case studies showing that cross-platform features are frequently implemented on one OS before support is added for another, OS-Bench identifies migration opportunities by comparing execution coverage collected from the same repository on both systems. Platform-specific regions are then verified, organized into five migration scenarios, and transformed into repository-level tasks with target-platform tests and source-platform regression tests. The resulting benchmark contains 140 validated tasks from 111 repositories across C, C++, C#, Go, and Rust, comprising 91 Windows-to-Linux and 49 Linux-to-Windows migrations. We evaluate six models with SWE-agent, Claude Code, OpenHands, and Win-agent. The best configurations achieve only 26.4% success on Linux-target tasks and 20.4% on Windows-target tasks while preserving correctness across both systems. These results indicate that cross-OS migration is a challenging and underexplored capability, and OS-Bench is a testbed for improving platform-aware SWE agents.

Explore related subjects

cs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Qirui Jin, Kenan Li, Liao Zhu, Lingching Tung, Yikai Zhang, Yijia Wu, Zijian Jin, Pengfei Gao, Yu Kang, Qingwei Lin, Wenke Lee. 2026-10-14. OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration. https://arxiv.org/abs/2610.19368

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Securing High-Concurrency Ticket Sales: A Framework Based on Microservice

The railway ticketing system is one of the most important public service infrastructure. In peak periods such as holidays, it is often faced with the challenge of high concurrency scenarios because of a large number of users accessing at the same time. The traditional aggregation architecture can not meet the peak user requirements because of its insufficient fault tolerance and low ability. Therefore, the system needs to use microservice architecture for development, and add multiple security methods to ensure that the system can have good stability and data consistency under high concurrency scenarios, and can respond quickly to user requests. This paper introduces the use of B/S architecture and Spring Cloud to design and develop a railway ticket purchase system that can maintain stability and reliability under high concurrency scenarios, and formulate multiple security design methods for the system. This system integrates a range of functions, such as real-time train inquiries, dynamic seat updates, online seat selection, and ticket purchasing, effectively addressing common problems associated with offline ticket purchasing, such as long queues and delayed information. It enables a complete online process from inquiry and booking to payment and refunds. Furthermore, the "add passenger" function allows users to purchase tickets for others, extending the convenience of online ticketing to people with limited internet access. The system design prioritizes security and stability, while also focusing on high performance, and achieves these goals through a carefully designed architecture and the integration of multiple middleware components. After the completion of the system development, the core interface of the system is tested, and then the results are analyzed. The test data proves that the system has good ability and stability under high concurrency.

cs.SE↗
arXiv2026

Terminal Agents Suffice for Enterprise Automation

There has been growing interest in building agents that can interact with digital platforms to execute meaningful enterprise tasks autonomously. Among the approaches explored are tool-augmented agents built on abstractions such as Model Context Protocol (MCP) and web agents that operate through graphical interfaces. Yet, it remains unclear whether such complex agentic systems are necessary given their cost and operational overhead. We argue that a coding agent equipped only with a terminal and a filesystem can solve many enterprise tasks more effectively by interacting directly with platform APIs. We evaluate this hypothesis across diverse real-world systems and show that these low-level terminal agents match or outperform more complex agent architectures at a fraction of the cost. Our findings suggest that simple, flexible programmatic interfaces combined with strong foundation models should be the backbone of enterprise automation.

cs.SE↗
arXiv2026

T2J-BENCH: Benchmarking Coding Agents on Execution-Grounded PyTorch-to-JAX Repository Migration

Migrating a machine-learning codebase across frameworks--for example from PyTorch to JAX--is a routine but costly engineering task that is fundamentally cross-paradigm: eager, stateful, object-oriented modules must be rewritten as pure, jittable functions over an explicit parameter pytree. We ask whether coding agents can perform such whole-repository migrations correctly, and argue that existing code benchmarks, which grade unit-test pass rates or surface-level similarity to a reference translation, cannot answer this: they neither require the migrated system to \emph{behave} the same nor prevent a JAX reimplementation from faking equivalence by secretly importing PyTorch itself. We introduce T2J-BENCH, a benchmark of 63 whole-repository PyTorch-to-JAX migration tasks spanning 18 model families across both training and serving, where torch-free checkpoint conversion is itself part of the deliverable. T2J-BENCH grades execution-grounded functional equivalence: each candidate is run inside a separate, torch-free, offline verifier and scored against references frozen from the PyTorch original, under a hard independence gate and a continuous, threshold-free reward. Across twenty off-the-shelf agent--model configurations, the strongest recovers only 0.676 of the reward against a 0.980 oracle, with substantial numeric and behavioral divergence despite high spec pass rates. Finally, we contribute TWINHARNESS, a reference-grounded agent harness that turns the agent's executable PyTorch source into legitimate self-verification feedback; applied to an opus-5 agent it raises the reward from 0.649 to 0.723--the highest of any configuration--using only task-visible resources.

cs.SE↗