arXiv subjects

Shiwen Ni

Publications and source records attributed to Shiwen Ni.

4 recordsLinked to original sources

arXiv2026

Do Code Language Models Follow Tests? Paired Interventions on Program Behavior

Visible tests specify concrete program behavior, but an improvement in benchmark accuracy does not establish that a model follows the rule expressed by those tests. We study test utilization through matched prompting controls, paired semantic interventions, and test suites selected by fault detection. Our semantic intervention holds an underspecified description and its example inputs fixed while changing the correct outputs to express one of two valid rules. Evaluation on unseen inputs measures whether both generated programs follow their respective rules. Across five models and three runs of 120 paired instances from 20 specification families, mean switching rates range from 11.1\% to 65.8\%. Qwen3.8-27B has the highest point estimate, followed by Qwen3.6-27B at 60.6\%; their paired difference remains uncertain. Explicit descriptions elicit both rules from these two models on every instance, exposing a gap between implementation capability and adoption of test-specified rules. Correct expected outputs improve MBPP+ accuracy beyond inputs alone for all five models. On 180 tasks with fixed three-test suites, high-detection suites detect 30.3 percentage points more errors in a held-out pool dominated by reference mutants. The corresponding generation differences range from $-0.6$ to $+1.1$ points; all intervals include zero and remain compatible with some benefit. Paired interventions make test-specified rule changes measurable alongside implementation capability and benchmark correctness.

cs.SE↗

arXiv2026

Correct Tests Are Not Enough: Measuring and Training Oracle Conversion in Specification-Based Test Generation

Generating tests from a natural-language specification requires both an input that exposes faulty behavior and a correct expected output. These requirements need not improve together: a model can increase test correctness by choosing easier inputs, or discover useful inputs whose expected outputs it cannot predict. We study this interaction through executable reward decomposition and suite-level oracle-conversion measurement. Our generator jointly emits five input--output tests in one response. During training, audited reference programs provide correctness feedback, while a fixed bank of faulty programs provides two utility signals: potential input kill and effective kill after checking the generated output. An additive GRPO objective preserves both signals without requiring execution at inference time. On an audited TC-Bench split with 506 training and 142 evaluation tasks, three independently trained Qwen3.5-9B runs at step 75 increase full-test correctness from 28.59\% to 42.54\%, input kill from 24.06\% to 25.27\%, and effective full kill from 12.23\% to 14.15\%. Matched 50-step ablations reveal a trade-off: removing kill rewards yields higher correctness and slightly higher full kill, but lowers input kill to 21.60\%. A fixed-input source--oracle crossover on 64 training-pool tasks attributes the principal NoKill-to-FullKill difference to harder input selection rather than worse output prediction on identical inputs. These results identify oracle conversion as a measurable bottleneck and show the benefits and limits of preserving input-utility feedback in joint test generation.

cs.SE↗

arXiv2026

How Output Format Confounds Data Quality and Capability in Instruction Tuning

Instruction-tuning data are judged by quality metrics, and tuned models are judged by benchmarks, but both judgments pass through an output interface: the surface format in which an answer is written. Using gradient signatures across 12 tasks, four semantically equivalent interfaces, three model families, and controlled corruptions, we show that this interface confounds both measurements. Spectral statistics such as effective rank are provably invariant to interface rotation and empirically blind to semantic corruption, while the direction of the update carries the quality signal. The interface-varying residual is not noise: it identifies each unit's own target task perfectly across all three families. Capability itself is stored relative to the training interface: a skill that raises accuracy by more than 40 points under the training format can be nearly invisible under every other, and correcting a single generation budget flips the measured effect of fine-tuning on GSM8K from a gain into a large loss. Pre-registered interventions delimit where this geometry stops short of control. Data quality and model capability are interface-conditioned quantities, and current practice often reports the interface instead of the content.

cs.CL↗

arXiv2026

Joint Training Is Not Enough: Conditioned Cross-Granularity Training for Multimodal Document Understanding

The Mutual Reinforcement Effect (MRE) asks whether a fine, span-level and a coarse, document-level task help each other when one model handles both. We test it in multimodal document understanding on three corpora, two of receipts and one of scanned business forms, comparing single-task, joint and conditioned training, which puts one granularity's gold output in the other's prompt during training only. We build Doc-MRE, an annotation layer pairing gold field extraction (point) with four document-level facets (line), from a three-judge LLM committee under a pre-registration, validated by blind re-annotation. One predicate, fixed in advance: at a shared recipe, a regime reinforces if it beats the matched single-task model on both granularities. Mixed joint training, the arrangement prior MRE work assumes, reinforces on no corpus at the main scale: it is below both single-task models on CORD and trades one granularity for the other on the two others, as single-task tuning does. Conditioned training reinforces on two of the three, CORD (+0.5 point, +4.8 line) and the forms corpus (+7.2 point, +11.0 line), resolvably on the coarse side and directionally on the fine one, and trades on WildReceipt; at that recipe no alternative measurably beats it on either side anywhere. Two byte-identical-prompt controls separate content from format: shuffled conditioning destroys the coarse-side skill but costs the fine side far less, and a neutral-content control reproduces the whole fine-side gain on WildReceipt, which is therefore prompt structure but buys nothing resolvable on the other two. On the forms corpus conditioning buys collapse avoidance: mixed training and the neutral control both assign the majority semantic label to all 50 test documents; only conditioning recovers the gold distribution. Probes find the information decodable under every regime with no resolvable increase under conditioning.

cs.CL↗