arXiv subjects

Noshin Ulfat

Publications and source records attributed to Noshin Ulfat.

3 recordsLinked to original sources

arXiv2026

PROGRESS: Property-Guided Regression Search for Semantic Falsification

Search-based regression-test generation effectively explores complex program structures, yielding high structural coverage, but its oracles are derived from the system under test: faults already present are recorded as expected behavior rather than exposed. Property-based testing offers independent semantic oracles, but depends on high-quality properties and gives little guidance for reaching deep states or satisfying selective preconditions. We present PROGRESS (PROperty-Guided REgression Search for Semantic Falsification), integrating intent-driven properties into coverage-guided, search-based evolutionary test generation to reach deep program states and detect violations of intended behavior. PROGRESS (1) extracts intent-bearing code context and uses a language-model pipeline to generate executable jqwik properties while limiting implementation leakage; (2) extends EvoSuite's DynaMOSA with a search objective and property-aware fitness function per property, rewarding progress through preconditions and prioritizing falsifying executions; and (3) binds property parameters and uses jqwik-provided generators to connect quantified inputs to evolving test sequences, steering generation toward coverage and bug-detection goals. We evaluate PROGRESS on 25 large-scale Java systems against regression-test generation, standalone property-based testing, and context ablations. PROGRESS detects 328/562 current-version bugs (58%) versus none for regression-test generation, and satisfies all preconditions for 70/150 hard-to-reach properties versus 18 for standalone jqwik. Ablations show documentation and caller/callee context are key to generating valid executable properties. PROGRESS preserves structural exploration while exposing faults missed by regression-derived assertions; we release a comprehensive artifact package.

cs.SE↗

arXiv2026

Measuring LLM Trust Allocation Across Conflicting Software Artifacts

LLM-based software engineering assistants often reason over multiple artifacts, including code, documentation, signatures, and tests, even when those artifacts are incomplete or mutually inconsistent. Existing evaluations primarily measure final outputs, leaving unclear whether a model recognized unreliable evidence, identified the faulty source, or prioritized the appropriate artifact. We introduce TRACE, a controlled method for evaluating how LLMs assess and prioritize conflicting software artifacts. TRACE constructs paired clean and perturbed versions of real-world Java method bundles by injecting known faults into the documentation, implementation, or both while holding the remaining artifacts fixed. Models then assess artifact quality, detect and localize inconsistencies, and rank the available sources by reliability. Using 22,339 valid responses from seven LLMs on 456 method bundles, we find that quality penalties are generally localized to the perturbed artifact and increase with fault severity. However, models exhibit a consistent source-origin asymmetry: they detect documentation faults at 67-94% and explicit documentation-implementation contradictions at 50-91%, but detection falls by 21-43 percentage points when only the implementation changes while documentation remains intact. Models also struggle to deprioritize faulty implementations, and confidence provides little separation between correct and incorrect judgments for six of seven models. These results show that current LLMs are not symmetric integrators of software evidence: they audit natural-language specifications more reliably than subtle implementation behavior. TRACE provides a controlled method for exposing this failure mode before LLMs are used in correctness-critical software engineering workflows.

cs.SE↗

arXiv2024

Using Large Language Models to Generate JUnit Tests: An Empirical Study

A code generation model generates code by taking a prompt from a code comment, existing code, or a combination of both. Although code generation models (e.g., GitHub Copilot) are increasingly being adopted in practice, it is unclear whether they can successfully be used for unit test generation without fine-tuning for a strongly typed language like Java. To fill this gap, we investigated how well three models (Codex, GPT-3.5-Turbo, and StarCoder) can generate unit tests. We used two benchmarks (HumanEval and Evosuite SF110) to investigate the effect of context generation on the unit test generation process. We evaluated the models based on compilation rates, test correctness, test coverage, and test smells. We found that the Codex model achieved above 80% coverage for the HumanEval dataset, but no model had more than 2% coverage for the EvoSuite SF110 benchmark. The generated tests also suffered from test smells, such as Duplicated Asserts and Empty Tests.

cs.SE↗