Ivan Vasilev | Searcharxiv

arXiv2026

DeepTest Tool Competition 2026: Benchmarking an LLM-Based Automotive Assistant

This report summarizes the results of the first edition of the Large Language Model (LLM) Testing competition, held as part of the DeepTest workshop at ICSE 2026. Four tools competed in benchmarking an LLM-based car manual information retrieval application, with the objective of identifying user inputs for which the system fails to appropriately mention warnings contained in the manual. The testing solutions were evaluated based on their effectiveness in exposing failures and the diversity of the discovered failure-revealing tests. We report on the experimental methodology, the competitors, and the results.

cs.AI↗

arXiv2026

STELLAR: A Search-Based Testing Framework for Large Language Model Applications

Large Language Model (LLM)-based applications are increasingly deployed across various domains, including customer service, education, and mobility. However, these systems are prone to inaccurate, fictitious, or harmful responses, and their vast, high-dimensional input space makes systematic testing particularly challenging. To address this, we present STELLAR, an automated search-based testing framework for LLM-based applications that systematically uncovers text inputs leading to inappropriate system responses. Our framework models test generation as an optimization problem and discretizes the input space into stylistic, content-related, and perturbation features. Unlike prior work that focuses on prompt optimization or coverage heuristics, our work employs evolutionary optimization to dynamically explore feature combinations that are more likely to expose failures. We evaluate STELLAR on three LLM-based conversational question-answering systems. The first focuses on safety, benchmarking both public and proprietary LLMs against malicious or unsafe prompts. The second and third target navigation, using an open-source and an industrial retrieval-augmented system for in-vehicle venue recommendations. Overall, STELLAR exposes up to 4.3 times (average 2.5 times) more failures than the existing baseline approaches.

cs.SE↗

arXiv2025

Unordered resolutions and homological stability for linear groups

In this paper, we develop a modified proof strategy for homological stability of linear groups, with the general linear groups serving as a primary example. Our arguments are more direct than those in the classical works of Quillen and Suslin--Nesterenko, although they apply only with localized coefficients. The localization at (n-1)! that arises in our approach appears to be closely related to several conjectures of Mirzaii as well as to Suslin's injectivity conjecture.

math.KT↗