arXiv subjects

Kaustubh Deshpande

Publications and source records attributed to Kaustubh Deshpande.

10 recordsLinked to original sources

arXiv2026

ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety

Safety evaluations for large language models (LLMs) increasingly target high-stakes National Security and Public Safety (NSPS) risks, yet multilingual safety is mostly assessed through translation-only benchmarks that preserve the underlying scenario, leaving how language and geopolitical context interact largely unexamined beyond a few language pairs. We introduce ROK-FORTRESS, a bilingual, culturally adversarial NSPS benchmark that uses the English-Korean language pair and U.S.-ROK geopolitical axis as a case study, separating the effects of language and geopolitical grounding via a transcreation matrix: adversarial intents are evaluated under controlled combinations of (i) English versus Korean language and (ii) U.S. versus Korean entities, institutions, and operational details. Each adversarial prompt is paired with a dual-use benign counterpart to quantify over-refusal, and responses are scored by calibrated LLM-as-a-judge panels using expert-crafted, prompt-specific binary rubrics. Across a dual-track set of frontier and Korean-optimized models, we find a consistent suppression effect in Korean variants and substantial model-to-model variation in how geopolitical grounding interacts with language; in a subset of models, Korean grounding further mitigates the language-driven suppression. This indicates that, at least in the English-Korean case, safety behavior is shaped by language-as-risk signals and context interactions that translation-only evaluations miss. A direct-request ablation that strips jailbreak wrappers separates a small but persistent reduction for closed-source models from a larger, wrapper-dependent effect that reverses for open-source models, suggesting part of the Korean suppression reflects prompt specialization rather than intrinsic language-based safety alignment. The transcreation matrix methodology is designed to generalize to other language-culture pairs.

cs.CL↗

arXiv2026

Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.

cs.AI↗

arXiv2025

FORTRESS: Frontier Risk Evaluation for National Security and Public Safety

The rapid advancement of large language models (LLMs) introduces dual-use capabilities that could both threaten and bolster national security and public safety (NSPS). Models implement safeguards to protect against potential misuse relevant to NSPS and allow for benign users to receive helpful information. However, current benchmarks often fail to test safeguard robustness to potential NSPS risks in an objective, robust way. We introduce FORTRESS: 500 expert-crafted adversarial prompts with instance-based rubrics of 4-7 binary questions for automated evaluation across 3 domains (unclassified information only): Chemical, Biological, Radiological, Nuclear and Explosive (CBRNE), Political Violence & Terrorism, and Criminal & Financial Illicit Activities, with 10 total subcategories across these domains. Each prompt-rubric pair has a corresponding benign version to test for model over-refusals. This evaluation of frontier LLMs' safeguard robustness reveals varying trade-offs between potential risks and model usefulness: Claude-3.5-Sonnet demonstrates a low average risk score (ARS) (14.09 out of 100) but the highest over-refusal score (ORS) (21.8 out of 100), while Gemini 2.5 Pro shows low over-refusal (1.4) but a high average potential risk (66.29). Deepseek-R1 has the highest ARS at 78.05, but the lowest ORS at only 0.06. Models such as o1 display a more even trade-off between potential risks and over-refusals (with an ARS of 21.69 and ORS of 5.2). To provide policymakers and researchers with a clear understanding of models' potential risks, we publicly release FORTRESS at https://huggingface.co/datasets/ScaleAI/fortress_public. We also maintain a private set for evaluation.

cs.CY↗

arXiv2025

MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy.

cs.CL↗

arXiv2021

TwInflation

The general structure of Hybrid Inflation remains a very well-motivated mechanism for lower-scale cosmic inflation in the face of improving constraints on the tensor-to-scalar ratio. However, as originally modeled, the "waterfall" field in this mechanism gives rise to a hierarchy problem ($η-$problem) for the inflaton after demanding standard effective field theory (EFT) control. We modify the hybrid mechanism and incorporate a discrete "twin" symmetry, thereby yielding a viable, natural and EFT-controlled model of non-supersymmetric low-scale inflation, "Twinflation". Analogously to Twin Higgs models, the discrete exchange-symmetry with a "twin" sector reduces quadratic sensitivity in the inflationary potential to ultra-violet physics, at the root of the hierarchy problem. The observed phase of inflation takes place on a hilltop-like potential but without fine-tuning of the initial inflaton position in field-space. We also show that all parameters of the model can take natural values, below any associated EFT-cutoff mass scales and field values, thus ensuring straightforward theoretical control. We discuss the basic phenomenological considerations and constraints, as well as possible future directions.

hep-ph↗

arXiv2021

Supersymmetric Inflation from the Fifth Dimension

We develop a supersymmetric bi-axion model of high-scale inflation coupled to supergravity, in which the axionic structure originates from, and is protected by, gauge symmetry in an extra dimension. While local supersymmetry (SUSY) is necessarily Higgsed at high scales during inflation we show that it can naturally survive down to the $\sim$ TeV scale in the current era in order to resolve the electroweak hierarchy problem. We show how a suitable inflationary effective potential for the axions can be generated at tree-level by charged fields under the higher-dimensional gauge symmetry. The inflationary trajectory lies along the lightest direction in the bi-axion field space, with periodic effective potential and an effective super-Planckian field range emerging from fundamentally sub-Planckian dynamics. The heavier direction in the field space is shown to also play an important role, as the dominant source of super-Higgsing during inflation. This model presents an interesting interplay of tuning considerations relating the electroweak hierarchy, cosmological constant and inflationary superpotential, where maximal naturalness favors SUSY breaking near the electroweak scale after inflation. The scalar superpartner of the axionic inflaton, the "sinflaton", can naturally have $\sim$ Hubble mass during inflation and sufficiently strong coupling to the inflaton to mediate primordial non-Gaussianities of observable strength in future 21-cm surveys. Non-minimal charged fields under the higher-dimensional gauge symmetry can contribute to periodic modulations in the CMB, within the sensitivity of ongoing measurements.

hep-ph↗

arXiv2018

Closing the light gluino gap with electron-proton colliders

The future electron-proton collider proposals, LHeC and FCC-he, can deliver $\mathcal{O}$(TeV) center-of-mass energy collisions, higher than most of the proposed lepton accelerators, with $\mathcal{O}$(ab$^{-1}$) luminosity, while maintaining a much cleaner experimental environment as compared to the hadron machines. This unique capability of $e^- p$ colliders can be harnessed in probing BSM scenarios giving final states that look like hadronic noise at $pp$ machines. In the present study, we explore the prospects of detecting such a prompt signal having multiple soft jets at the LHeC. Such a signal can come from the decay of gluino in RPV or Stealth SUSY, where there exists a gap in the current experimental search with $m_{\tilde{g}} \approx 50 - 70$ GeV. We perform a simple analysis to demonstrate that, with simple signal selection cuts, we can close this gap at the LHeC at 95 % confidence level, even in the presence of a reasonable systematic error. More sophisticated signal selection strategies and detailed knowledge of the detector can be used to improve the prospects of signal detection.

hep-ph↗

arXiv2018

Probing BSM physics with electron-proton colliders

In this talk I will illustrate with two examples (Higgsino dark matter and Exotic Higgs decays) how electron-proton colliders present unique opportunities to probe BSM scenarios where proton-proton colliders fall short due to the experimental difficulties in reconstructing the signal due to the large hadronic backgrounds. The leit-motiv of these examples are long-lived particles (LLPs), which have received recently a lot of attention from both the experimental and theoretical communities. We find that the proposed $e^-p$ colliders can be competitive against their more energetic $pp$ incarnations for lifetimes between a millimeter and a micron, depending on the physics scenario under consideration.

hep-ph↗

arXiv2017

New Physics Opportunities for Long-Lived Particles at Electron-Proton Colliders

Future electron-proton collider proposals like the LHeC or the FCC-eh can supply 1/ab of collisions with a center-of-mass energy in the TeV range, while maintaining a clean experimental environment more commonly associated with lepton colliders. We point out that this makes electron-proton colliders ideally suited to probe BSM signatures with final states that look like "hadronic noise" in the high-energy, pile-up-rich environment of hadron colliders. We focus on the generic vector boson fusion production mechanism, which is available for all BSM particles with electroweak charges at mass scales far above the reach of most lepton colliders. This is in contrast to previous BSM studies at these machines, which focused on BSM processes with large production rates from the asymmetric initial state. We propose to exploit the unique experimental environment in the search for long-lived particle signals arising from Higgsinos or exotic Higgs decays. At electron-proton colliders, the soft decay products of long-lived Higgsinos can be explicitly reconstructed ("displaced single pion"), and very short lifetimes can be probed. We find that electron-proton colliders can explore significant regions of BSM parameter space inaccessible to other collider searches, with important implications for the design of such machines.

hep-ph↗

arXiv2015

On geodesic deviation in Schwarzschild spacetime

For metrology, geodesy and gravimetry in space, satellite based instruments and measurement techniques are used and the orbits of the satellites as well as possible deviations between nearby ones are of central interest. The measurement of this deviation itself gives insight into the underlying structure of the spacetime geometry, which is curved and therefore described by the theory of general relativity (GR). In the context of GR, the deviation of nearby geodesics can be described by the Jacobi equation that is a result of linearizing the geodesic equation around a known reference geodesic with respect to the deviation vector and the relative velocity. We review the derivation of this Jacobi equation and restrict ourselves to the simple case of the spacetime outside a spherically symmetric mass distribution and circular reference geodesics to find solutions by projecting the Jacobi equation on a parallel propagated tetrad as done by Fuchs. Using his results, we construct solutions of the Jacobi equation for different physical initial scenarios inspired by satellite gravimetry missions and give a set of parameter together with their precise impact on satellite orbit deviation. We further consider the Newtonian analog and construct the full solution, that exhibits a similar structure, within this theory.

gr-qc↗