arXiv subjects

Jay Roberts

Publications and source records attributed to Jay Roberts.

9 recordsLinked to original sources

arXiv2026

Laguna M.1/XS.2 Technical Report

We present Laguna M.1 and Laguna XS.2, two Mixture-of-Experts foundation models built for long-horizon, agentic coding: M.1 has $225.8$B total parameters ($23.4$B activated per token) and XS.2 has $33.4$B total ($3$B activated). Both models were trained from scratch end-to-end inside the same internal system that we refer to as our Model Factory: a tightly-integrated stack of versioned data, training, evaluation, and inference components that turn model development into an industrial process. We describe the principles and design choices of the Model Factory and also detail the end-to-end training process of our models, throughout pre-training data and architecture, post-training stages, evaluation, and quantization. On agentic software engineering and terminal benchmarks (SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro, and Terminal-Bench 2.0) M.1 and XS.2 are competitive with state-of-the-art open models in their respective weight classes. Laguna XS.2 weights are released under Apache~2.0 at https://huggingface.co/collections/poolside/laguna-xs2.

cs.AI↗

arXiv2025

Learning Obfuscations Of LLM Embedding Sequences: Stained Glass Transform

The high cost of ownership of AI compute infrastructure and challenges of robust serving of large language models (LLMs) has led to a surge in managed Model-as-a-service deployments. Even when enterprises choose on-premises deployments, the compute infrastructure is typically shared across many teams in order to maximize the return on investment. In both scenarios the deployed models operate only on plaintext data, and so enterprise data owners must allow their data to appear in plaintext on a shared or multi-tenant compute infrastructure. This results in data owners with private or sensitive data being hesitant or restricted in what data they use with these types of deployments. In this work we introduce the Stained Glass Transform, a learned, stochastic, and sequence dependent transformation of the word embeddings of an LLM which information theoretically provides privacy to the input of the LLM while preserving the utility of model. We theoretically connect a particular class of Stained Glass Transforms to the theory of mutual information of Gaussian Mixture Models. We then calculate a-postiori privacy estimates, based on mutual information, and verify the privacy and utility of instances of transformed embeddings through token level metrics of privacy and standard LLM performance benchmarks.

cs.LG↗

arXiv2025

BeamClean: Language Aware Embedding Reconstruction

In this work, we consider an inversion attack on the obfuscated input embeddings sent to a language model on a server, where the adversary has no access to the language model or the obfuscation mechanism and sees only the obfuscated embeddings along with the model's embedding table. We propose BeamClean, an inversion attack that jointly estimates the noise parameters and decodes token sequences by integrating a language-model prior. Against Laplacian and Gaussian obfuscation mechanisms, BeamClean always surpasses naive distance-based attacks. This work highlights the necessity for and robustness of more advanced learned, input-dependent methods.

cs.CR↗

arXiv2025

THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering

We propose THELMA (Task Based Holistic Evaluation of Large Language Model Applications), a reference free framework for RAG (Retrieval Augmented generation) based question answering (QA) applications. THELMA consist of six interdependent metrics specifically designed for holistic, fine grained evaluation of RAG QA applications. THELMA framework helps developers and application owners evaluate, monitor and improve end to end RAG QA pipelines without requiring labelled sources or reference responses.We also present our findings on the interplay of the proposed THELMA metrics, which can be interpreted to identify the specific RAG component needing improvement in QA applications.

cs.CL↗

arXiv2020

Understanding and Increasing Efficiency of Frank-Wolfe Adversarial Training

Deep neural networks are easily fooled by small perturbations known as adversarial attacks. Adversarial Training (AT) is a technique that approximately solves a robust optimization problem to minimize the worst-case loss and is widely regarded as the most effective defense. Due to the high computation time for generating strong adversarial examples in the AT process, single-step approaches have been proposed to reduce training time. However, these methods suffer from catastrophic overfitting where adversarial accuracy drops during training, and although improvements have been proposed, they increase training time and robustness is far from that of multi-step AT. We develop a theoretical framework for adversarial training with FW optimization (FW-AT) that reveals a geometric connection between the loss landscape and the $\ell_2$ distortion of $\ell_\infty$ FW attacks. We analytically show that high distortion of FW attacks is equivalent to small gradient variation along the attack path. It is then experimentally demonstrated on various deep neural network architectures that $\ell_\infty$ attacks against robust models achieve near maximal distortion, while standard networks have lower distortion. It is experimentally shown that catastrophic overfitting is strongly correlated with low distortion of FW attacks. This mathematical transparency differentiates FW from Projected Gradient Descent (PGD) optimization. To demonstrate the utility of our theoretical framework we develop FW-AT-Adapt, a novel adversarial training algorithm which uses a simple distortion measure to adapt the number of attack steps during training to increase efficiency without compromising robustness. FW-AT-Adapt provides training time on par with single-step fast AT methods and closes the gap between fast AT methods and multi-step PGD-AT with minimal loss in adversarial accuracy in white-box and black-box settings.

cs.LG↗

arXiv2020

Ultrasound Diagnosis of COVID-19: Robustness and Explainability

Diagnosis of COVID-19 at point of care is vital to the containment of the global pandemic. Point of care ultrasound (POCUS) provides rapid imagery of lungs to detect COVID-19 in patients in a repeatable and cost effective way. Previous work has used public datasets of POCUS videos to train an AI model for diagnosis that obtains high sensitivity. Due to the high stakes application we propose the use of robust and explainable techniques. We demonstrate experimentally that robust models have more stable predictions and offer improved interpretability. A framework of contrastive explanations based on adversarial perturbations is used to explain model predictions that aligns with human visual perception.

eess.IV↗

arXiv2020

Second Order Optimization for Adversarial Robustness and Interpretability

Deep neural networks are easily fooled by small perturbations known as adversarial attacks. Adversarial Training (AT) is a technique aimed at learning features robust to such attacks and is widely regarded as a very effective defense. However, the computational cost of such training can be prohibitive as the network size and input dimensions grow. Inspired by the relationship between robustness and curvature, we propose a novel regularizer which incorporates first and second order information via a quadratic approximation to the adversarial loss. The worst case quadratic loss is approximated via an iterative scheme. It is shown that using only a single iteration in our regularizer achieves stronger robustness than prior gradient and curvature regularization schemes, avoids gradient obfuscation, and, with additional iterations, achieves strong robustness with significantly lower training time than AT. Further, it retains the interesting facet of AT that networks learn features which are well-aligned with human perception. We demonstrate experimentally that our method produces higher quality human-interpretable features than other geometric regularization techniques. These robust features are then used to provide human-friendly explanations to model predictions.

cs.LG↗

arXiv2018

Affine motion of 2d incompressible fluids surrounded by vacuum and flows in ${\rm SL}(2,{\mathbb R})$

The affine motion of two-dimensional (2d) incompressible fluids surrounded by vacuum can be reduced to a completely integrable and globally solvable Hamiltonian system of ordinary differential equations for the deformation gradient in ${\rm SL}(2,{\mathbb R})$. In the case of perfect fluids, the motion is given by geodesic flow in ${\rm SL}(2,{\mathbb R})$ with the Euclidean metric, while for magnetically conducting fluids (MHD), the motion is governed by a harmonic oscillator in ${\rm SL}(2,{\mathbb R})$. A complete classification of the dynamics is given including rigid motions, rotating eddies with stable and unstable manifolds, and solutions with vanishing pressure. For perfect fluids, the displacement generically becomes unbounded, as $t\to\pm\infty$. For MHD, solutions are bounded and generically quasi-periodic and recurrent.

math.AP↗

arXiv2018

Deep Learning Seismic Substructure Detection using the Frozen Gaussian Approximation

We propose a deep learning algorithm for seismic interface and pocket detection with neural networks trained by synthetic high-frequency displacement data efficiently generated by the frozen Gaussian approximation (FGA). In seismic imaging high-frequency data is advantageous since it can provide high resolution of substructures. However, generation of sufficient synthetic high-frequency data sets for training neural networks is computationally challenging. This bottleneck is overcome by a highly scalable computational platform built upon the FGA, which comes from the semiclassical theory and approximates the wavefields by a sum of fixed-width (frozen) Gaussian wave packets. Data is generated from a forward simulation of the elastic wave equation using the FGA. This data contains accurate traveltime information (from the ray path) but not exact amplitude information (with asymptotic errors not shrinking to zero even at extremely fine numerical resolution). Using this data we build convolutional neural network models using an open source API, GeoSeg, developed using Keras and Tensorflow. On a simple model, networks, despite only being trained on FGA data, can detect an interface with a high success rate from displacement data generated by the spectral element method. Benchmark tests are done for P-waves (acoustic) and P- and S-waves (elastic) generated using the FGA and a spectral element method. Further, results with a high accuracy are shown for more complicated geometries including a three layered model, and a 2D-pocket model where the neural networks trained by both clean and noisy data.

physics.geo-ph↗