arXiv subjects

Vinay Ramasesh

Publications and source records attributed to Vinay Ramasesh.

4 recordsLinked to original sources

arXiv2024

Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries

We introduce Michelangelo: a minimal, synthetic, and unleaked long-context reasoning evaluation for large language models which is also easy to automatically score. This evaluation is derived via a novel, unifying framework for evaluations over arbitrarily long contexts which measure the model's ability to do more than retrieve a single piece of information from its context. The central idea of the Latent Structure Queries framework (LSQ) is to construct tasks which require a model to ``chisel away'' the irrelevant information in the context, revealing a latent structure in the context. To verify a model's understanding of this latent structure, we query the model for details of the structure. Using LSQ, we produce three diagnostic long-context evaluations across code and natural-language domains intended to provide a stronger signal of long-context language model capabilities. We perform evaluations on several state-of-the-art models and demonstrate both that a) the proposed evaluations are high-signal and b) that there is significant room for improvement in synthesizing long-context information.

cs.CL↗

arXiv2022

Exploring Length Generalization in Large Language Models

The ability to extrapolate from short problem instances to longer ones is an important form of out-of-distribution generalization in reasoning tasks, and is crucial when learning from datasets where longer problem instances are rare. These include theorem proving, solving quantitative mathematics problems, and reading/summarizing novels. In this paper, we run careful empirical studies exploring the length generalization capabilities of transformer-based language models. We first establish that naively finetuning transformers on length generalization tasks shows significant generalization deficiencies independent of model scale. We then show that combining pretrained large language models' in-context learning abilities with scratchpad prompting (asking the model to output solution steps before producing an answer) results in a dramatic improvement in length generalization. We run careful failure analyses on each of the learning modalities and identify common sources of mistakes that highlight opportunities in equipping language models with the ability to generalize to longer problems.

cs.CL↗

arXiv2022

Solving Quantitative Reasoning Problems with Language Models

Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.

cs.CL↗

arXiv2018

Scattering into one-dimensional waveguides from a coherently-driven quantum-optical system

We develop a new computational tool and framework for characterizing the scattering of photons by energy-nonconserving Hamiltonians into unidirectional (chiral) waveguides, for example, with coherent pulsed excitation. The temporal waveguide modes are a natural basis for characterizing scattering in quantum optics, and afford a powerful technique based on a coarse discretization of time. This overcomes limitations imposed by singularities in the waveguide-system coupling. Moreover, the integrated discretized equations can be faithfully converted to a continuous-time result by taking the appropriate limit. This approach provides a complete solution to the scattered photon field in the waveguide, and can also be used to track system-waveguide entanglement during evolution. We further develop a direct connection between quantum measurement theory and evolution of the scattered field, demonstrating the correspondence between quantum trajectories and the scattered photon state. Our method is most applicable when the number of photons scattered is known to be small, i.e. for a single-photon or photon-pair source. We illustrate two examples: analytical solutions for short laser pulses scattering off a two-level system and numerically exact solutions for short laser pulses scattering off a spontaneous parametric downconversion (SPDC) or spontaneous four-wave mixing (SFWM) source. Finally, we note that our technique can easily be extended to systems with multiple ground states and generalized scattering problems with both finite photon number input and coherent state drive, potentially enhancing the understanding of, e.g., light-matter entanglement and photon phase gates.

quant-ph↗