arXiv subjects

Ryan Park

Publications and source records attributed to Ryan Park.

11 recordsLinked to original sources

arXiv2025

OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observation

Earth observation data presents a unique challenge: it is spatial like images, sequential like video or text, and highly multimodal. We present OlmoEarth: a multimodal, spatio-temporal foundation model that employs a novel self-supervised learning formulation, masking strategy, and loss all designed for the Earth observation domain. OlmoEarth achieves state-of-the-art performance compared to 12 other foundation models across a variety of research benchmarks and real-world tasks from external partners. When evaluating embeddings OlmoEarth achieves the best performance on 15 out of 24 tasks, and with full fine-tuning it is the best on 19 of 29 tasks. We deploy OlmoEarth as the backbone of an end-to-end platform for data collection, labeling, training, and inference of Earth observation models. The OlmoEarth Platform puts frontier foundation models and powerful data management tools into the hands of non-profits and NGOs working to solve the world's biggest problems. OlmoEarth source code, training data, and pre-trained weights are available at $\href{https://github.com/allenai/olmoearth_pretrain}{\text{https://github.com/allenai/olmoearth_pretrain}}$.

cs.CV↗

arXiv2024

Inferring Foresightedness in Dynamic Noncooperative Games

Dynamic game theory is an increasingly popular tool for modeling multi-agent, e.g. human-robot, interactions. Game-theoretic models presume that each agent wishes to minimize a private cost function that depends on others' actions. These games typically evolve over a fixed time horizon, specifying how far into the future each agent plans. In practical settings, however, decision-makers may vary in foresightedness, or how much they care about their current cost in relation to their past and future costs. We conjecture that quantifying and estimating each agent's foresightedness from online data will enable safer and more efficient interactions with other agents. To this end, we frame this inference problem as an inverse dynamic game. We consider a specific objective function parametrization that smoothly interpolates myopic and farsighted planning. Games of this form are readily transformed into parametric mixed complementarity problems; we exploit the directional differentiability of solutions to these problems with respect to their hidden parameters to solve for agents' foresightedness. We conduct three experiments: one with synthetically generated delivery robot motion, one with real-world data involving people walking, biking, and driving vehicles, and one using high-fidelity simulators. The results of these experiments demonstrate that explicitly inferring agents' foresightedness enables game-theoretic models to make 33% more accurate models for agents' behavior.

cs.RO↗

arXiv2024

Improving Inverse Folding for Peptide Design with Diversity-regularized Direct Preference Optimization

Inverse folding models play an important role in structure-based design by predicting amino acid sequences that fold into desired reference structures. Models like ProteinMPNN, a message-passing encoder-decoder model, are trained to reliably produce new sequences from a reference structure. However, when applied to peptides, these models are prone to generating repetitive sequences that do not fold into the reference structure. To address this, we fine-tune ProteinMPNN to produce diverse and structurally consistent peptide sequences via Direct Preference Optimization (DPO). We derive two enhancements to DPO: online diversity regularization and domain-specific priors. Additionally, we develop a new understanding on improving diversity in decoder models. When conditioned on OpenFold generated structures, our fine-tuned models achieve state-of-the-art structural similarity scores, improving base ProteinMPNN by at least 8%. Compared to standard DPO, our regularized method achieves up to 20% higher sequence diversity with no loss in structural similarity score.

cs.LG↗

arXiv2024

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms

Reinforcement Learning from Human Feedback (RLHF) has been crucial to the recent success of Large Language Models (LLMs), however, it is often a complex and brittle process. In the classical RLHF framework, a reward model is first trained to represent human preferences, which is in turn used by an online reinforcement learning (RL) algorithm to optimize the LLM. A prominent issue with such methods is reward over-optimization or reward hacking, where performance as measured by the learned proxy reward model increases, but true quality plateaus or even deteriorates. Direct Alignment Algorithms (DDAs) like Direct Preference Optimization have emerged as alternatives to the classical RLHF pipeline by circumventing the reward modeling phase. However, although DAAs do not use a separate proxy reward model, they still commonly deteriorate from over-optimization. While the so-called reward hacking phenomenon is not well-defined for DAAs, we still uncover similar trends: at higher KL budgets, DAA algorithms exhibit similar degradation patterns to their classic RLHF counterparts. In particular, we find that DAA methods deteriorate not only across a wide range of KL budgets but also often before even a single epoch of the dataset is completed. Through extensive empirical experimentation, this work formulates and formalizes the reward over-optimization or hacking problem for DAAs and explores its consequences across objectives, training regimes, and model scales.

cs.LG↗

arXiv2024

From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function

Reinforcement Learning From Human Feedback (RLHF) has been critical to the success of the latest generation of generative AI models. In response to the complex nature of the classical RLHF pipeline, direct alignment algorithms such as Direct Preference Optimization (DPO) have emerged as an alternative approach. Although DPO solves the same objective as the standard RLHF setup, there is a mismatch between the two approaches. Standard RLHF deploys reinforcement learning in a specific token-level MDP, while DPO is derived as a bandit problem in which the whole response of the model is treated as a single arm. In this work we rectify this difference. We theoretically show that we can derive DPO in the token-level MDP as a general inverse Q-learning algorithm, which satisfies the Bellman equation. Using our theoretical results, we provide three concrete empirical insights. First, we show that because of its token level interpretation, DPO is able to perform some type of credit assignment. Next, we prove that under the token level formulation, classical search-based algorithms, such as MCTS, which have recently been applied to the language generation space, are equivalent to likelihood-based search on a DPO policy. Empirically we show that a simple beam search yields meaningful improvement over the base DPO policy. Finally, we show how the choice of reference policy causes implicit rewards to decline during training. We conclude by discussing applications of our work, including information elicitation in multi-turn dialogue, reasoning, agentic applications and end-to-end training of multi-model systems.

cs.LG↗

arXiv2024

Smooth Information Gathering in Two-Player Noncooperative Games

We present a mathematical framework for modeling two-player noncooperative games in which one player is uncertain of the other player's costs but can preemptively allocate information-gathering resources to reduce this uncertainty. We refer to the players as the uncertain player (UP) and the certain player (CP), respectively. We obtain UP's decisions by solving a two-stage problem where, in Stage 1, UP allocates information-gathering resources that smoothly transform the information structure in the second stage. Then, in Stage 2, a signal (that is, a function of the Stage 1 allocation) informs UP about CP's costs, and both players execute strategies which depend upon the signal's value. This framework allows for a smooth resource allocation, in contrast to existing literature on the topic. We also identify conditions under which the gradient of UP's overall cost with respect to the information-gathering resources is well-defined. Then we provide a gradient-based algorithm to solve the two-stage game. Finally, we apply our framework to a tower-defense game which can be interpreted as a variant of a Colonel Blotto game with smooth payoff functions and uncertainty over battlefield valuations. We include an analysis of how optimal decisions shift with changes in information-gathering allocations and perturbations in the cost functions.

cs.GT↗

arXiv2024

Disentangling Length from Quality in Direct Preference Optimization

Reinforcement Learning from Human Feedback (RLHF) has been a crucial component in the recent success of Large Language Models. However, RLHF is know to exploit biases in human preferences, such as verbosity. A well-formatted and eloquent answer is often more highly rated by users, even when it is less helpful and objective. A number of approaches have been developed to control those biases in the classical RLHF literature, but the problem remains relatively under-explored for Direct Alignment Algorithms such as Direct Preference Optimization (DPO). Unlike classical RLHF, DPO does not train a separate reward model or use reinforcement learning directly, so previous approaches developed to control verbosity cannot be directly applied to this setting. Our work makes several contributions. For the first time, we study the length problem in the DPO setting, showing significant exploitation in DPO and linking it to out-of-distribution bootstrapping. We then develop a principled but simple regularization strategy that prevents length exploitation, while still maintaining improvements in model quality. We demonstrate these effects across datasets on summarization and dialogue, where we achieve up to 20\% improvement in win rates when controlling for length, despite the GPT4 judge's well-known verbosity bias.

cs.CL↗

arXiv2023

Preference Optimization for Molecular Language Models

Molecular language modeling is an effective approach to generating novel chemical structures. However, these models do not \emph{a priori} encode certain preferences a chemist may desire. We investigate the use of fine-tuning using Direct Preference Optimization to better align generated molecules with chemist preferences. Our findings suggest that this approach is simple, efficient, and highly effective.

stat.ML↗

arXiv2023

ThunderBoltz: An Open-Source DSMC-based Boltzmann Solver for Plasma Transport, Chemical Kinetics, and 0D Plasma Modeling

Plasma-neutral interactions, including reactive kinetics, are often either studied in 0D using ODE based descriptions, or in multi-dimensional fluid or particle based plasma codes. The latter case involves a complex assembly of procedures that are not always necessary to test effects of underlying physical models and mechanisms for particle-based descriptions. Here we present ThunderBoltz, a lightweight, publicly available 0D Direct Simulation Monte Carlo (DSMC) code designed to accommodate a generalized combination of species and arbitrary cross sections without the overhead of expensive field solves. It can efficiently produce high-quality electron, ion, and neutral velocity distributions in applied AC/DC $E$-field and static $B$-field scenarios. The code is built in the C++ standard library and includes a convenient Python interface that allows for input file generation (including compatibility with cross section data from the LXCat database), electron transport and reaction rate constant post-processing, input parameter constraint satisfaction, calculation scheduling, and diagnostic plotting. In this work we compare ThunderBoltz transport calculations against Bolsig+ calculations, benchmark test problems, and swarm experiment data, finding good agreement with all three in the appropriate field regimes. In addition to this, we present example use cases where the electron, ion, and background neutral particle species are self consistently evolved providing an ability to model the background kinetics, a feature that is absent in fixed background Monte Carlo and n-term Boltzmann solvers. The latter functionality allows for the possibility of particle-based chemical kinetics simulations of the plasma and neutral species and is a new alternative to ODE based approaches.

physics.plasm-ph↗

arXiv2023

EchoVest: Real-Time Sound Classification and Depth Perception Expressed through Transcutaneous Electrical Nerve Stimulation

Over 1.5 billion people worldwide live with hearing impairment. Despite various technologies that have been created for individuals with such disabilities, most of these technologies are either extremely expensive or inaccessible for everyday use in low-medium income countries. In order to combat this issue, we have developed a new assistive device, EchoVest, for blind/deaf people to intuitively become more aware of their environment. EchoVest transmits vibrations to the user's body by utilizing transcutaneous electric nerve stimulation (TENS) based on the source of the sounds. EchoVest also provides various features, including sound localization, sound classification, noise reduction, and depth perception. We aimed to outperform CNN-based machine-learning models, the most commonly used machine learning model for classification tasks, in accuracy and computational costs. To do so, we developed and employed a novel audio pipeline that adapts the Audio Spectrogram Transformer (AST) model, an attention-based model, for our sound classification purposes, and Fast Fourier Transforms for noise reduction. The application of Otsu's Method helped us find the optimal thresholds for background noise sound filtering and gave us much greater accuracy. In order to calculate direction and depth accurately, we applied Complex Time Difference of Arrival algorithms and SOTA localization. Our last improvement was to use blind source separation to make our algorithms applicable to multiple microphone inputs. The final algorithm achieved state-of-the-art results on numerous checkpoints, including a 95.7\% accuracy on the ESC-50 dataset for environmental sound classification.

cs.SD↗

arXiv2019

Interior properties of the inner Saturnian moons from space astrometry data

During the thirteen years in orbit around Saturn before its final plunge, the Cassini spacecraft provided more than ten thousand astrometric measurements. Such large amounts of accurate data enable the search for extremely faint signals in the orbital motion of the saturnian moons. Among these, the detection of the dynamical feedback of the rotation of the inner moons of Saturn on their respective orbits becomes possible. Using all the currently available astrometric data associated with Atlas, Prometheus, Pandora, Janus and Epimetheus, we first provide a detailed analysis of the Cassini Imaging Science Subsystem (ISS) data, with special emphasis on their statistical behavior and sources of bias. Then, we give updated estimates of the moons' averaged densities and try to infer more details about their interior properties by estimating the physical librations for Prometheus, Pandora, Epimetheus and Janus from anomalies in their apsidal precession. Our results are compatible with a homogeneous interior for Janus and Epimetheus, within the uncertainty of the measurements. On the other hand, we found some inconsistency for Pandora and Prometheus, which might result from a dynamical mismodeling of Saturn's gravity field. Last, we show how the synergistic introduction of libration measurements directly derived from imaging should allow the moons' moments of inertia to be better constrained.

astro-ph.EP↗