arXiv subjects

David Yao

Publications and source records attributed to David Yao.

8 recordsLinked to original sources

arXiv2026

OPD+: Rethinking the Advantage Design for On-Policy Distillation

On-policy distillation (OPD) is a widely used technique to transfer capabilities from capable teacher language models to the base student models, and can be formulated in a reinforcement learning style objective using student generated rollouts. Yet, despite the divergence reward being dependent on student model likelihood, existing works usually adopt a stop gradient design primarily for stability, which makes the resulting advantage estimation questionable. In this work, we provide a generic optimization framework based on f-divergence between the student and teacher, and mathematically revisit whether such design space is valid. We prove that general stop-gradient operation would lead to biased estimates of the reward objective and corresponding gradient for general divergence functions. We propose OPD+, the corrected version of OPD that demonstrates improved performance over the baseline KL approach and also supports the choice of various f-divergence. We validate our findings on mathematical reasoning and tool-use benchmarks.

cs.LG↗

arXiv2025

SOCRATES: Simulation Optimization with Correlated Replicas and Adaptive Trajectory Evaluations

The field of simulation optimization (SO) encompasses various methods developed to optimize complex, expensive-to-sample stochastic systems. Established methods include, but are not limited to, ranking-and-selection for finite alternatives and surrogate-based methods for continuous domains, with broad applications in engineering and operations management. The recent advent of large language models (LLMs) offers a new paradigm for exploiting system structure and automating the strategic selection and composition of these established SO methods into a tailored optimization procedure. This work introduces SOCRATES (Simulation Optimization with Correlated Replicas and Adaptive Trajectory Evaluations), a novel two-stage procedure that leverages LLMs to automate the design of tailored SO algorithms. The first stage constructs an ensemble of digital replicas of the real system. An LLM is employed to implement causal discovery from a textual description of the system, generating a structural `skeleton' that guides the sample-efficient learning of the replicas. In the second stage, this replica ensemble is used as an inexpensive testbed to evaluate a set of baseline SO algorithms. An LLM then acts as a meta-optimizer, analyzing the performance trajectories of these algorithms to iteratively revise and compose a final, hybrid optimization schedule. This schedule is designed to be adaptive, with the ability to be updated during the final execution on the real system when the optimization performance deviates from expectations. By integrating LLM-driven reasoning with LLM-assisted trajectory-aware meta-optimization, SOCRATES creates an effective and sample-efficient solution for complex SO optimization problems.

stat.ML↗

arXiv2025

DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning

We propose DiFFPO, Diffusion Fast and Furious Policy Optimization, a unified framework for training masked diffusion large language models (dLLMs) to reason not only better (furious), but also faster via reinforcement learning (RL). We first unify the existing baseline approach such as d1 by proposing to train surrogate policies via off-policy RL, whose likelihood is much more tractable as an approximation to the true dLLM policy. This naturally motivates a more accurate and informative two-stage likelihood approximation combined with importance sampling correction, which leads to generalized RL algorithms with better sample efficiency and superior task performance. Second, we propose a new direction of joint training efficient samplers/controllers of dLLMs policy. Via RL, we incentivize dLLMs' natural multi-token prediction capabilities by letting the model learn to adaptively allocate an inference threshold for each prompt. By jointly training the sampler, we yield better accuracies with lower number of function evaluations (NFEs) compared to training the model only, obtaining the best performance in improving the Pareto frontier of the inference-time compute of dLLMs. We showcase the effectiveness of our pipeline by training open source large diffusion language models over benchmark math and planning tasks.

cs.LG↗

arXiv2025

Optimal Decisions for Liquid Staking: Allocation and Exit Timing

In this paper, we study an investor's optimal entry and exit decisions in a liquid staking protocol (LSP) and an automated market maker (AMM), primarily from the standpoint of the investor. Our analysis focuses on two key investor actions: the initial allocation decision at time $t=0$, and the optimal timing of exit. First, we derive an optimal allocation strategy that enables the investor to distribute risk across the LSP, AMM, and direct holding. Our results also offer insights for LSP and AMM designers, identifying the necessary and sufficient conditions under which the investor is incentivized to stake through an LSP, and further, to provide liquidity in addition to staking. These conditions include a lower bound on the transaction fee, for which we propose a fee mechanism that attains the bound. Second, given a fixed protocol design, we model the optimal exit timing of an individual investor using Laplace transforms and free-boundary techniques. We analyze scenarios with and without transaction fees. In the absence of fees, we decompose the investor's payoff into impermanent loss and opportunity cost, and provide theoretical results characterizing the investor's payoff and the optimal exit threshold. With transaction fees, we conduct numerical analyses to examine how fee accumulation influences exit strategies. Our results reveal that in both settings, a stop-loss strategy often maximizes the investor's expected payoff, driven by opportunity gains and the accumulation of fees where fees are present. Our analyses rely on various tools from stochastic processes and control theory, as well as convex optimization and analysis. We further support our theoretical insights with numerical experiments and explore additional properties of the investor's value function and optimal behavior.

q-fin.MF↗

arXiv2025

Analysis of the Order Flow Auction under Proposer-Builder Separation on Blockchain

We study the impact of the order flow auction (OFA) in the context of the proposer-builder separation (PBS) mechanism in blockchains through a game-theoretic perspective. The OFA is designed to improve user welfare by redistributing maximal extractable value (MEV) to the users, in which two sequential auctions take place: the order flow auction and the block-building auction. We formulate the OFA as a multiplayer game, and establish the existence of a Nash equilibrium, and in the two-player case derive a closed-form solution (and prove its uniqueness) via a quartic equation. Our result shows that the builder with a competitive advantage pays a lower cost, leading to a higher revenue, and adding to centralization in the builder space. In contrast, the proposer's shares evolve as a martingale process, which implies decentralization in the proposer/validator space. Our analyses rely on various tools from stochastic processes, convex optimization, and polynomial equations. We also conduct numerical studies to corroborate our findings, and to bring out other features of the OFA under the PBS mechanism.

econ.TH↗

arXiv2024

MallowsPO: Fine-Tune Your LLM with Preference Dispersions

Direct Preference Optimization (DPO) has recently emerged as a popular approach to improve reinforcement learning with human feedback (RLHF), leading to better techniques to fine-tune large language models (LLM). A weakness of DPO, however, lies in its lack of capability to characterize the diversity of human preferences. Inspired by Mallows' theory of preference ranking, we develop in this paper a new approach, the MallowsPO. A distinct feature of this approach is a dispersion index, which reflects the dispersion of human preference to prompts. We show that existing DPO models can be reduced to special cases of this dispersion index, thus unified with MallowsPO. More importantly, we demonstrate (empirically) how to use this dispersion index to enhance the performance of DPO in a broad array of benchmark tasks, from synthetic bandit selection to controllable generations and dialogues, while maintaining great generalization capabilities. MallowsPO is also compatible with other SOTA offline preference optimization methods, boosting nearly 2\% extra LC win rate when used as a plugin for fine-tuning Llama3-Instruct.

cs.LG↗

arXiv2008

Linear Optimization over a Polymatroid with Side Constraints -- Scheduling Queues and Minimizing Submodular Functions

Two seemingly unrelated problems, scheduling a multiclass queueing system and minimizing a submodular function, share a rather deep connection via the polymatroid that is characterized by a submodular set function on the one hand and represents the performance polytope of the queueing system on the other hand. We first develop what we call a {\it grouping} algorithm that solves the queueing scheduling problem under side constraints, with a computational effort of $O(n^3LP(n))$, $n$ being the number of job classes, and LP(n) being the computational efforts of solving a linear program with no more than $n$ variables and $n$ constraints. The algorithm organizes the job classes into groups, and identifies the optimal policy to be a priority rule across the groups and a randomized rule within each group (to enforce the side constraints). We then apply the grouping algorithm to the submodular function minimization, mapping the latter to a queueing scheduling problem with side constraints. %Each time the algorithm is applied, it identifies a subset; and We show the minimizing subset can be identified by applying the grouping algorithm $n$ times. Hence, this results in a algorithm that minimizes a submodular function with an effort of $O(n^4LP(n))$.

math.OC↗

arXiv2007

Stochastic Knapsack Problem Revisited: Switch-Over Policies and Dynamic Pricing

The stochastic knapsack has been used as a model in wide ranging applications from dynamic resource allocation to admission control in telecommunication. In recent years, a variation of the model has become a basic tool in studying problems that arise in revenue management and dynamic/flexible pricing; and it is in this context that our study is undertaken. Based on a dynamic programming formulation and associated properties of the value function, we study in this paper a class of control that we call switch-over policies -- start from accepting only orders of the highest price, and switch to including lower prices as time goes by, with the switch-over times optimally decided via convex programming. We establish the asymptotic optimality of the switch-over policy, and develop pricing models based on this policy to optimize the price reductions over the decision horizon.

q-fin.PR↗