arXiv subjects

Pinyan Lu

Publications and source records attributed to Pinyan Lu.

At least 19 recordsLinked to original sources

arXiv2026

Cold-Start Pricing of LLM Services under Two-Sided Learning

Online large language model (LLM) services face a pricing cold start: providers initially lack the historical data and user information needed for precise pricing, while users' willingness to pay evolves with repeated use. We formulate this challenge as a \emph{two-sided learning} problem between a provider and a user. The user learns from private usage experience, whereas the provider incurs a serving cost and dynamically adapts prices using only purchase decisions to maximize cumulative profit. The provider's central challenge is to learn to price against an evolving willingness to pay without observing the private experiences driving that evolution. We establish matching upper and lower bounds on the minimax regret over horizon $T$, identifying two cost-dependent regimes. When serving costs are sufficiently low, our Calibration-and-Tracking (CAT) algorithm achieves the optimal rate $Θ(T^{1/3})$. In the higher-cost regime, where the provider must learn whether continued service is profitable, Calibration-and-Tracking with Retirement (CAT-R) attains the optimal rate $Θ(\sqrt{T})$. This phase transition shows that serving costs affect not only profit margins but also the fundamental difficulty of learning to price LLM services. Simulations support the predicted regret scaling and show that our algorithms outperform standard online learning baselines in the evaluated settings.

cs.GT↗

arXiv2026

When and How Does Muon Precondition? The Role of Structured Gradient Noise

Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study how gradient noise geometry shapes Muon's preconditioning benefits. We first show, theoretically and empirically, that under the classical isotropic noise assumption, Muon offers no acceleration beyond a rescaling of the learning rate. This motivates us to investigate structured noise arising from mini-batch sampling in associative memory with heterogeneous task frequencies. We establish an exact balancing threshold for two tasks and extend the analysis to multiple tasks, revealing how task coverage and gradient reliability jointly shape learning. Our analysis shows that Muon can soften the dependence on task frequency from linear to square-root, helping balance the learning of frequent and rare tasks. We further identify rotation-induced negative transfer in SignSGD that Muon avoids, and explain how momentum reduces sampling noise while tracking changing gradients. Controlled numerical experiments validate these mechanisms, and a preliminary LLM pretraining study illustrates the practical promise of fast-slow momentum.

cs.LG↗

arXiv2026

Internalizing Chain-of-thought into Looped Transformers

Chain-of-thought (CoT) enables constant-bit-size transformers to perform general-purpose computation through generated tokens (Li and Wang, 2025a), whereas looped transformers carry out the same computation by recurrence in hidden states (latent thought). We construct a family of constant-bit-size looped transformers that interpolates between CoT and latent thought. For a prescribed loop count $L=L(n)$ and a fixed integer $k\ge1$ satisfying $s(n)^{1/k}=Ω(L)$, our transformer simulates a time-$t(n)$, space-$s(n)$ Turing machine using a context window of $O(s(n)/L)$ and $O(t(n)s(n)^{1/k}/L)$ generated tokens, reducing both resources by a factor of $L$ relative to the corresponding unlooped queue-machine simulation. The construction also holds with softmax attention at constant bit-size, under the finite-precision kernel specified in this paper.

cs.CC↗

arXiv2026

Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks

Neural networks trained on modular addition tasks often develop Fourier-structured representations that support exact generalization. While prior work has identified these Fourier circuits, the mechanism by which gradient-based training selects them from the data distribution remains unclear. We address this question using probability signatures, which express leading gradient interactions through conditional statistics of the training distribution. For modular addition, these signatures are cyclic shift operators and are diagonalized by the discrete Fourier transform, yielding approximately decoupled Fourier-mode dynamics. This explains the emergence of Fourier sparsity, frequency matching, and phase alignment. The same framework resolves a puzzle under label noise: corrupted examples can show faster early loss decrease than clean examples, despite lacking a coherent generalization rule. We show that noise increases conditional label collisions, strengthening early shared-coordinate reinforcement. Finally, this method can be applied to other operators. Taking XOR as an example, we observed the predicted frequency in experiments.

cs.AI↗

arXiv2026

Are Parameter-Efficient Fine-tuning Methods Really Different?

Parameter-efficient fine-tuning (PEFT) offers many parameterizations, yet their methodological and functional differences remain unclear. We compare six methods in language and diffusion models to examine how their parameterizations relate to task performance, forgetting, and changes in pretrained weight geometry. Motivated by the spectrum-preserving design of orthogonal fine-tuning (OFT), we first ask whether spectral preservation is itself important for adaptation and retention. We find that the selected LoRA-family methods also approximately preserve pretrained geometry, and that restoring their slightly drifted singular-value spectra largely preserves task performance, questioning the necessity of explicit geometric preservation. Beyond this, we observe that some methods exhibit distinct adaptation--retention trade-offs that vary across settings: LoRA most consistently limits forgetting at competitive performance, DoRA achieves higher mean task scores than LoRA in most comparisons, while PiSSA often incurs greater retention costs. Further intervention experiments suggest that while performance gains from different PEFT methods can be attributed to modifications in different groups of spectral components, we consistently find that restoring dominant rather than intermediate or trailing components produces the largest mean reduction in general-text NLL or base-image drift. Together, these results motivate evaluating geometric constraints through their functional consequences rather than preservation alone. Code is available at https://github.com/Kuaaannn/PEFT_methods.

cs.LG↗

arXiv2026

AutoDataBench: A Data-centric Testbed for Accelerating Auto Research

Existing auto-research benchmarks often entangle multiple sources of improvement, including training frameworks, hyperparameters, compute budgets, and data, making it difficult to attribute why one frontier agent outperforms another to specific research capabilities. In this work, we isolate and systematically evaluate Data Intelligence: an agent's ability to understand, manipulate, and improve the data that shapes model capabilities. We introduce AutoDataBench, a controlled testbed built on a conceptual framework of data intelligence spanning data diagnosis, data organization, and data construction, instantiated through three highly curated optimization tasks while holding non-data factors fixed. Across tool use, retrieval, and knowledge injection, we evaluate frontier LLMs' ability to improve training data through iterative experimentation under task-specific resource budgets. Beyond optimization performance, we ask: do LLMs understand what their data interventions do? We compare predictions made before training with observed outcomes to seek evidence of data-effect reasoning beyond trial and error, and explore whether iterative feedback helps LLMs better understand how changes to training data affect model performance. Finally, we show that reusing AutoDataBench trajectories for mid-training improves downstream coding performance, highlighting its value in both evaluating data intelligence and generating high-quality training data. Code and resources are available at https://github.com/AutoDataBench/AutoDataBench.

cs.CL↗

arXiv2026

Auctions with Contract Design

We consider a new auction model where the bidders' utilities and the auctioneer's revenue depend on a quality factor of the transaction determined by costly and strategic investments of the bidders. Applications of our model include ad auctions, government concessions and crowdsourcing contests. Crucially, these quality-enhancing efforts made by the bidders are often sunk costs incurred prior to the allocation, creating a fundamental moral hazard problem where the risk of losing the auction discourages investments. In this paper, we study the design of revenue-maximizing contracts integrated into auctions: the auctioneer commits to a transfer rule that rewards the winner for the ex-post realized quality of the transaction to incentivize higher effort. Our new framework is a natural generalization of both the auction theory and the principal-agent model. We consider both the second-price and the first-price auctions. We show that natural symmetric Bayes Nash equilibria exist in both auctions. Assuming these natural equilibria are played by the bidders and the number of bidders is large, we study linear contracts and derive the optimal reward factor of the transfer rule that maximizes the auctioneer's revenue. As the main result, we show that the optimal reward factor converges to the auctioneer's marginal benefit from the quality, as the number of bidders grows. That is, it is optimal for the auctioneer to fully pass through the quality value to the winner. This observation is largely independent of the auction rule used: we derive a revenue equivalence theorem showing that the revenue remains the same as long as symmetric Bayes Nash equilibria exist. Lastly, by quantitatively comparing with the standard auctions where no quality reward is used, we show that the use of contracts effectively improves the revenue by incentivizing high investments from the bidders.

cs.GT↗

arXiv2026

Fair Division by Contribution: A Shapley Value Perspective

In many resource allocation problems, agents' valuations are best interpreted not as subjective preferences, but as the value they generate from receiving resources. Such valuations capture productivity, effectiveness, or technology, and may differ significantly across agents. In these settings, classical fairness notions such as proportionality or envy-freeness fail to reflect agents' heterogeneous contributions to the collective outcome. Motivated by this perspective, we introduce \emph{Shapley Value Fairness (SVF)} for the allocation of divisible goods without monetary transfers. SVF interprets an agent's entitlement as her expected marginal contribution to optimal social welfare, and uses the Shapley value of the associated welfare maximization game as a normative fairness benchmark. We position SVF relative to existing fairness notions and show that it provides a natural bridge between fairness and efficiency in contribution-based environments. Since exact implementation of the Shapley value is generally infeasible without transfers, SVF naturally leads to the problem of finding allocations that approximate this benchmark as well as possible. We provide a systematic worst-case analysis of the achievable Shapley approximation ratio. For general concave valuations, we establish a tight $Θ(\ln n)$ bound. For capped concave valuations with bounded demands, this bound improves to $Θ(\ln D)$, where $D$ is the maximum aggregate demand for any item. For linear valuations, we further refine the bound to $Θ(\min\{k, \ln γ, \ln n\})$ in terms of the number of agent types $k$ and the value fluctuation ratio $γ$, and show that all bounds are asymptotically tight. Regarding per-instance guarantees, we show that a near-optimal approximation allocation can be computed efficiently (with high probability) via sampling for general concave valuations.

cs.GT↗

arXiv2026

Benchmark-Tight Approximation Ratio of Simple Mechanism for a Unit-Demand Buyer

We study revenue maximization in the unit-demand single-buyer setting. Our main result is that \textsf{Uniform-Ironed-Virtual-Value Item Pricing} guarantees a {\em tight} $3$-approximation to the \textsf{Duality Relaxation Benchmark} [Chawla-Malec-Sivan, EC'10/GEB'15; Cai-Devanur-Weinberg, STOC'16/ SICOMP'21], breaking the barrier of $4$ since [Chawla-Hartline-Malec-Sivan, STOC'10; Chawla-Malec-Sivan, EC'10/GEB'15]. To our knowledge, this is the first {\em benchmark-tight} revenue guarantee of any simple multi-item mechanism. Technically, all previous works employ \textsf{Myerson Auction} as an intermediary. The barrier of $4$ follows as \textsf{Uniform-Ironed-Virtual-Value Item Pricing} achieves a {\em tight} $2$-approximation to \textsf{Myerson Auction}, which then achieves a {\em tight} $2$-approximation to \textsf{Duality Relaxation Benchmark}. Instead, our new approach avoids \textsf{Myerson Auction}, thus enabling the improvement. Central to our work are a {\em benchmark-based} $3$-competitive prophet inequality and its {\em fully constructive} proof. Such variant prophet inequalities shall find future applications, e.g., to Multi-Item Mechanism Design where optimal revenues are relaxed to various more accessible benchmarks. We complement our benchmark-tight ratio with an impossibility result. All previous works and ours follow the {\em single-dimensional representative} approach introduced by [Chawla-Hartline-Kleinberg, EC'07]. Against \textsf{Duality Relaxation Benchmark}, it turns out that this approach cannot beat our bound of $3$ for a large class of \textsf{Item Pricing}'s.

cs.GT↗

arXiv2026

Muon in Associative Memory Learning: Training Dynamics and Scaling Laws

Muon updates matrix parameters via the matrix sign of the gradient and has shown strong empirical gains, yet its dynamics and scaling behavior remain unclear in theory. We study Muon in a linear associative memory model with softmax retrieval and a hierarchical frequency spectrum over query-answer pairs, with and without label noise. In this setting, we show that Gradient Descent (GD) learns frequency components at highly imbalanced rates, leading to slow convergence bottlenecked by low-frequency components. In contrast, the Muon optimizer mitigates this imbalance, leading to faster and more uniform progress. Specifically, in the noiseless case, Muon achieves an exponential speedup over GD; in the noisy case with a power-law frequency spectrum, we derive Muon's scaling law and demonstrate its superior scaling efficiency over GD. Furthermore, we show that Muon can be interpreted as an implicit matrix preconditioner arising from adaptive task alignment and block-symmetric gradient structure. In contrast, the preconditioner with coordinate-wise sign operator could match Muon under oracle access to unknown task representations, which is infeasible for SignGD in practice. Experiments on synthetic long-tail classification and LLaMA-style pre-training corroborate the theory.

cs.LG↗

arXiv2026

Understanding the Ability of LLMs to Handle Character-Level Perturbation

This work investigates the resilience of contemporary large language models (LLMs) against frequent character-level perturbations. We examine three types of character-level perturbations including introducing numerous typos within words, shuffling the characters in each word, and inserting a large number of invisible characters into the text. Surprisingly, even under severe perturbation, such as shuffling nearly all words character-wise to produce text that is almost unreadable to humans, or inserting invisible characters which are several times more than the visible ones as noise, many LLMs still maintain notable performance. We explore the underlying causes of this robustness and find that LLMs exhibit remarkable resilience to chaotic segmentation and fragmented tokenization. Furthermore, we examine the mechanisms by which LLMs remove perturbations to correctly comprehend text, including both implicit and explicit mechanisms for character-level perturbation. We hope that our findings on the low-level robustness of LLMs will unveil their inherent architectural strengths, reveal the potential risks of their misuse, and inform the reliable deployment of LLMs across diverse application scenarios.

cs.CL↗

arXiv2026

Tight Regret Bounds for Fixed-Price Bilateral Trade

We examine fixed-price mechanisms in bilateral trade through the lens of regret minimization. Our main results are twofold. (i) For independent values, a near-optimal $\widetildeΘ(T^{2/3})$ tight bound for $\textsf{Global Budget Balance}$ fixed-price mechanisms with two-bit/one-bit feedback. (ii) For correlated/adversarial values, a near-optimal $Ω(T^{3/4})$ lower bound for $\textsf{Global Budget Balance}$ fixed-price mechanisms with two-bit/one-bit feedback, which improves the best known $Ω(T^{5/7})$ lower bound obtained in the work [BCCF24] and, up to polylogarithmic factors, matches the $\widetilde{\mathcal{O}}(T^{3 / 4})$ upper bound obtained in the same work. Our work in combination with the previous works [CCCFL24mor, CCCFL24jmlr, AFF24, BCCF24] (essentially) gives a thorough understanding of regret minimization for fixed-price bilateral trade. En route, we have developed two technical ingredients that might be of independent interest: (i) A novel algorithmic paradigm, called $\textit{fractal elimination}$, to address one-bit feedback and independent values. (ii) A new $\textit{lower-bound construction}$ with novel proof techniques, to address the $\textsf{Global Budget Balance}$ constraint and correlated values.

cs.GT↗

arXiv2026

Unveiling Memorization-Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise

Highly over-parameterized models can simultaneously memorize noisy labels and generalize well, yet how these behaviors coexist remains poorly understood. In this work, we investigate the underlying mechanisms of this coexistence using modular arithmetic tasks under heavy label noise. Through extensive experiments on two-layer neural networks, we find that larger models tend to generalize better under appropriate optimization and model configurations, while noisy labels are memorized faster than clean data. Over-parameterized models internally form a generalization structure, but its expression in the output is suppressed by the need to fit noisy labels. Remarkably, even with 80\% label noise, near-perfect test accuracy can be achieved by extracting this internal structure using frequency-based methods. We further propose a task-agnostic method to partition networks into generalization and memorization components. Although this subnetwork improves generalization, it is limited compared with frequency-based extraction, indicating that the generalization structure is distributed across neurons and motivating the development of new tools to retrieve generalizable knowledge from over-parameterized networks.

cs.LG↗

arXiv2026

SHAPE: Stage-aware Hierarchical Advantage via Potential Estimation for LLM Reasoning

Process supervision has emerged as a promising approach for enhancing LLM reasoning, yet existing methods fail to distinguish meaningful progress from mere verbosity, leading to limited reasoning capabilities and unresolved token inefficiency. To address this, we propose Stage-aware Hierarchical Advantage via Potential Estimation (SHAPE), a framework that formalizes reasoning as a trajectory through a state space of empirical solvability. SHAPE introduces a hierarchical credit assignment mechanism: at the segment level, it employs a stage-aware advantage function to prioritize efficient breakthroughs in low-potential states; at the token level, it utilizes entropy-driven redistribution to sharpen execution signals. Extensive experiments in math reasoning across three base models and five benchmarks demonstrate that SHAPE achieves an average accuracy gain of 3% with 30% reduced token consumption.

cs.LG↗

arXiv2026

Budget-Constrained Agentic Large Language Models: Intention-Based Planning for Costly Tool Use

We study budget-constrained tool-augmented agents, where a large language model must solve multi-step tasks by invoking external tools under a strict monetary budget. We formalize this setting as sequential decision making in context space with priced and stochastic tool executions, making direct planning intractable due to massive state-action spaces, high variance of outcomes and prohibitive exploration cost. To address these challenges, we propose INTENT, an inference-time planning framework that leverages an intention-aware hierarchical world model to anticipate future tool usage, risk-calibrated cost, and guide decisions online. Across cost-augmented StableToolBench, INTENT strictly enforces hard budget feasibility while substantially improving task success over baselines, and remains robust under dynamic market shifts such as tool price changes and varying budgets.

cs.AI↗

arXiv2026

A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention

Transformers serve as the foundation of most modern large language models. To mitigate the quadratic complexity of standard full attention, various efficient attention mechanisms, such as linear and hybrid attention, have been developed. A fundamental gap remains: their expressive power relative to full attention lacks a rigorous theoretical characterization. In this work, we theoretically characterize the performance differences among these attention mechanisms. Our theory applies to all linear attention variants that can be formulated as a recurrence, including Mamba, DeltaNet, etc. Specifically, we establish an expressiveness hierarchy: for the sequential function composition-a multi-step reasoning task that must occur within a model's forward pass, an ($L+1$)-layer full attention network is sufficient, whereas any hybrid network interleaving $L-1$ layers of full attention with a substantially larger number ($2^{3L^2}$) of linear attention layers cannot solve it. This result demonstrates a clear separation in expressive power between the two types of attention. Our work provides the first provable separation between hybrid attention and standard full attention, offering a theoretical perspective for understanding the fundamental capabilities and limitations of different attention mechanisms.

cs.LG↗

arXiv2026

BAPS: A Fine-Grained Low-Precision Scheme for Softmax in Attention via Block-Aware Precision reScaling

As the performance gains from accelerating quantized matrix multiplication plateau, the softmax operation becomes the critical bottleneck in Transformer inference. This bottleneck stems from two hardware limitations: (1) limited data bandwidth between matrix and vector compute cores, and (2) the significant area cost of high-precision (FP32/16) exponentiation units (EXP2). To address these issues, we introduce a novel low-precision workflow that employs a specific 8-bit floating-point format (HiF8) and block-aware precision rescaling for softmax. Crucially, our algorithmic innovations make low-precision softmax feasible without the significant model accuracy loss that hampers direct low-precision approaches. Specifically, our design (i) halves the required data movement bandwidth by enabling matrix multiplication outputs constrained to 8-bit, and (ii) substantially reduces the EXP2 unit area by computing exponentiations in low (8-bit) precision. Extensive evaluation on language models and multi-modal models confirms the validity of our method. By alleviating the vector computation bottleneck, our work paves the way for doubling end-to-end inference throughput without increasing chip area, and offers a concrete co-design path for future low-precision hardware and software.

cs.LG↗

arXiv2026

The Query Complexity of Uniform Pricing

Real-world pricing mechanisms are typically optimized using training data, a setting corresponding to the \textit{pricing query complexity} problem in Mechanism Design. The previous work [LSTW23] studies the \textit{single-distribution} case, with tight bounds of $\widetildeΘ(\varepsilon^{-3})$ for a \textit{general} distribution and $\widetildeΘ(\varepsilon^{-2})$ for either a \textit{regular} or \textit{monotone-hazard-rate (MHR)} distribution, where $\varepsilon \in (0, 1)$ denotes the (additive) revenue loss of a learned uniform price relative to the Bayesian-optimal uniform price. This can be directly interpreted as ``the query complexity of the {\em \textsf{Uniform Pricing}} mechanism, in the \textit{single-distribution} case''. Yet in the \textit{multi-distribution} case, can the regularity and MHR conditions still lead to improvements over the tight bound $\widetildeΘ(\varepsilon^{-3})$ for general distributions? We answer this question in the negative, by establishing a (near-)matching lower bound $Ω(\varepsilon^{-3})$ for either \textit{two regular distributions} or \textit{three MHR distributions}. We also address the \textit{regret minimization} problem and, in comparison with the folklore upper bound $\widetilde{O}(T^{2 / 3})$ for general distributions (see, e.g., [SW24]), establish a (near-)matching lower bound $Ω(T^{2 / 3})$ for either \textit{two regular distributions} or \textit{three MHR distributions}, via a black-box reduction. Again, this is in stark contrast to the tight bound $\widetildeΘ(T^{1 / 2})$ for a single regular or MHR distribution.

cs.GT↗