arXiv subjects

Zheng Wen

Publications and source records attributed to Zheng Wen.

At least 19 recordsLinked to original sources

arXiv2026

Efficient Exploration at Scale

We develop an online learning algorithm that dramatically improves the data efficiency of reinforcement learning from human feedback (RLHF). Our algorithm incrementally updates reward and language models as choice data is received. The reward model is fit to the choice data, while the language model is updated by a variation of reinforce, with reinforcement signals provided by the reward model. Several features enable the efficiency gains: a small affirmative nudge added to each reinforcement signal, an epistemic neural network that models reward uncertainty, and information-directed exploration. With Gemma large language models (LLMs), our algorithm matches the performance of offline RLHF trained on 200K labels using fewer than 20K labels, representing more than a 10x gain in data efficiency. Extrapolating from our results, we expect our algorithm trained on 1M labels to match offline RLHF trained on 1B labels. This represents a 1,000x gain. To our knowledge, these are the first results to demonstrate that such large improvements are possible.

cs.LG↗

arXiv2026

A Machine Learning-based Non-precipitating Clouds Estimation for THz Dual-Frequency Radar

Accurate measurement of non-precipitable clouds is important for early prediction of heavy rainfall disasters caused by extreme weather events. However, microwave cloud radar cannot observe the early stages of cloud development from non-precipitation clouds (cumulus) to cumulonimbus. In this paper, we propose a terahertz dual-frequency cloud radar using 150 GHz and 95 GHz bands to detect cloud particles in cumulus smaller than 10 μm. Using a dataset generated by the ITU-R radio propagation model, we estimate the liquid water content of non-precipitation clouds and water vapor content in atmospheric gases, respectively, by using a machine learning-based approach. The effectiveness of using the dual wavelength ratio as an explanatory variable is examined.

eess.SP↗

arXiv2026

Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models

Post-training LLMs with RLHF and preference optimization methods (e.g., DPO, IPO) has greatly improved alignment, yet these approaches assume a single objective. In reality, humans express multiple, often conflicting objectives, such as helpfulness and harmlessness, with no natural scalarization. We study the multi-objective preference alignment problem, where a policy must balance several objectives simultaneously. We propose Multi-Objective Preference Optimization (MOPO), a constrained KL-regularized framework that maximizes a primary objective while enforcing lower bounds on secondary objectives via tunable safety thresholds. MOPO operates directly on pairwise preferences without point-wise rewards, and admits simple closed-form iterative updates. Empirically, MOPO recovers Pareto-optimal policies on synthetic benchmarks and, when fine-tuned on human-preference data, yields multi-billion parameter models that achieve higher rewards and Pareto-dominate baselines, with stable and robust optimization dynamics.

cs.LG↗

arXiv2026

Best Policy Learning from Trajectory Preference Feedback

Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful approach for aligning generative models, but its reliance on learned reward models makes it vulnerable to mis-specification and reward hacking. Preference-based Reinforcement Learning (PbRL) offers a more robust alternative by directly leveraging noisy binary comparisons over trajectories. We study the best policy identification problem in PbRL, motivated by post-training optimization of generative models, for example, during multi-turn interactions. Learning in this setting combines an offline preference dataset - potentially biased or out-of-distribution and collected from a rater of subpar `competence' - with online pure exploration, making systematic online learning essential. To this end, we propose Posterior Sampling for Preference Learning ($\mathsf{PSPL}$), a novel algorithm inspired by Top-Two Thompson Sampling that maintains posteriors over the reward model and dynamics. We provide the first Bayesian simple regret guarantees for PbRL and introduce an efficient approximation that outperforms existing baselines on simulation and image generation benchmarks.

cs.LG↗

arXiv2026

Budgeted Online Influence Maximization

We introduce a new budgeted framework for online influence maximization, considering the total cost of an advertising campaign instead of the common cardinality constraint on a chosen influencer set. Our approach better models the real-world setting where the cost of influencers varies and advertisers want to find the best value for their overall social advertising budget. We propose an algorithm assuming an independent cascade diffusion model and edge level semi-bandit feedback, and provide both theoretical and experimental results. Our analysis is also valid for the cardinality constraint setting and improves the state of the art regret bound in this case.

cs.LG↗

arXiv2026

Uniqueness of vertex operator algebras arising from GKO-construction

A series of vertex operator algebras are constructed by GKO-construction, which is a generalization of 3A-algebra and 6A-algebra. It is proved their vertex operator algebra structures are unique under nonzero assumptions on some elements of braiding matrices. Furthermore, we show each of them is generated by weight two subspace, i.e. the Griess algebra.

math.QA↗

arXiv2025

A continuous variable quantum battery with wireless and remote charging

Quantum battery has become one of the hot issues at the research frontiers of quantum physics recently. Charging power, extractable work and wireless charging over long-distance are three important aspects of interest. Non-contact electromagnetic interaction provides an important avenue for wireless charging. In this paper, we design a wireless and remote charging scheme based on the quantized Hamiltonian of two coupled LC circuits, and focus on the charging dynamics of a continuous variable quantum battery. It is found that the quantum entanglement, which is regarded as a significant quantum resource, is not a prerequisite for charging the battery and extracting useful work. On the contrary, all of the energy in the battery could be converted into useful work in the absence of the entanglement for our model. The often overlooked counter-rotating wave coupling in the interacting Hamiltonian is helpful for extracting more useful work from the charger. If the coupling between the battery and the charger is designed to contain both the rotating and counter-rotating wave couplings, the extractable work can be obtained even if the charger is prepared in thermal states, while such effect cannot be achieved in the presence of a rotating or counter-rotating wave coupling alone. Finally, the effect of the system's parameters on the extractable work is discussed, which will provide a reference for full use of the thermal state energy.

quant-ph↗

arXiv2025

Capacity-Constrained Continual Learning

Any agents we can possibly build are subject to capacity constraints, as memory and compute resources are inherently finite. However, comparatively little attention has been dedicated to understanding how agents with limited capacity should allocate their resources for optimal performance. The goal of this paper is to shed some light on this question by studying a simple yet relevant continual learning problem: the capacity-constrained linear-quadratic-Gaussian (LQG) sequential prediction problem. We derive a solution to this problem under appropriate technical conditions. Moreover, for problems that can be decomposed into a set of sub-problems, we also demonstrate how to optimally allocate capacity across these sub-problems in the steady state. We view the results of this paper as a first step in the systematic theoretical study of learning under capacity constraints.

cs.LG↗

arXiv2025

Dicke-Ising quantum battery of an ion chain driven by a mechanical oscillator

A scheme for implementing quantum batteries in a realizable and controllable platform based on a trapped ion chain driven by a mechanical oscillator is proposed. The effects of the hopping interaction between the two-level ions and the coupling interaction between the ions and the external mechanical oscillator on the charging process of the battery are investigated. The importance of the counter-rotating wave terms in the system's Hamiltonian, which are often ignored, is analyzed, and it is found that the charging energy and the ergotropy of the battery are dramatically affected by the counter-rotating wave terms. The quantum phase transition of the two-level system is restrained by the counter-rotating wave terms due to the destruction of the quantum coherence. Lastly, the power-law dependence of the charging process on the distance between the ions is discussed. Our theoretical analysis provides a solid foundation for the development of a practical quantum battery.

quant-ph↗

arXiv2025

Doping-induced Polyamorphic Transitions in Fluorite Oxides

Fluorite oxides such as HfO$_2$ exhibit rich and tunable phase behavior, making them promising candidates for next generation electronic devices. A key challenge is to design amorphous HfO$_2$-based high-$k$ materials with both structural and performance stability. Here, using molecular dynamics simulations supported by experimental measurements, we reveal that Ba doping stimulates a polyamorphic transition in HfO$_2$, yielding a semi-ordered amorphous (SA) phase characterized by disordered oxygens embedded within an ordered metal sublattice. We find that this phase arises from degenerate short-range symmetry breaking modes, consistent with Pauling's parsimony rule. Notably, the SA structure is thermodynamically stable and displays a wider bandgap and higher dielectric constant than conventional random-packing amorphous structure, owing to suppressed subgap states and increased Born effective charges. We further demonstrate that this structural motif generalizes to Ba-, Sr-, and Ca-doped HfO$_2$ and ZrO$_2$, establishing a broadly applicable strategy for designing high-performance amorphous dielectrics.

cond-mat.mtrl-sci↗

arXiv2025

Online Bandit Learning with Offline Preference Data for Improved RLHF

Reinforcement Learning with Human Feedback (RLHF) is at the core of fine-tuning methods for generative AI models for language and images. Such feedback is often sought as rank or preference feedback from human raters, as opposed to eliciting scores since the latter tends to be noisy. On the other hand, RL theory and algorithms predominantly assume that a reward feedback is available. In particular, approaches for online learning that can be helpful in adaptive data collection via active learning cannot incorporate offline preference data. In this paper, we adopt a finite-armed linear bandit model as a prototypical model of online learning. We consider an offline preference dataset to be available generated by an expert of unknown 'competence'. We propose warmPref-PS, a posterior sampling algorithm for online learning that can be warm-started with an offline dataset with noisy preference feedback. We show that by modeling the 'competence' of the expert that generated it, we are able to use such a dataset most effectively. We support our claims with novel theoretical analysis of its Bayesian regret, as well as, extensive empirical evaluation of an approximate loss function that optimizes for infinitely many arms, and performs substantially better than baselines.

cs.LG↗

arXiv2025

Theoretical modeling of synergistic effect of pores and grains on transmittance in transparent piezoelectric ceramics

Transparent piezoelectric ceramics (TPCs) have great application potential in electro-optical-mechanical multi-functional devices. Preparing high-performance TPCs, especially improving the transparency through microstructure regulation, has recently caused extensive discussion. However, there is still controversy about the influence of grains and pores on the transmittance of ceramics, and there is arbitrariness in the estimation of the theoretical transmittance limit. In this paper, taking PMN-PT-based ceramics as an example, theoretical mechanisms for the transmittance are discussed. An inhomogeneous reflection model is derived to improve the theoretical limit of transmittance. The effects of pores and grains scattering on transmittance are investigated. Rayleigh and RGD approximation are discussed to reveal the underlying scattering mechanisms. It is found that Rayleigh approximation is suitable for describing pore scattering, while RGD approximation is suitable for grain scattering. Thus, a Rayleigh-RGD combined model is proposed to describe light scattering in TPCs and successfully employed to fit experimentally measured transmittance curves.

cond-mat.mtrl-sci↗

arXiv2024

Evaluating Large Language Models on Financial Report Summarization: An Empirical Study

In recent years, Large Language Models (LLMs) have demonstrated remarkable versatility across various applications, including natural language understanding, domain-specific knowledge tasks, etc. However, applying LLMs to complex, high-stakes domains like finance requires rigorous evaluation to ensure reliability, accuracy, and compliance with industry standards. To address this need, we conduct a comprehensive and comparative study on three state-of-the-art LLMs, GLM-4, Mistral-NeMo, and LLaMA3.1, focusing on their effectiveness in generating automated financial reports. Our primary motivation is to explore how these models can be harnessed within finance, a field demanding precision, contextual relevance, and robustness against erroneous or misleading information. By examining each model's capabilities, we aim to provide an insightful assessment of their strengths and limitations. Our paper offers benchmarks for financial report analysis, encompassing proposed metrics such as ROUGE-1, BERT Score, and LLM Score. We introduce an innovative evaluation framework that integrates both quantitative metrics (e.g., precision, recall) and qualitative analyses (e.g., contextual fit, consistency) to provide a holistic view of each model's output quality. Additionally, we make our financial dataset publicly available, inviting researchers and practitioners to leverage, scrutinize, and enhance our findings through broader community engagement and collaborative improvement. Our dataset is available on huggingface.

cs.CL↗

arXiv2024

Reinforcement Learning in Credit Scoring and Underwriting

This paper proposes a novel reinforcement learning (RL) framework for credit underwriting that tackles ungeneralizable contextual challenges. We adapt RL principles for credit scoring, incorporating action space renewal and multi-choice actions. Our work demonstrates that the traditional underwriting approach aligns with the RL greedy strategy. We introduce two new RL-based credit underwriting algorithms to enable more informed decision-making. Simulations show these new approaches outperform the traditional method in scenarios where the data aligns with the model. However, complex situations highlight model limitations, emphasizing the importance of powerful machine learning models for optimal performance. Future research directions include exploring more sophisticated models alongside efficient exploration mechanisms.

stat.ML↗

arXiv2024

Active learning of effective Hamiltonian for super-large-scale atomic structures

The first-principles-based effective Hamiltonian scheme provides one of the most accurate modeling technique for large-scale structures, especially for ferroelectrics. However, the parameterization of the effective Hamiltonian is complicated and can be difficult for some complex systems such as high-entropy perovskites. Here, we propose a general form of effective Hamiltonian and develop an active machine learning approach to parameterize the effective Hamiltonian based on Bayesian linear regression. The parameterization is employed in molecular dynamics simulations with the prediction of energy, forces, stress and their uncertainties at each step, which decides whether first-principles calculations are executed to retrain the parameters. Structures of BaTiO$_3$, Pb(Zr$_{0.75}$Ti$_{0.25}$)O$_3$ and (Pb,Sr)TiO$_3$ system are taken as examples to show the accuracy of this approach, as compared with conventional parametrization method and experiments. This machine learning approach provides a universal and automatic way to compute the effective Hamiltonian parameters for any considered complex systems with super-large-scale (more than $10^7$ atoms) atomic structures.

cond-mat.mtrl-sci↗

arXiv2024

Stochastic Low-rank Tensor Bandits for Multi-dimensional Online Decision Making

Multi-dimensional online decision making plays a crucial role in many real applications such as online recommendation and digital marketing. In these problems, a decision at each time is a combination of choices from different types of entities. To solve it, we introduce stochastic low-rank tensor bandits, a class of bandits whose mean rewards can be represented as a low-rank tensor. We consider two settings, tensor bandits without context and tensor bandits with context. In the first setting, the platform aims to find the optimal decision with the highest expected reward, a.k.a, the largest entry of true reward tensor. In the second setting, some modes of the tensor are contexts and the rest modes are decisions, and the goal is to find the optimal decision given the contextual information. We propose two learning algorithms tensor elimination and tensor epoch-greedy for tensor bandits without context, and derive finite-time regret bounds for them. Comparing with existing competitive methods, tensor elimination has the best overall regret bound and tensor epoch-greedy has a sharper dependency on dimensions of the reward tensor. Furthermore, we develop a practically effective Bayesian algorithm called tensor ensemble sampling for tensor bandits with context. Extensive simulations and real analysis in online advertising data back up our theoretical findings and show that our algorithms outperform various state-of-the-art approaches that ignore the tensor low-rank structure.

stat.ML↗

arXiv2024

Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approach

In this paper, we study the problem of efficient online reinforcement learning in the infinite horizon setting when there is an offline dataset to start with. We assume that the offline dataset is generated by an expert but with unknown level of competence, i.e., it is not perfect and not necessarily using the optimal policy. We show that if the learning agent models the behavioral policy (parameterized by a competence parameter) used by the expert, it can do substantially better in terms of minimizing cumulative regret, than if it doesn't do that. We establish an upper bound on regret of the exact informed PSRL algorithm that scales as $\tilde{O}(\sqrt{T})$. This requires a novel prior-dependent regret analysis of Bayesian online learning algorithms for the infinite horizon setting. We then propose the Informed RLSVI algorithm to efficiently approximate the iPSRL algorithm.

cs.LG↗

arXiv2024

RLHF and IIA: Perverse Incentives

Existing algorithms for reinforcement learning from human feedback (RLHF) can incentivize responses at odds with preferences because they are based on models that assume independence of irrelevant alternatives (IIA). The perverse incentives induced by IIA hinder innovations on query formats and learning algorithms.

cs.LG↗