arXiv subjects

Ethan Lee

Publications and source records attributed to Ethan Lee.

4 recordsLinked to original sources

arXiv2026

Optimal Value Inference for Reinforcement Learning

We study offline inference for the optimal value in reinforcement learning. Two new nuisances are derived as fixed points of a self-induced Bellman equation, in which we approximate the maximum Bellman operator by its softmax correspondence. We propose a debiased estimator through the Neyman orthogonality and establish its asymptotic normality under diverging horizons even when the behavior policy changes with time, as long as the nuisances have the statistical rates that can be achieved by many machine learning methods. We provide a concrete estimating procedure for these nuisances and show they can lead to valid inference. Synthetic experiments validate the numerical performance of our inference method, and we implement it in real-life decision-making problems, including bike repositioning and AI agentic tool use.

stat.ML↗

arXiv2025

Contextual Online Uncertainty-Aware Preference Learning for Human Feedback

Reinforcement Learning from Human Feedback (RLHF) has become a pivotal paradigm in artificial intelligence to align large models with human preferences. In this paper, we propose a novel statistical framework to simultaneously conduct the online decision-making and statistical inference on the optimal model using human preference data based on dynamic contextual information. Our approach introduces an efficient decision strategy that achieves both the optimal regret bound and the asymptotic distribution of the estimators. A key challenge in RLHF is handling the dependent online human preference outcomes with dynamic contexts. To address this, in the methodological aspect, we propose a two-stage algorithm starting with $\epsilon$-greedy followed by exploitations; in the theoretical aspect, we tailor anti-concentration inequalities and matrix martingale concentration techniques to derive the uniform estimation rate and asymptotic normality of the estimators using dependent samples from both stages. Extensive simulation results demonstrate that our method outperforms state-of-the-art strategies. We apply the proposed framework to analyze the human preference data for ranking large language models on the Massive Multitask Language Understanding dataset, yielding insightful results on the performance of different large language models for medical anatomy knowledge.

stat.ML↗

arXiv2023

Education for a Future in Crisis: Developing a Humanities-Informed STEM Curriculum

In the popular imagination, science and technology are often seen as fields of knowledge production critical to social progress and a cooperative future. This optimistic portrayal of technological advancement also features prominently in internal discourses amongst scientists, industry leaders, and STEM students alike. Yet, an overwhelming body of research, investigation, and first-person accounts highlight the varying ways modern science, technology, and engineering industries contribute to the degradation of our changing environments and exploit and harm global low-income and marginalized populations. By and large, siloed higher-education STEM curricula provide inadequate opportunities for undergraduate and graduate students to critically analyze the historical and epistemological foundations of scientific knowledge production and even fewer tools to engage with and respond to modern community-based cases. Here, we describe the development of a humanities- and social sciences-informed curriculum designed to address the theory, content, and skill-based needs of traditional STEM students considering technoscientific careers. In essence, this course is designed to foster behavior change, de-center dominant ways of knowing in the sciences, and bolster self-reflection and critical-thinking skills to equip the developing STEM workforce with a more nuanced and accurate understanding of the social, political, and economic role of science and technology. This curriculum has the potential to empower STEM-educated professionals to contribute to a more promising, inclusive future. Our framework foregrounds key insights from science and technology studies, Black and Native feminisms, queer theory, and disability studies, alongside real-world case studies using critical pedagogies.

physics.ed-ph↗

arXiv2022

On the Joint Evolution Problem for a Scalar Field and its Singularity

In the classical electrodynamics of point charges in vacuum, the electromagnetic field, and therefore the Lorentz force, is ill-defined at the locations of the charges. Kiessling resolved this problem by using the momentum balance between the field and the particles, extracting an equation for the force that is well-defined where the charges are located, so long as the field momentum density is locally integrable in a neighborhood of the charges. In this paper, we examine the effects of such a force by analyzing a simplified model in one space dimension. We study the joint evolution of a massless scalar field together with its singularity, which we identify with the trajectory of a particle. The static solution arises in the presence of no incoming radiation, in which case the particle remains at rest forever. We will prove the stability of the static solution for particles with positive bare mass by showing that a pulse of incoming radiation that is compactly supported away from the point charge will result in the particle eventually coming back to rest. We will also prove the nonlinear instability of the static solution for particles with negative bare mass by showing that an incoming radiation with arbitrarily small amplitude will cause the particle to reach the speed of light in finite time. We conclude by discussing modifications to this simple model that could make it more realistic.

math-ph↗