arXiv subjects

Ethan Blaser

Publications and source records attributed to Ethan Blaser.

8 recordsLinked to original sources

arXiv2026

Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes

The average reward is a fundamental performance metric in reinforcement learning (RL) focusing on the long-run performance of an agent. Differential temporal difference (TD) learning algorithms are a major advance for average reward RL as they provide an efficient online method to learn the value functions associated with the average reward in both on-policy and off-policy settings. However, existing convergence guarantees require a local clock in learning rates tied to state visit counts, which practitioners do not use and does not extend beyond tabular settings. We address this limitation by proving the almost sure convergence of on-policy $n$-step differential TD for any $n$ using standard diminishing learning rates without a local clock. We then derive three sufficient conditions under which off-policy $n$-step differential TD also converges without a local clock. These results strengthen the theoretical foundations of differential TD and bring its convergence analysis closer to practical implementations.

cs.LG↗

arXiv2025

A Survey of In-Context Reinforcement Learning

Reinforcement learning (RL) agents typically optimize their policies by performing expensive backward passes to update their network parameters. However, some agents can solve new tasks without updating any parameters by simply conditioning on additional context such as their action-observation histories. This paper surveys work on such behavior, known as in-context reinforcement learning.

cs.LG↗

arXiv2024

Asymptotic and Finite Sample Analysis of Nonexpansive Stochastic Approximations with Markovian Noise

Stochastic approximation is a powerful class of algorithms with celebrated success. However, a large body of previous analysis focuses on stochastic approximations driven by contractive operators, which is not applicable in some important reinforcement learning settings like the average reward setting. This work instead investigates stochastic approximations with merely nonexpansive operators. In particular, we study nonexpansive stochastic approximations with Markovian noise, providing both asymptotic and finite sample analysis. Key to our analysis are novel bounds of noise terms resulting from the Poisson equation. As an application, we prove for the first time that classical tabular average reward temporal difference learning converges to a sample-path dependent fixed point.

cs.LG↗

arXiv2024

Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning

Traditionally, reinforcement learning (RL) agents learn to solve new tasks by updating their neural network parameters through interactions with the task environment. However, recent works demonstrate that some RL agents, after certain pretraining procedures, can learn to solve unseen new tasks without parameter updates, a phenomenon known as in-context reinforcement learning (ICRL). The empirical success of ICRL is widely attributed to the hypothesis that the forward pass of the pretrained agent neural network implements an RL algorithm. In this paper, we support this hypothesis by showing, both empirically and theoretically, that when a transformer is trained for policy evaluation tasks, it can discover and learn to implement temporal difference learning in its forward pass.

cs.LG↗

arXiv2024

Federated Linear Contextual Bandits with Heterogeneous Clients

The demand for collaborative and private bandit learning across multiple agents is surging due to the growing quantity of data generated from distributed systems. Federated bandit learning has emerged as a promising framework for private, efficient, and decentralized online learning. However, almost all previous works rely on strong assumptions of client homogeneity, i.e., all participating clients shall share the same bandit model; otherwise, they all would suffer linear regret. This greatly restricts the application of federated bandit learning in practice. In this work, we introduce a new approach for federated bandits for heterogeneous clients, which clusters clients for collaborative bandit learning under the federated learning setting. Our proposed algorithm achieves non-trivial sub-linear regret and communication cost for all clients, subject to the communication protocol under federated learning that at anytime only one model can be shared by the server.

cs.LG↗

arXiv2022

Graph Structural Attack by Perturbing Spectral Distance

Graph Convolutional Networks (GCNs) have fueled a surge of research interest due to their encouraging performance on graph learning tasks, but they are also shown vulnerability to adversarial attacks. In this paper, an effective graph structural attack is investigated to disrupt graph spectral filters in the Fourier domain, which are the theoretical foundation of GCNs. We define the notion of spectral distance based on the eigenvalues of graph Laplacian to measure the disruption of spectral filters. We realize the attack by maximizing the spectral distance and propose an efficient approximation to reduce the time complexity brought by eigen-decomposition. The experiments demonstrate the remarkable effectiveness of the proposed attack in both black-box and white-box settings for both test-time evasion attacks and training-time poisoning attacks. Our qualitative analysis suggests the connection between the imposed spectral changes in the Fourier domain and the attack behavior in the spatial domain, which provides empirical evidence that maximizing spectral distance is an effective way to change the graph structural property and thus disturb the frequency components for graph filters to affect the learning of GCNs.

cs.LG↗

arXiv2021

Graph Embedding with Hierarchical Attentive Membership

The exploitation of graph structures is the key to effectively learning representations of nodes that preserve useful information in graphs. A remarkable property of graph is that a latent hierarchical grouping of nodes exists in a global perspective, where each node manifests its membership to a specific group based on the context composed by its neighboring nodes. Most prior works ignore such latent groups and nodes' membership to different groups, not to mention the hierarchy, when modeling the neighborhood structure. Thus, they fall short of delivering a comprehensive understanding of the nodes under different contexts in a graph. In this paper, we propose a novel hierarchical attentive membership model for graph embedding, where the latent memberships for each node are dynamically discovered based on its neighboring context. Both group-level and individual-level attentions are performed when aggregating neighboring states to generate node embeddings. We introduce structural constraints to explicitly regularize the inferred memberships of each node, such that a well-defined hierarchical grouping structure is captured. The proposed model outperformed a set of state-of-the-art graph embedding solutions on node classification and link prediction tasks in a variety of graphs including citation networks and social networks. Qualitative evaluations visualize the learned node embeddings along with the inferred memberships, which proved the concept of membership hierarchy and enables explainable embedding learning in graphs.

cs.AI↗

arXiv2020

Utilizing Small Telescopes Operated by Citizen Scientists for Transiting Exoplanet Follow-up

Due to the efforts by numerous ground-based surveys and NASA's Kepler and TESS, there will be hundreds, if not thousands, of transiting exoplanets ideal for atmospheric characterization via spectroscopy with large platforms such as JWST and ARIEL. However their next predicted mid-transit time could become so increasingly uncertain over time that significant overhead would be required to ensure the detection of the entire transit. As a result, follow-up observations to characterize these exoplanetary atmospheres would require less-efficient use of an observatory's time---which is an issue for large platforms where minimizing observing overheads is a necessity. Here we demonstrate the power of citizen scientists operating smaller observatories ($\le$1-m) to keep ephemerides "fresh", defined here as when the 1$σ$ uncertainty in the mid-transit time is less than half the transit duration. We advocate for the creation of a community-wide effort to perform ephemeris maintenance on transiting exoplanets by citizen scientists. Such observations can be conducted with even a 6-inch telescope, which has the potential to save up to $\sim$10,000~days for a 1000-planet survey. Based on a preliminary analysis of 14 transits from a single 6-inch MicroObservatory telescope, we empirically estimate the ability of small telescopes to benefit the community. Observations with a small-telescope network operated by citizen scientists are capable of resolving stellar blends to within 5''/pixel, can follow-up long period transits in short-baseline TESS fields, monitor epoch-to-epoch stellar variability at a precision 0.67\%$\pm$0.12\% for a 11.3 V-mag star, and search for new planets or constrain the masses of known planets with transit timing variations greater than two minutes.

astro-ph.EP↗