arXiv subjects

Hsin-Yu Lai

Publications and source records attributed to Hsin-Yu Lai.

3 recordsLinked to original sources

arXiv2024

Harnessing Causality in Reinforcement Learning With Bagged Decision Times

We consider reinforcement learning (RL) for a class of problems with bagged decision times. A bag contains a finite sequence of consecutive decision times. The transition dynamics are non-Markovian and non-stationary within a bag. All actions within a bag jointly impact a single reward, observed at the end of the bag. For example, in mobile health, multiple activity suggestions in a day collectively affect a user's daily commitment to being active. Our goal is to develop an online RL algorithm to maximize the discounted sum of the bag-specific rewards. To handle non-Markovian transitions within a bag, we utilize an expert-provided causal directed acyclic graph (DAG). Based on the DAG, we construct states as a dynamical Bayesian sufficient statistic of the observed history, which results in Markov state transitions within and across bags. We then formulate this problem as a periodic Markov decision process (MDP) that allows non-stationarity within a period. An online RL algorithm based on Bellman equations for stationary MDPs is generalized to handle periodic MDPs. We show that our constructed state achieves the maximal optimal value function among all state constructions for a periodic MDP. Finally, we evaluate the proposed method on testbed variants built from real data in a mobile health clinical trial.

cs.LG↗

arXiv2020

App-based saccade latency and error determination across the adult age spectrum

We aid in neurocognitive monitoring outside the hospital environment by enabling app-based measurements of visual reaction time (saccade latency) and error rate in a cohort of subjects spanning the adult age spectrum. Methods: We developed an iOS app to record subjects with the frontal camera during pro- and anti-saccade tasks. We further developed automated algorithms for measuring saccade latency and error rate that take into account the possibility that it might not always be possible to determine the eye movement from app-based recordings. Results: To measure saccade latency on a tablet, we ensured that the absolute timing error between on-screen task presentation and the camera recording is within 5 ms. We collected over 235,000 eye movements in 80 subjects ranging in age from 20 to 92 years, with 96% of recorded eye movements either declared good or directional errors. Our error detection code achieved a sensitivity of 0.97 and a specificity of 0.97. Confirming prior reports, we observed a positive correlation between saccade latency and age while the relationship between error rate and age was not significant. Finally, we observed significant intra- and inter-subject variations in saccade latency and error rate distributions, which highlights the importance of individualized tracking of these visual digital biomarkers. Conclusion and Significance: Our system and algorithms allow ubiquitous tracking of saccade latency and error rate, which opens up the possibility of quantifying patient state on a finer timescale in a broader population than previously possible.

q-bio.NC↗

arXiv2020

Delta-Ramp Encoder for Amplitude Sampling and its Interpretation as Time Encoding

The theoretical basis for conventional acquisition of bandlimited signals typically relies on uniform time sampling and assumes infinite-precision amplitude values. In this paper, we explore signal representation and recovery based on uniform amplitude sampling with assumed infinite precision timing information. The approach is based on the delta-ramp encoder which consists of applying a one-level level-crossing detector to the result of adding an appropriate sawtooth-like waveform to the input signal. The output samples are the time instants of these level crossings, thus representing a time-encoded version of the input signal. For theoretical purposes, this system can be equivalently analyzed by reversibly transforming through ramp addition a nonmonotonic input signal into a monotonic one which is then uniformly sampled in amplitude. The monotonic function is then represented by the times at which the signal crosses a predefined and equally-spaced set of amplitude values. We refer to this technique as amplitude sampling. The time sequence generated can be interpreted alternatively as nonuniform time sampling of the original source signal. We derive duality and frequency-domain properties for the functions involved in the transformation. Iterative algorithms are proposed and implemented for recovery of the original source signal. As indicated in the simulations, the proposed iterative amplitude-sampling algorithm achieves a faster convergence rate than frame-based reconstruction for nonuniform sampling. The performance can also be improved by appropriate choice of the parameters while maintaining the same sampling density.

eess.SP↗