arXiv subjects

Yi Wan

Publications and source records attributed to Yi Wan.

54 records · Page 3Linked to original sources

arXiv2022

Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods

In recent years, a growing number of deep model-based reinforcement learning (RL) methods have been introduced. The interest in deep model-based RL is not surprising, given its many potential benefits, such as higher sample efficiency and the potential for fast adaption to changes in the environment. However, we demonstrate, using an improved version of the recently introduced Local Change Adaptation (LoCA) setup, that well-known model-based methods such as PlaNet and DreamerV2 perform poorly in their ability to adapt to local environmental changes. Combined with prior work that made a similar observation about the other popular model-based method, MuZero, a trend appears to emerge, suggesting that current deep model-based methods have serious limitations. We dive deeper into the causes of this poor performance, by identifying elements that hurt adaptive behavior and linking these to underlying techniques frequently used in deep model-based RL. We empirically validate these insights in the case of linear function approximation by demonstrating that a modified version of linear Dyna achieves effective adaptation to local changes. Furthermore, we provide detailed insights into the challenges of building an adaptive nonlinear model-based method, by experimenting with a nonlinear version of Dyna.

cs.LG↗

arXiv2022

Loop closure detection using local 3D deep descriptors

We present a simple yet effective method to address loop closure detection in simultaneous localisation and mapping using local 3D deep descriptors (L3Ds). L3Ds are emerging compact representations of patches extracted from point clouds that are learnt from data using a deep learning algorithm. We propose a novel overlap measure for loop detection by computing the metric error between points that correspond to mutually-nearest-neighbour descriptors after registering the loop candidate point cloud by its estimated relative pose. This novel approach enables us to accurately detect loops and estimate six degrees-of-freedom poses in the case of small overlaps. We compare our L3D-based loop closure approach with recent approaches on LiDAR data and achieve state-of-the-art loop closure detection accuracy. Additionally, we embed our loop closure approach in RESLAM, a recent edge-based SLAM system, and perform the evaluation on real-world RGBD-TUM and synthetic ICL datasets. Our approach enables RESLAM to achieve a better localisation accuracy compared to its original loop closure strategy. Our project page is available at github.com/yiming107/l3d_loop_closure.

cs.CV↗

arXiv2022

LiDAR-guided Stereo Matching with a Spatial Consistency Constraint

The complementary fusion of light detection and ranging (LiDAR) data and image data is a promising but challenging task for generating high-precision and high-density point clouds. This study proposes an innovative LiDAR-guided stereo matching approach called LiDAR-guided stereo matching (LGSM), which considers the spatial consistency represented by continuous disparity or depth changes in the homogeneous region of an image. The LGSM first detects the homogeneous pixels of each LiDAR projection point based on their color or intensity similarity. Next, we propose a riverbed enhancement function to optimize the cost volume of the LiDAR projection points and their homogeneous pixels to improve the matching robustness. Our formulation expands the constraint scopes of sparse LiDAR projection points with the guidance of image information to optimize the cost volume of pixels as much as possible. We applied LGSM to semi-global matching and AD-Census on both simulated and real datasets. When the percentage of LiDAR points in the simulated datasets was 0.16%, the matching accuracy of our method achieved a subpixel level, while that of the original stereo matching algorithm was 3.4 pixels. The experimental results show that LGSM is suitable for indoor, street, aerial, and satellite image datasets and provides good transferability across semi-global matching and AD-Census. Furthermore, the qualitative and quantitative evaluations demonstrate that LGSM is superior to two state-of-the-art optimizing cost volume methods, especially in reducing mismatches in difficult matching areas and refining the boundaries of objects.

cs.CV↗

arXiv2022

Scalable CMOS-BEOL compatible AlScN/2D Channel FE-FETs

Intimate integration of memory devices with logic transistors is a frontier challenge in computer hardware. This integration is essential for augmenting computational power concurrently with enhanced energy efficiency in big-data applications such as artificial intelligence. Despite decades of efforts, reliable, compact, energy efficient and scalable memory devices are elusive. Ferroelectric Field Effect Transistors (FE-FETs) are a promising candidate but their scalability and performance in a back-end-of-line (BEOL) process remain unattained. Here, we present scalable BEOL compatible FE-FETs using two-dimensional (2D) MoS2 channel and AlScN ferroelectric dielectric. We have fabricated a large array of FE-FETs with memory windows larger than 7.8 V, ON/OFF ratios of greater than 10^7, and ON current density greater than 250 uA/um, all at ~80 nm channel lengths. Our devices show stable retention up to 20000 secs and endurance up to 20000 cycles in addition to 4-bit pulse programmable memory features thereby opening a path towards scalable 3D hetero-integration of 2D semiconductor memory with Si CMOS logic.

physics.app-ph↗

arXiv2021

Average-Reward Learning and Planning with Options

We extend the options framework for temporal abstraction in reinforcement learning from discounted Markov decision processes (MDPs) to average-reward MDPs. Our contributions include general convergent off-policy inter-option learning algorithms, intra-option algorithms for learning values and models, as well as sample-based planning variants of our learning algorithms. Our algorithms and convergence proofs extend those recently developed by Wan, Naik, and Sutton. We also extend the notion of option-interrupting behavior from the discounted to the average-reward formulation. We show the efficacy of the proposed algorithms with experiments on a continuing version of the Four-Room domain.

cs.LG↗

arXiv2021

Learning and Planning in Average-Reward Markov Decision Processes

We introduce learning and planning algorithms for average-reward MDPs, including 1) the first general proven-convergent off-policy model-free control algorithm without reference states, 2) the first proven-convergent off-policy model-free prediction algorithm, and 3) the first off-policy learning algorithm that converges to the actual value function rather than to the value function plus an offset. All of our algorithms are based on using the temporal-difference error rather than the conventional error when updating the estimate of the average reward. Our proof techniques are a slight generalization of those by Abounadi, Bertsekas, and Borkar (2001). In experiments with an Access-Control Queuing Task, we show some of the difficulties that can arise when using methods that rely on reference states and argue that our new algorithms can be significantly easier to use.

cs.LG↗

arXiv2021

Planning with Expectation Models for Control

In model-based reinforcement learning (MBRL), Wan et al. (2019) showed conditions under which the environment model could produce the expectation of the next feature vector rather than the full distribution, or a sample thereof, with no loss in planning performance. Such expectation models are of interest when the environment is stochastic and non-stationary, and the model is approximate, such as when it is learned using function approximation. In these cases a full distribution model may be impractical and a sample model may be either more expensive computationally or of high variance. Wan et al. considered only planning for prediction to evaluate a fixed policy. In this paper, we treat the control case - planning to improve and find a good approximate policy. We prove that planning with an expectation model must update a state-value function, not an action-value function as previously suggested (e.g., Sorg & Singh, 2010). This opens the question of how planning influences action selections. We consider three strategies for this and present general MBRL algorithms for each. We identify the strengths and weaknesses of these algorithms in computational experiments. Our algorithms and experiments are the first to treat MBRL with expectation models in a general setting.

cs.AI↗

arXiv2021

Detecting Log Anomalies with Multi-Head Attention (LAMA)

Anomaly detection is a crucial and challenging subject that has been studied within diverse research areas. In this work, we explore the task of log anomaly detection (especially computer system logs and user behavior logs) by analyzing logs' sequential information. We propose LAMA, a multi-head attention based sequential model to process log streams as template activity (event) sequences. A next event prediction task is applied to train the model for anomaly detection. Extensive empirical studies demonstrate that our new model outperforms existing log anomaly detection methods including statistical and deep learning methodologies, which validate the effectiveness of our proposed method in learning sequence patterns of log data.

cs.LG↗

arXiv2020

Planning with Expectation Models

Distribution and sample models are two popular model choices in model-based reinforcement learning (MBRL). However, learning these models can be intractable, particularly when the state and action spaces are large. Expectation models, on the other hand, are relatively easier to learn due to their compactness and have also been widely used for deterministic environments. For stochastic environments, it is not obvious how expectation models can be used for planning as they only partially characterize a distribution. In this paper, we propose a sound way of using approximate expectation models for MBRL. In particular, we 1) show that planning with an expectation model is equivalent to planning with a distribution model if the state value function is linear in state features, 2) analyze two common parametrization choices for approximating the expectation: linear and non-linear expectation models, 3) propose a sound model-based policy evaluation algorithm and present its convergence results, and 4) empirically demonstrate the effectiveness of the proposed planning algorithm.

cs.LG↗

arXiv2020

Spin-Valley Locking Effect in Defect States of Monolayer MoS$_2$

Valley pseudospin in two-dimensional (2D) transition-metal dichalcogenides (TMDs) allows optical control of spin-valley polarization and intervalley quantum coherence. Defect states in TMDs give rise to new exciton features and theoretically exhibit spin-valley polarization; however, experimental achievement of this phenomenon remains challenges. Here, we report unambiguous valley pseudospin of defect-bound localized excitons in CVD-grown monolayer MoS2; enhanced valley Zeeman splitting with an effective g-factor of -6.2 is observed. Our results reveal that all five d-orbitals and the increased effective electron mass contribute to the band shift of defect states, demonstrating a new physics of the magnetic responses of defect-bound localized excitons, strikingly different from that of A excitons. Our work paves the way for the manipulation of the spin-valley degrees of freedom through defects toward valleytronic devices.

cond-mat.mtrl-sci↗

arXiv2020

Off-policy Maximum Entropy Reinforcement Learning : Soft Actor-Critic with Advantage Weighted Mixture Policy(SAC-AWMP)

The optimal policy of a reinforcement learning problem is often discontinuous and non-smooth. I.e., for two states with similar representations, their optimal policies can be significantly different. In this case, representing the entire policy with a function approximator (FA) with shared parameters for all states maybe not desirable, as the generalization ability of parameters sharing makes representing discontinuous, non-smooth policies difficult. A common way to solve this problem, known as Mixture-of-Experts, is to represent the policy as the weighted sum of multiple components, where different components perform well on different parts of the state space. Following this idea and inspired by a recent work called advantage-weighted information maximization, we propose to learn for each state weights of these components, so that they entail the information of the state itself and also the preferred action learned so far for the state. The action preference is characterized via the advantage function. In this case, the weight of each component would only be large for certain groups of states whose representations are similar and preferred action representations are also similar. Therefore each component is easy to be represented. We call a policy parameterized in this way an Advantage Weighted Mixture Policy (AWMP) and apply this idea to improve soft-actor-critic (SAC), one of the most competitive continuous control algorithm. Experimental results demonstrate that SAC with AWMP clearly outperforms SAC in four commonly used continuous control tasks and achieve stable performance across different random seeds.

cs.LG↗

arXiv2019

Observation of Fano resonance in photoluminescence of a two-dimensional magnetic semiconductor

Quantum interference gives rise to the asymmetric Fano resonance line shape when the final states of an electronic transition follows within a continuum of states and a discrete state, which has significant applications in optical switching and sensing. The resonant optical phenomena associated with Fano resonance have been observed by absorption spectra, Raman spectra, transmission spectra, etc., but have rarely been reported in photoluminescence (PL) spectroscopy. In this work, we performed spectroscopic studies on layered chromium thiophosphate (CrPS4), a promising ternary antiferromagnetic semiconductor with PL in near-infrared wavelength region and observed Fano resonance when CrPS4 experiences phase transition into the antiferromagnetic state below Néel temperature (38 K). The photoluminescence of the continuum states results from the d band transitions localized at Cr3+ ions, while the discrete state reaches saturation at high excitation power and can be enhanced by the external magnetic field, suggesting it is formed by an impurity level from extra atomic phosphorus. Our findings provide insights into the electronic transitions of CrPS4 and their connection to its intrinsic magnetic properties.

cond-mat.mtrl-sci↗

arXiv2019

Discovery of Two-Dimensional Multiferroicity in van der Waals CuCrP2S6 Layers

Multiferroic materials are potential to be applied in novel magnetoelectric devices, for example, high-density non-volatile storage. Last decades, research on multiferroic materials was focused on three-dimensional (3D) materials. However, 3D materials suffer from the dangling bonds and quantum tunneling in the nano-scale thin films. Two-dimensional (2D) materials might provide an elegant solution to these problems, and thus are highly on demand. Using first-principles calculations, we predict ferromagnetism and driven ferroelectricity in the monolayer and even a few-layers of CuCrP2S6. Although the total energy of the ferroelectric phase of monolayer is higher than that of the antiferroelectric phase, the ferroelectric phases can be realized by applying a large electric field. Besides the degrees of freedoms in the common multiferroic materials, the valley degree of freedom is also polarized according to our calculations. The spins, electric dipoles and valleys are coupled with each other as shown in the computational results. In experiment, we observe the out-of-plane ferroelectricity in a few-layer CuCrP2S6 (approximately 13 nm thick) at room temperature. 2D ferromagnetism of few-layers is inferred from magnetic hysteresis loops of the massively stacked nanosheets at 10 K. The experimental observations support our calculation very well. Our findings may provide a series of 2D materials for further device applications.

cond-mat.mtrl-sci↗

arXiv2015

Clustering Assisted Fundamental Matrix Estimation

In computer vision, the estimation of the fundamental matrix is a basic problem that has been extensively studied. The accuracy of the estimation imposes a significant influence on subsequent tasks such as the camera trajectory determination and 3D reconstruction. In this paper we propose a new method for fundamental matrix estimation that makes use of clustering a group of 4D vectors. The key insight is the observation that among the 4D vectors constructed from matching pairs of points obtained from the SIFT algorithm, well-defined cluster points tend to be reliable inliers suitable for fundamental matrix estimation. Based on this, we utilizes a recently proposed efficient clustering method through density peaks seeking and propose a new clustering assisted method. Experimental results show that the proposed algorithm is faster and more accurate than currently commonly used methods.

cs.CV↗

arXiv1998

Superconductivity in a two-dimensional Electron Gas

In a series of recent experiments, Kravchenko and colleagues observed unexpectedly that a two-dimensional electron gas in zero magnetic field can be a conductor. The two-dimensionality was imposed by confining the electron gas to move laterally at the interface between two semiconductors. The observation of a conductor in two dimensions (2D) is surprising as the conventional theory of metals precludes the presence of a metallic state at zero temperature in 2D. Nonetheless, there are now several experiments confirming the existence of the new conducting phase in a dilute two-dimensional electron gas in zero magnetic field. Here we argue based on an analysis of the experiments and general theoretical grounds that this phase is a zero-temperature superconductor with an inhomogeneous charge density.

cond-mat.str-el↗

arXiv1996

Size Dependence In The Disordered Kondo Problem

We study here the role randomly-placed non-magnetic scatterers play on the Kondo effect. We show that spin relaxation effects (with time $τ_s^o$)in the vertex corrections to the Kondo self-energy lead to an exact cancellation of the singular temperature dependence arising from the diffusion poles. For a thin film of thickness $L$ and a mean-free path $\ell$, disorder provides a correction to the Kondo resistivity of the form $τ_s^o/(k_FL\ell^2)\ln T$ that explains both the disorder and sample-size depression of the Kondo effect observed by Blachly and Giordano (PRB {\bf 51}, 12537 (1995)).

cond-mat↗

arXiv1995

Pair Tunneling in Semiconductor Quantum Dots

We propose here a model for the pair tunneling states observed by Ashoori and co-workers (Phys. Rev. Lett. {\bf 68}, 3088 (1992)) in GaAs quantum dots. We show that while GaAs is a weakly-polar semiconductor, coupling to optical phonons is sufficiently strong to mediate a negative-U pairing state. The physical potential in which the two electrons are bound can be composed of a Si impurity and a parabolic well that originates from the potential created by the $δ-$dopants in the backing layer of the dot. Such a pair state breaks up at moderate magnetic field strengths ($\approx$ 2 T), as is seen experimentally, and is unstable when the confining radius of the dot is smaller than $\approx 400$Å.

cond-mat↗

arXiv1994

Suppression of the Kondo Effect in Quantum Dots by Even-Odd Asymmetry

We analyze here a model for single-electron charging in semiconductor quantum dots that includes the standard Anderson on-site repulsion (U) as well as the spin-exchange ($J_d$) that is inherently present among the electrons occupying the various quantum levels of the dot. We show explicitly that for ferromagnetic coupling ($J_d>0$), an s-d exchange for an S=1 Kondo problem is recovered. In contrast, for the antiferromagnetic case, $J_d<0$, we find that the Kondo effect is present only if there are an odd number of electrons on the dot. In addition, we find that spin-exchange produces a second period in the conductance that is consistent with experimental measurements.

cond-mat↗