arXiv subjects

Tianyi Liu

Publications and source records attributed to Tianyi Liu.

At least 91 records · Page 5Linked to original sources

arXiv2021

Recovery under Side Constraints

This paper addresses sparse signal reconstruction under various types of structural side constraints with applications in multi-antenna systems. Side constraints may result from prior information on the measurement system and the sparse signal structure. They may involve the structure of the sensing matrix, the structure of the non-zero support values, the temporal structure of the sparse representationvector, and the nonlinear measurement structure. First, we demonstrate how a priori information in form of structural side constraints influence recovery guarantees (null space properties) using L1-minimization. Furthermore, for constant modulus signals, signals with row-, block- and rank-sparsity, as well as non-circular signals, we illustrate how structural prior information can be used to devise efficient algorithms with improved recovery performance and reduced computational complexity. Finally, we address the measurement system design for linear and nonlinear measurements of sparse signals. Moreover, we discuss the linear mixing matrix design based on coherence minimization. Then we extend our focus to nonlinear measurement systems where we design parallel optimization algorithms to efficiently compute stationary points in the sparse phase retrieval problem with and without dictionary learning.

cs.IR↗

arXiv2021

GODSAC: Graph Optimized DSAC for Robot Relocalization

Deep learning based camera pose estimation from monocular camera images has seen a recent uptake in Visual SLAM research. Even though such pose estimation approaches have excellent results in small confined areas like offices and apartment buildings, they tend to do poorly when applied to larger areas like outdoor settings, mainly because of the scarcity of distinctive features. We propose GODSAC* as a camera pose estimation approach that augments pose predictions from a trained neural network with noisy odometry data through the optimization of a pose graph. GODSAC* outperforms the state-of-the-art approaches in pose estimation accuracy, as we demonstrate in our experiments.

cs.RO↗

arXiv2021

Tendencies of enhanced electronic nematicity in the Hubbard model and a comparison with Raman scattering on high-temperature superconductors

The pseudogap regime of the cuprate high-temperature superconductors is characterized by a variety of competing orders, the nature of which are still widely debated. Recent experiments have provided evidence for electron nematic order, in which the electron fluid breaks rotational symmetry while preserving translational invariance. Raman spectroscopy, with its ability to symmetry resolve low energy excitations, is a unique tool that can be used to assess nematic fluctuations and nematic ordering tendencies. Here, we compare results from determinant quantum Monte Carlo simulations of the Hubbard model to experimental results from Raman spectroscopy in $\text{La}_{2-x}\text{Sr}_{x}\text{CuO}_{4}$, which show a prominent increase in the $B_{1g}$ response around 10% hole doping as the temperature decreases, indicative of a rise in nematic fluctuations at low energy. Our results support a picture of nematic fluctuations with $B_{1g}$ symmetry occurring in underdoped cuprates, which may arise from melted stripes at elevated temperatures.

cond-mat.supr-con↗

arXiv2021

A Diffusion Approximation Theory of Momentum SGD in Nonconvex Optimization

Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical success, there is still a lack of theoretical understanding of convergence properties of MSGD. To fill this gap, we propose to analyze the algorithmic behavior of MSGD by diffusion approximations for nonconvex optimization problems with strict saddle points and isolated local optima. Our study shows that the momentum helps escape from saddle points, but hurts the convergence within the neighborhood of optima (if without the step size annealing or momentum annealing). Our theoretical discovery partially corroborates the empirical success of MSGD in training deep neural networks.

cs.LG↗

arXiv2021

Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization

Numerous empirical evidences have corroborated the importance of noise in nonconvex optimization problems. The theory behind such empirical observations, however, is still largely unknown. This paper studies this fundamental problem through investigating the nonconvex rectangular matrix factorization problem, which has infinitely many global minima due to rotation and scaling invariance. Hence, gradient descent (GD) can converge to any optimum, depending on the initialization. In contrast, we show that a perturbed form of GD with an arbitrary initialization converges to a global optimum that is uniquely determined by the injected noise. Our result implies that the noise imposes implicit bias towards certain optima. Numerical experiments are provided to support our theory.

cs.LG↗

arXiv2021

Towards Understanding Acceleration Tradeoff between Momentum and Asynchrony in Nonconvex Stochastic Optimization

Asynchronous momentum stochastic gradient descent algorithms (Async-MSGD) is one of the most popular algorithms in distributed machine learning. However, its convergence properties for these complicated nonconvex problems is still largely unknown, because of the current technical limit. Therefore, in this paper, we propose to analyze the algorithm through a simpler but nontrivial nonconvex problem - streaming PCA, which helps us to understand Aync-MSGD better even for more general problems. Specifically, we establish the asymptotic rate of convergence of Async-MSGD for streaming PCA by diffusion approximation. Our results indicate a fundamental tradeoff between asynchrony and momentum: To ensure convergence and acceleration through asynchrony, we have to reduce the momentum (compared with Sync-MSGD). To the best of our knowledge, this is the first theoretical attempt on understanding Async-MSGD for distributed nonconvex stochastic optimization. Numerical experiments on both streaming PCA and training deep neural networks are provided to support our findings for Async-MSGD.

cs.LG↗

arXiv2020

Regularized Attentive Capsule Network for Overlapped Relation Extraction

Distantly supervised relation extraction has been widely applied in knowledge base construction due to its less requirement of human efforts. However, the automatically established training datasets in distant supervision contain low-quality instances with noisy words and overlapped relations, introducing great challenges to the accurate extraction of relations. To address this problem, we propose a novel Regularized Attentive Capsule Network (RA-CapNet) to better identify highly overlapped relations in each informal sentence. To discover multiple relation features in an instance, we embed multi-head attention into the capsule network as the low-level capsules, where the subtraction of two entities acts as a new form of relation query to select salient features regardless of their positions. To further discriminate overlapped relation features, we devise disagreement regularization to explicitly encourage the diversity among both multiple attention heads and low-level capsules. Extensive experiments conducted on widely used datasets show that our model achieves significant improvements in relation extraction.

cs.CL↗

arXiv2020

Low-loss integrated nanophotonic circuits with layered semiconductor materials

Monolayer transition metal dichalcogenides with direct bandgaps are emerging candidates for microelectronics, nano-photonics, and optoelectronics. Transferred onto photonic integrated circuits (PICs), these semiconductor materials have enabled new classes of light-emitting diodes, modulators and photodetectors, that could be amenable to wafer-scale manufacturing. For integrated photonic devices, the optical losses of the PICs are critical. In contrast to silicon, silicon nitride (Si3N4) has emerged as a low-loss integrated platform with a wide transparency window from ultraviolet to mid-infrared and absence of two-photon absorption at telecommunication bands. Moreover, it is suitable for nonlinear integrated photonics due to its high Kerr nonlinearity and high-power handing capability. These features of Si3N4 are intrinsically beneficial for nanophotonics and optoelectronics applications. Here we report a low-loss integrated platform incorporating monolayer molybdenum ditelluride (1L-MoTe2) with Si3N4 photonic microresonators. We show that, with the 1L-MoTe2, microresonator quality factors exceeding 3 million in the telecommunication O-band to E-band are maintained. We further investigate the change of microresonator dispersion and resonance shift due to the presence of 1L-MoTe2, and extrapolate the optical loss introduced by 1L-MoTe2 in the telecommunication bands, out of the excitonic transition region. Our work presents a key step for low-loss, hybrid PICs with layered semiconductors without using heterogeneous wafer bonding.

physics.optics↗

arXiv2020

High-yield wafer-scale fabrication of ultralow-loss, dispersion-engineered silicon nitride photonic circuits

Low-loss photonic integrated circuits (PIC) and microresonators have enabled novel applications ranging from narrow-linewidth lasers, microwave photonics, to chip-scale optical frequency combs and quantum frequency conversion. To translate these results into a widespread technology, attaining ultralow optical losses with established foundry manufacturing is critical. Recent advances in fabrication of integrated Si3N4 photonics have shown that ultralow-loss, dispersion-engineered microresonators can be attained at die-level throughput. For emerging nonlinear applications such as integrated travelling-wave parametric amplifiers and mode-locked lasers, PICs of length scales of up to a meter are required, placing stringent demands on yield and performance that have not been met with current fabrication techniques. Here we overcome these challenges and demonstrate a fabrication technology which meets all these requirements on wafer-level yield, performance and length scale. Photonic microresonators with a mean Q factor exceeding 30 million, corresponding to a linear propagation loss of 1.0 dB/m, are obtained over full 4-inch wafers, as determined from a statistical analysis of tens of thousands of optical resonances and cavity ringdown with 19 ns photon storage time. The process operates over large areas with high yield, enabling 1-meter-long spiral waveguides with 2.4 dB/m loss in dies of only 5x5 mm size. Using a modulation response measurement self-calibrated via the Kerr nonlinearity, we reveal that, strikingly, the intrinsic absorption-limited Q factor of our Si3N4 microresonators exceeds a billion. Transferring the present Si3N4 photonics technology to standard commercial foundries, and merging it with silicon photonics using heterogeneous integration technology, will significantly expand the scope of today's integrated photonics and seed new applications.

physics.app-ph↗

arXiv2020

A Benchmarking Framework for Interactive 3D Applications in the Cloud

With the growing popularity of cloud gaming and cloud virtual reality (VR), interactive 3D applications have become a major type of workloads for the cloud. However, despite their growing importance, there is limited public research on how to design cloud systems to efficiently support these applications, due to the lack of an open and reliable research infrastructure, including benchmarks and performance analysis tools. The challenges of generating human-like inputs under various system/application randomness and dissecting the performance of complex graphics systems make it very difficult to design such an infrastructure. In this paper, we present the design of a novel cloud graphics rendering research infrastructure, Pictor. Pictor employs AI to mimic human interactions with complex 3D applications. It can also provide in-depth performance measurements for the complex software and hardware stack used for cloud 3D graphics rendering. With Pictor, we designed a benchmark suite with six interactive 3D applications. Performance analyses were conducted with these benchmarks to characterize 3D applications in the cloud and reveal new performance bottlenecks. To demonstrate the effectiveness of Pictor, we also implemented two optimizations to address two performance bottlenecks discovered in a state-of-the-art cloud 3D-graphics rendering system, which improved the frame rate by 57.7% on average.

cs.DC↗

arXiv2020

Online Quantification of Input Model Uncertainty by Two-Layer Importance Sampling

Stochastic simulation has been widely used to analyze the performance of complex stochastic systems and facilitate decision making in those systems. Stochastic simulation is driven by the input model, which is a collection of probability distributions that model the stochasticity in the system. The input model is usually estimated using a finite amount of data, which introduces the so-called input model uncertainty to the simulation output. How to quantify input uncertainty has been studied extensively, and many methods have been proposed for the batch data setting, i.e., when all the data are available at once. However, methods for "streaming data" arriving sequentially in time are still in demand, despite that streaming data have become increasingly prevalent in modern applications. To fill this gap, we propose a two-layer importance sampling framework that incorporates streaming data for online input uncertainty quantification. Under this framework, we develop two algorithms that suit different application scenarios: the first scenario is when data come at a fast speed and there is no time for any new simulation in between updates; the second is when data come at a moderate speed and a few but limited simulations are allowed at each time stage. We prove the consistency and asymptotic convergence rate results, which theoretically show the efficiency of our proposed approach. We further demonstrate the proposed algorithms on a numerical example of the news vendor problem.

q-fin.RM↗

arXiv2020

Monolithic piezoelectric control of soliton microcombs

High-speed laser frequency actuation is critical in all applications employing lasers and frequency combs, and is prerequisite for phase locking, frequency stabilization and stability transfer among multiple optical carriers. Soliton microcombs have emerged as chip-scale, broadband and low-power-consumption frequency comb sources.Yet, integrated microcombs relying on thermal heaters for on-chip actuation all exhibit only kilohertz actuation bandwidth. Consequently, high-speed actuation and locking of microcombs have been attained only with off-chip bulk modulators. Here, we present high-speed microcomb actuation using integrated components. By monolithically integrating piezoelectric AlN actuators on ultralow-loss Si3N4 photonic circuits, we demonstrate voltage-controlled soliton tuning, modulation and stabilization. The integrated AlN actuators feature bi-directional tuning with high linearity and low hysteresis, operate with 300 nW power and exhibit flat actuation response up to megahertz frequency, significantly exceeding bulk piezo tuning bandwidth. We use this novel capability to demonstrate a microcomb engine for parallel FMCW LiDAR, via synchronously tuning the laser and microresonator. By applying a triangular sweep at the modulation rate matching the frequency spacing of HBAR modes, we exploit the resonant build-up of bulk acoustic energy to significantly lower the required driving to a CMOS voltage of only 7 Volts. Our approach endows soliton microcombs with integrated, ultralow-power-consumption, and fast actuation, significantly expanding the repertoire of technological applications.

physics.app-ph↗

arXiv2020

On Computation and Generalization of Generative Adversarial Imitation Learning

Generative Adversarial Imitation Learning (GAIL) is a powerful and practical approach for learning sequential decision-making policies. Different from Reinforcement Learning (RL), GAIL takes advantage of demonstration data by experts (e.g., human), and learns both the policy and reward function of the unknown environment. Despite the significant empirical progresses, the theory behind GAIL is still largely unknown. The major difficulty comes from the underlying temporal dependency of the demonstration data and the minimax computational formulation of GAIL without convex-concave structure. To bridge such a gap between theory and practice, this paper investigates the theoretical properties of GAIL. Specifically, we show: (1) For GAIL with general reward parameterization, the generalization can be guaranteed as long as the class of the reward functions is properly controlled; (2) For GAIL, where the reward is parameterized as a reproducing kernel function, GAIL can be efficiently solved by stochastic first order optimization algorithms, which attain sublinear convergence to a stationary solution. To the best of our knowledge, these are the first results on statistical and computational guarantees of imitation learning with reward/policy function approximation. Numerical experiments are provided to support our analysis.

cs.LG↗

arXiv2019

Integrated turnkey soliton microcombs operated at CMOS frequencies

While soliton microcombs offer the potential for integration of powerful frequency metrology and precision spectroscopy systems, their operation requires complex startup and feedback protocols that necessitate difficult-to-integrate optical and electrical components. Moreover, CMOS-rate microcombs, required in nearly all comb systems, have resisted integration because of their power requirements. Here, a regime for turnkey operation of soliton microcombs co-integrated with a pump laser is demonstrated and theoretically explained. Significantly, a new operating point is shown to appear from which solitons are generated through binary turn-on and turn-off of the pump laser, thereby eliminating all photonic/electronic control circuitry. These features are combined with high-Q $Si_3N_4$ resonators to fully integrate into a butterfly package microcombs with CMOS frequencies as low as 15 GHz, offering compelling advantages for high-volume production.

physics.optics↗

arXiv2019

Towards Understanding the Importance of Shortcut Connections in Residual Networks

Residual Network (ResNet) is undoubtedly a milestone in deep learning. ResNet is equipped with shortcut connections between layers, and exhibits efficient training using simple first order algorithms. Despite of the great empirical success, the reason behind is far from being well understood. In this paper, we study a two-layer non-overlapping convolutional ResNet. Training such a network requires solving a non-convex optimization problem with a spurious local optimum. We show, however, that gradient descent combined with proper normalization, avoids being trapped by the spurious local optimum, and converges to a global optimum in polynomial time, when the weight of the first layer is initialized at 0, and that of the second layer is initialized arbitrarily in a ball. Numerical experiments are provided to support our theory.

cs.LG↗

arXiv2019

Towards Understanding the Importance of Noise in Training Neural Networks

Numerous empirical evidence has corroborated that the noise plays a crucial rule in effective and efficient training of neural networks. The theory behind, however, is still largely unknown. This paper studies this fundamental problem through training a simple two-layer convolutional neural network model. Although training such a network requires solving a nonconvex optimization problem with a spurious local optimum and a global optimum, we prove that perturbed gradient descent and perturbed mini-batch stochastic gradient algorithms in conjunction with noise annealing is guaranteed to converge to a global optimum in polynomial time with arbitrary initialization. This implies that the noise enables the algorithm to efficiently escape from the spurious local optimum. Numerical experiments are provided to support our theory.

cs.LG↗

arXiv2018

Neural Relation Extraction via Inner-Sentence Noise Reduction and Transfer Learning

Extracting relations is critical for knowledge base completion and construction in which distant supervised methods are widely used to extract relational facts automatically with the existing knowledge bases. However, the automatically constructed datasets comprise amounts of low-quality sentences containing noisy words, which is neglected by current distant supervised methods resulting in unacceptable precisions. To mitigate this problem, we propose a novel word-level distant supervised approach for relation extraction. We first build Sub-Tree Parse(STP) to remove noisy words that are irrelevant to relations. Then we construct a neural network inputting the sub-tree while applying the entity-wise attention to identify the important semantic features of relational words in each instance. To make our model more robust against noisy words, we initialize our network with a priori knowledge learned from the relevant task of entity classification by transfer learning. We conduct extensive experiments using the corpora of New York Times(NYT) and Freebase. Experiments show that our approach is effective and improves the area of Precision/Recall(PR) from 0.35 to 0.39 over the state-of-the-art work.

cs.CL↗

arXiv2017

Risk Quantification in Stochastic Simulation under Input Uncertainty

When simulating a complex stochastic system, the behavior of output response depends on input parameters estimated from finite real-world data, and the finiteness of data brings input uncertainty into the system. The quantification of the impact of input uncertainty on output response has been extensively studied. Most of the existing literature focuses on providing inferences on the mean response at the true but unknown input parameter, including point estimation and confidence interval construction. Risk quantification of mean response under input uncertainty often plays an important role in system evaluation and control, because it provides inferences on extreme scenarios of mean response in all possible input models. To the best of our knowledge, it has rarely been systematically studied in the literature. In this paper, first we introduce risk measures of mean response under input uncertainty, and propose a nested Monte Carlo simulation approach to estimate them. Then we develop asymptotical properties such as consistency and asymptotic normality for the proposed nested risk estimators. We further study the associated budget allocation problem for efficient nested risk simulation, and finally use a sharing economy example to illustrate the importance of accessing and controlling risk due to input uncertainty.

q-fin.RM↗