arXiv subjects

Yi Zhou

Publications and source records attributed to Yi Zhou.

At least 433 records · Page 24Linked to original sources

arXiv2020

Boosting One-Point Derivative-Free Online Optimization via Residual Feedback

Zeroth-order optimization (ZO) typically relies on two-point feedback to estimate the unknown gradient of the objective function. Nevertheless, two-point feedback can not be used for online optimization of time-varying objective functions, where only a single query of the function value is possible at each time step. In this work, we propose a new one-point feedback method for online optimization that estimates the objective function gradient using the residual between two feedback points at consecutive time instants. Moreover, we develop regret bounds for ZO with residual feedback for both convex and nonconvex online optimization problems. Specifically, for both deterministic and stochastic problems and for both Lipschitz and smooth objective functions, we show that using residual feedback can produce gradient estimates with much smaller variance compared to conventional one-point feedback methods. As a result, our regret bounds are much tighter compared to existing regret bounds for ZO with conventional one-point feedback, which suggests that ZO with residual feedback can better track the optimizer of online optimization problems. Additionally, our regret bounds rely on weaker assumptions than those used in conventional one-point feedback methods. Numerical experiments show that ZO with residual feedback significantly outperforms existing one-point feedback methods also in practice.

cs.LG↗

arXiv2020

Some generalizations of the DDVV and BW inequalities

In this paper we generalize the known DDVV-type inequalities for real (skew-)symmetric and complex (skew-)Hermitian matrices to arbitrary real, complex and quaternionic matrices. Inspired by the Erdős-Mordell inequality, we establish the DDVV-type inequalities for matrices in the subspaces spanned by a Clifford system or a Clifford algebra. We also generalize the Böttcher-Wenzel inequality to quaternionic matrices.

math.DG↗

arXiv2020

Quality Assurance Test of Silicon Photomultipliers and Electronic Boards for STAR Event Plane Detector

The event plane detector (EPD), installed in the Solenoid Tracker at the Relativistic Heavy-Ion Collider located at the Brookhaven National Laboratory is a plastic scintillator-based device that measures the reaction centrality and event plane in the forward region of the relativistic heavy-ion collisions. We used silicon photomultiplier (SiPM) arrays to detect the photons produced in the scintillator via the fiber connection. Signals from the SiPM arrays were amplified by the front-end electronic (FEE) board, and sent to the analog-to-digital converter (ADC) boards for further processing via the receiver(RX) board. The full EPD system consisted of 24 super-sectors (SSs); each SS was equipped with two SiPM boards, two FEE boards and two RX boards, and they corresponded to 744 readout channels. All these boards were mass produced at the University of Science and Technology of China, with a dedicated quality assurance (QA) procedures applied to identify any problems before deployment. This article describes the details of the QA method and the related test system. The QA test results are presented along with the discussions.

physics.ins-det↗

arXiv2020

Contrastive Weight Regularization for Large Minibatch SGD

The minibatch stochastic gradient descent method (SGD) is widely applied in deep learning due to its efficiency and scalability that enable training deep networks with a large volume of data. Particularly in the distributed setting, SGD is usually applied with large batch size. However, as opposed to small-batch SGD, neural network models trained with large-batch SGD can hardly generalize well, i.e., the validation accuracy is low. In this work, we introduce a novel regularization technique, namely distinctive regularization (DReg), which replicates a certain layer of the deep network and encourages the parameters of both layers to be diverse. The DReg technique introduces very little computation overhead. Moreover, we empirically show that optimizing the neural network with DReg using large-batch SGD achieves a significant boost in the convergence and improved generalization performance. We also demonstrate that DReg can boost the convergence of large-batch SGD with momentum. We believe that DReg can be used as a simple regularization trick to accelerate large-batch training in deep learning.

cs.LG↗

arXiv2020

Observation of two-dimensional superconductivity at the LaAlO3/KTaO3(110) heterointerface

We report on the observation of a Tc ~0.9 K superconductivity at the interface between LaAlO3 film and the 5d transition-metal oxide KTaO3(110) single crystal. The interface shows a large anisotropy of the upper critical field, and its superconducting transition is consistent with a Berezinskii-Kosterlitz-Thouless transition. Both facts suggest that the superconductivity is two-dimensional (2D) in nature. The carrier density measured at 5 K is ~7 time 10^13 cm-2. The superconducting layer thickness and coherence length are estimated to be ~8 and ~30 nm, respectively. Our result provides a new platform for the study of 2D superconductivity at oxide interfaces.

cond-mat.supr-con↗

arXiv2020

Neural Network Training Techniques Regularize Optimization Trajectory: An Empirical Study

Modern deep neural network (DNN) trainings utilize various training techniques, e.g., nonlinear activation functions, batch normalization, skip-connections, etc. Despite their effectiveness, it is still mysterious how they help accelerate DNN trainings in practice. In this paper, we provide an empirical study of the regularization effect of these training techniques on DNN optimization. Specifically, we find that the optimization trajectories of successful DNN trainings consistently obey a certain regularity principle that regularizes the model update direction to be aligned with the trajectory direction. Theoretically, we show that such a regularity principle leads to a convergence guarantee in nonconvex optimization and the convergence rate depends on a regularization parameter. Empirically, we find that DNN trainings that apply the training techniques achieve a fast convergence and obey the regularity principle with a large regularization parameter, implying that the model updates are well aligned with the trajectory. On the other hand, DNN trainings without the training techniques have slow convergence and obey the regularity principle with a small regularization parameter, implying that the model updates are not well aligned with the trajectory. Therefore, different training techniques regularize the model update direction via the regularity principle to facilitate the convergence.

cs.LG↗

arXiv2020

DR-GAN: Conditional Generative Adversarial Network for Fine-Grained Lesion Synthesis on Diabetic Retinopathy Images

Diabetic retinopathy (DR) is a complication of diabetes that severely affects eyes. It can be graded into five levels of severity according to international protocol. However, optimizing a grading model to have strong generalizability requires a large amount of balanced training data, which is difficult to collect particularly for the high severity levels. Typical data augmentation methods, including random flipping and rotation, cannot generate data with high diversity. In this paper, we propose a diabetic retinopathy generative adversarial network (DR-GAN) to synthesize high-resolution fundus images which can be manipulated with arbitrary grading and lesion information. Thus, large-scale generated data can be used for more meaningful augmentation to train a DR grading and lesion segmentation model. The proposed retina generator is conditioned on the structural and lesion masks, as well as adaptive grading vectors sampled from the latent grading space, which can be adopted to control the synthesized grading severity. Moreover, a multi-scale spatial and channel attention module is devised to improve the generation ability to synthesize details. Multi-scale discriminators are designed to operate from large to small receptive fields, and joint adversarial losses are adopted to optimize the whole network in an end-to-end manner. With extensive experiments evaluated on the EyePACS dataset connected to Kaggle, as well as the FGADR dataset, we validate the effectiveness of our method, which can both synthesize highly realistic (1280 x 1280) controllable fundus images and contribute to the DR grading task.

eess.IV↗

arXiv2020

A Benchmark for Studying Diabetic Retinopathy: Segmentation, Grading, and Transferability

People with diabetes are at risk of developing an eye disease called diabetic retinopathy (DR). This disease occurs when high blood glucose levels cause damage to blood vessels in the retina. Computer-aided DR diagnosis is a promising tool for early detection of DR and severity grading, due to the great success of deep learning. However, most current DR diagnosis systems do not achieve satisfactory performance or interpretability for ophthalmologists, due to the lack of training data with consistent and fine-grained annotations. To address this problem, we construct a large fine-grained annotated DR dataset containing 2,842 images (FGADR). This dataset has 1,842 images with pixel-level DR-related lesion annotations, and 1,000 images with image-level labels graded by six board-certified ophthalmologists with intra-rater consistency. The proposed dataset will enable extensive studies on DR diagnosis. We set up three benchmark tasks for evaluation: 1. DR lesion segmentation; 2. DR grading by joint classification and segmentation; 3. Transfer learning for ocular multi-disease identification. Moreover, a novel inductive transfer learning method is introduced for the third task. Extensive experiments using different state-of-the-art methods are conducted on our FGADR dataset, which can serve as baselines for future research.

cs.CV↗

arXiv2020

Fully Convolutional Mesh Autoencoder using Efficient Spatially Varying Kernels

Learning latent representations of registered meshes is useful for many 3D tasks. Techniques have recently shifted to neural mesh autoencoders. Although they demonstrate higher precision than traditional methods, they remain unable to capture fine-grained deformations. Furthermore, these methods can only be applied to a template-specific surface mesh, and is not applicable to more general meshes, like tetrahedrons and non-manifold meshes. While more general graph convolution methods can be employed, they lack performance in reconstruction precision and require higher memory usage. In this paper, we propose a non-template-specific fully convolutional mesh autoencoder for arbitrary registered mesh data. It is enabled by our novel convolution and (un)pooling operators learned with globally shared weights and locally varying coefficients which can efficiently capture the spatially varying contents presented by irregular mesh connections. Our model outperforms state-of-the-art methods on reconstruction accuracy. In addition, the latent codes of our network are fully localized thanks to the fully convolutional structure, and thus have much higher interpolation capability than many traditional 3D mesh generation models.

cs.CV↗

arXiv2020

Normalization Techniques in Training DNNs: Methodology, Analysis and Application

Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs), and have successfully been used in various applications. This paper reviews and comments on the past, present and future of normalization methods in the context of DNN training. We provide a unified picture of the main motivation behind different approaches from the perspective of optimization, and present a taxonomy for understanding the similarities and differences between them. Specifically, we decompose the pipeline of the most representative normalizing activation methods into three components: the normalization area partitioning, normalization operation and normalization representation recovery. In doing so, we provide insight for designing new normalization technique. Finally, we discuss the current progress in understanding normalization methods, and provide a comprehensive review of the applications of normalization for particular tasks, in which it can effectively solve the key issues.

cs.LG↗

arXiv2020

FedCluster: Boosting the Convergence of Federated Learning via Cluster-Cycling

We develop FedCluster--a novel federated learning framework with improved optimization efficiency, and investigate its theoretical convergence properties. The FedCluster groups the devices into multiple clusters that perform federated learning cyclically in each learning round. Therefore, each learning round of FedCluster consists of multiple cycles of meta-update that boost the overall convergence. In nonconvex optimization, we show that FedCluster with the devices implementing the local {stochastic gradient descent (SGD)} algorithm achieves a faster convergence rate than the conventional {federated averaging (FedAvg)} algorithm in the presence of device-level data heterogeneity. We conduct experiments on deep learning applications and demonstrate that FedCluster converges significantly faster than the conventional federated learning under diverse levels of device-level data heterogeneity for a variety of local optimizers.

cs.LG↗

arXiv2020

Electric field control of disorder-tunable superconductivity and the emergence of quantum metal at an oxide interface

We report on an extraordinary field effect of the superconducting LaAlO3/KTaO3(111) interface with Tc ~2 K. By applying a gate voltage (VG) across KTaO3, the interface can be continuously tuned from superconducting into insulating states, yielding a dome-shaped Tc-VG dependence. The electric gating has only a minor effect on carrier density as evidenced in the Hall-effect measurement, while it changes spatial profile of the carriers in the interface, hence the carrier's disorder level significantly. As temperature is decreased, the resistance saturates at lowest temperature in both superconducting and insulating sides, despite an initial dramatic dropping or increasing, which suggests an emergence of quantum metallic state associated with failed superconductor and/or fragile insulator. A VG-modulation of the magnetic-field-driven superconductor to insulator quantum phase transition reveals a non-universal criticality.

cond-mat.supr-con↗

arXiv2020

Exploring the Hierarchy in Relation Labels for Scene Graph Generation

By assigning each relationship a single label, current approaches formulate the relationship detection as a classification problem. Under this formulation, predicate categories are treated as completely different classes. However, different from the object labels where different classes have explicit boundaries, predicates usually have overlaps in their semantic meanings. For example, sit\_on and stand\_on have common meanings in vertical relationships but different details of how these two objects are vertically placed. In order to leverage the inherent structures of the predicate categories, we propose to first build the language hierarchy and then utilize the Hierarchy Guided Feature Learning (HGFL) strategy to learn better region features of both the coarse-grained level and the fine-grained level. Besides, we also propose the Hierarchy Guided Module (HGM) to utilize the coarse-grained level to guide the learning of fine-grained level features. Experiments show that the proposed simple yet effective method can improve several state-of-the-art baselines by a large margin (up to $33\%$ relative gain) in terms of Recall@50 on the task of Scene Graph Generation in different datasets.

cs.CV↗

arXiv2020

Small-floating Target Detection in Sea Clutter via Visual Feature Classifying in the Time-Doppler Spectra

It is challenging to detect small-floating object in the sea clutter for a surface radar. In this paper, we have observed that the backscatters from the target brake the continuity of the underlying motion of the sea surface in the time-Doppler spectra (TDS) images. Following this visual clue, we exploit the local binary pattern (LBP) to measure the variations of texture in the TDS images. It is shown that the radar returns containing target and those only having clutter are separable in the feature space of LBP. An unsupervised one-class support vector machine (SVM) is then utilized to detect the deviation of the LBP histogram of the clutter. The outiler of the detector is classified as the target. In the real-life IPIX radar data sets, our visual feature based detector shows favorable detection rate compared to other three existing approaches.

eess.SP↗

arXiv2020

High-speed InAs/GaSb Mid-Wave Infrared Interband Cascade Photodetector at Room Temperature

High speed mid-wave infrared (MWIR) photodetectors have important applications in the emerging areas such high-precision frequency comb spectroscopy and light detection and ranging (LIDAR). In this work, we report a high-speed room-temperature mid-wave infrared interband cascade photodetector (ICIP) based on a type-II InAs/GaSb superlattice. The devices show an optical cut-off wavelength around 5um and a 3-dB bandwidth up to 7.04 GHz. The relatively low dark current density around 9.39 x 10-2 A/cm2 under -0.1 V is also demonstrated at 300 K. These results validate the advantages of ICIPs to achieve both high-frequency operation and low noise at room temperature. Limitations on the high-speed performance of the detector are also discussed based on the S-parameter analysis and other RF performance measurement.

physics.app-ph↗

arXiv2020

Accelerating Power Methods for Higher-order Markov Chains

Higher-order Markov chains play a very important role in many fields, ranging from multilinear PageRank to financial modeling. In this paper, we propose three accelerated higher-order power methods for computing the limiting probability distribution of higher-order Markov chains, namely higher-order power method with momentum and higher-order quadratic extrapolation method. The convergence results are established, and numerical experiments are reported to show the efficiency of the proposed algorithms. In particular, the non-parametric quadratic extrapolation method is very competitive, and outperforms state-of-the-art competitions.

math.OC↗

arXiv2020

Learning to Generate Diverse Dance Motions with Transformer

With the ongoing pandemic, virtual concerts and live events using digitized performances of musicians are getting traction on massive multiplayer online worlds. However, well choreographed dance movements are extremely complex to animate and would involve an expensive and tedious production process. In addition to the use of complex motion capture systems, it typically requires a collaborative effort between animators, dancers, and choreographers. We introduce a complete system for dance motion synthesis, which can generate complex and highly diverse dance sequences given an input music sequence. As motion capture data is limited for the range of dance motions and styles, we introduce a massive dance motion data set that is created from YouTube videos. We also present a novel two-stream motion transformer generative model, which can generate motion sequences with high flexibility. We also introduce new evaluation metrics for the quality of synthesized dance motions, and demonstrate that our system can outperform state-of-the-art methods. Our system provides high-quality animations suitable for large crowds for virtual concerts and can also be used as reference for professional animation pipelines. Most importantly, we show that vast online videos can be effective in training dance motion models.

cs.CV↗

arXiv2020

Spatio-temporal Attention Model for Tactile Texture Recognition

Recently, tactile sensing has attracted great interest in robotics, especially for facilitating exploration of unstructured environments and effective manipulation. A detailed understanding of the surface textures via tactile sensing is essential for many of these tasks. Previous works on texture recognition using camera based tactile sensors have been limited to treating all regions in one tactile image or all samples in one tactile sequence equally, which includes much irrelevant or redundant information. In this paper, we propose a novel Spatio-Temporal Attention Model (STAM) for tactile texture recognition, which is the very first of its kind to our best knowledge. The proposed STAM pays attention to both spatial focus of each single tactile texture and the temporal correlation of a tactile sequence. In the experiments to discriminate 100 different fabric textures, the spatially and temporally selective attention has resulted in a significant improvement of the recognition accuracy, by up to 18.8%, compared to the non-attention based models. Specifically, after introducing noisy data that is collected before the contact happens, our proposed STAM can learn the salient features efficiently and the accuracy can increase by 15.23% on average compared with the CNN based baseline approach. The improved tactile texture perception can be applied to facilitate robot tasks like grasping and manipulation.

cs.RO↗