arXiv subjects

Chaoyue Liu

Publications and source records attributed to Chaoyue Liu.

25 records · Page 2Linked to original sources

arXiv2021

Hyper-parameter optimization based on soft actor critic and hierarchical mixture regularization

Hyper-parameter optimization is a crucial problem in machine learning as it aims to achieve the state-of-the-art performance in any model. Great efforts have been made in this field, such as random search, grid search, Bayesian optimization. In this paper, we model hyper-parameter optimization process as a Markov decision process, and tackle it with reinforcement learning. A novel hyper-parameter optimization method based on soft actor critic and hierarchical mixture regularization has been proposed. Experiments show that the proposed method can obtain better hyper-parameters in a shorter time.

cs.LG↗

arXiv2021

Loss landscapes and optimization in over-parameterized non-linear systems and neural networks

The success of deep learning is due, to a large extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks. The purpose of this work is to propose a modern view and a general mathematical framework for loss landscapes and efficient optimization in over-parameterized machine learning models and systems of non-linear equations, a setting that includes over-parameterized deep neural networks. Our starting observation is that optimization problems corresponding to such systems are generally not convex, even locally. We argue that instead they satisfy PL$^*$, a variant of the Polyak-Lojasiewicz condition on most (but not all) of the parameter space, which guarantees both the existence of solutions and efficient optimization by (stochastic) gradient descent (SGD/GD). The PL$^*$ condition of these systems is closely related to the condition number of the tangent kernel associated to a non-linear system showing how a PL$^*$-based non-linear theory parallels classical analyses of over-parameterized linear equations. We show that wide neural networks satisfy the PL$^*$ condition, which explains the (S)GD convergence to a global minimum. Finally we propose a relaxation of the PL$^*$ condition applicable to "almost" over-parameterized systems.

cs.LG↗

arXiv2021

On the linearity of large non-linear models: when and why the tangent kernel is constant

The goal of this work is to shed light on the remarkable phenomenon of transition to linearity of certain neural networks as their width approaches infinity. We show that the transition to linearity of the model and, equivalently, constancy of the (neural) tangent kernel (NTK) result from the scaling properties of the norm of the Hessian matrix of the network as a function of the network width. We present a general framework for understanding the constancy of the tangent kernel via Hessian scaling applicable to the standard classes of neural networks. Our analysis provides a new perspective on the phenomenon of constant tangent kernel, which is different from the widely accepted "lazy training". Furthermore, we show that the transition to linearity is not a general property of wide neural networks and does not hold when the last layer of the network is non-linear. It is also not necessary for successful optimization by gradient descent.

cs.LG↗

arXiv2019

Accelerating SGD with momentum for over-parameterized learning

Nesterov SGD is widely used for training modern neural networks and other machine learning models. Yet, its advantages over SGD have not been theoretically clarified. Indeed, as we show in our paper, both theoretically and empirically, Nesterov SGD with any parameter selection does not in general provide acceleration over ordinary SGD. Furthermore, Nesterov SGD may diverge for step sizes that ensure convergence of ordinary SGD. This is in contrast to the classical results in the deterministic scenario, where the same step size ensures accelerated convergence of the Nesterov's method over optimal gradient descent. To address the non-acceleration issue, we introduce a compensation term to Nesterov SGD. The resulting algorithm, which we call MaSS, converges for same step sizes as SGD. We prove that MaSS obtains an accelerated convergence rates over SGD for any mini-batch size in the linear setting. For full batch, the convergence rate of MaSS matches the well-known accelerated rate of the Nesterov's method. We also analyze the practically important question of the dependence of the convergence rate and optimal hyper-parameters on the mini-batch size, demonstrating three distinct regimes: linear scaling, diminishing returns and saturation. Experimental evaluation of MaSS for several standard architectures of deep networks, including ResNet and convolutional networks, shows improved performance over SGD, Nesterov SGD and Adam.

cs.LG↗

arXiv2019

High-performance silicon-graphene hybrid plasmonic waveguide photodetectors beyond 1.55 μm

A fast silicon-graphene hybrid plasmonic waveguide photodetectors beyond 1.55 μm is proposed and realized by introducing an ultra-thin wide silicon-on-insulator ridge core region with a narrow metal cap. With this novel design, the light absorption in graphene is enhanced while the metal absorption loss is reduced simultaneously, which helps greatly improve the responsivity as well as shorten the absorption region for achieving fast responses. Furthermore, metal-graphene-metal sandwiched electrodes are introduced to reduce the metal-graphene contact resistance, which is also helpful for improving the response speed. When the photodetector operates at 2 μm, the measured 3dB-bandwidth is >20 GHz (which is limited by the experimental setup) while the 3dB-bandwith calculated from the equivalent circuit with the parameters extracted from the measured S11 is as high as ~100 GHz. To the best of our knowledge, it is the first time to report the waveguide photodetector at 2 μm with a 3dB-bandwidth over 20 GHz. Besides, the present photodetectors also work very well at 1.55 μm. The measured responsivity is about 0.4 A/W under a bias voltage of -0.3 V for an optical power of 0.16 mW, while the measured 3dB-bandwidth is over 40 GHz (limited by the test setup) and the 3 dB-bandwidth estimated from the equivalent circuit is also as high as ~100 GHz, which is one of the best results reported for silicon-graphene photodetectors at 1.55 μm.

physics.app-ph↗

arXiv2018

High-speed and high-responsivity hybrid silicon/black-phosphorus waveguide photodetectors at 2 μm

Silicon photonics is being extended from the near-infrared (near-IR) window of 1.3-1.5 μm for optical fiber communications to the mid-infrared (mid-IR) wavelength-band of 2 μm or longer for satisfying the increasing demands in many applications. Mid-IR waveguide photodetectors on silicon have attracted intensive attention as one of the indispensable elements for various photonic systems. Previously high-performance waveguide photodetectors on silicon were realized for the near-IR window of 1.3-1.5 μm by introducing another semiconductor material (e.g., Ge, and III-V compounds) in the active region. Unfortunately, these traditional semiconductor materials do not work well for the wavelength of ~2 μm or longer because the light absorption becomes very weak. As an alternative, two-dimensional materials provide a new and promising option for enabling active photonic devices on silicon. Here black-phosphorus (BP) thin films with optimized medium thicknesses (~40 nm) are introduced as the active material for light absorption and silicon/BP hybrid ridge waveguide photodetectors are demonstrated with a high responsivity at a low bias voltage. And up to 4.0Gbps data transmission is achieved at 2μm.

physics.app-ph↗

arXiv2018

Parametrized Accelerated Methods Free of Condition Number

Analyses of accelerated (momentum-based) gradient descent usually assume bounded condition number to obtain exponential convergence rates. However, in many real problems, e.g., kernel methods or deep neural networks, the condition number, even locally, can be unbounded, unknown or mis-estimated. This poses problems in both implementing and analyzing accelerated algorithms. In this paper, we address this issue by proposing parametrized accelerated methods by considering the condition number as a free parameter. We provide spectral-level analysis for several important accelerated algorithms, obtain explicit expressions and improve worst case convergence rates. Moreover, we show that those algorithm converge exponentially even when the condition number is unknown or mis-estimated.

cs.LG↗