arXiv subjects

Bin Gu

Publications and source records attributed to Bin Gu.

101 records · Page 6Linked to original sources

arXiv2018

Inexact Proximal Gradient Methods for Non-convex and Non-smooth Optimization

In machine learning research, the proximal gradient methods are popular for solving various optimization problems with non-smooth regularization. Inexact proximal gradient methods are extremely important when exactly solving the proximal operator is time-consuming, or the proximal operator does not have an analytic solution. However, existing inexact proximal gradient methods only consider convex problems. The knowledge of inexact proximal gradient methods in the non-convex setting is very limited. % Moreover, for some machine learning models, there is still no proposed solver for exactly solving the proximal operator. To address this challenge, in this paper, we first propose three inexact proximal gradient algorithms, including the basic version and Nesterov's accelerated version. After that, we provide the theoretical analysis to the basic and Nesterov's accelerated versions. The theoretical results show that our inexact proximal gradient algorithms can have the same convergence rates as the ones of exact proximal gradient algorithms in the non-convex setting. Finally, we show the applications of our inexact proximal gradient algorithms on three representative non-convex learning problems. All experimental results confirm the superiority of our new inexact proximal gradient algorithms.

cs.LG↗

arXiv2018

Decoupled Parallel Backpropagation with Convergence Guarantee

Backpropagation algorithm is indispensable for the training of feedforward neural networks. It requires propagating error gradients sequentially from the output layer all the way back to the input layer. The backward locking in backpropagation algorithm constrains us from updating network layers in parallel and fully leveraging the computing resources. Recently, several algorithms have been proposed for breaking the backward locking. However, their performances degrade seriously when networks are deep. In this paper, we propose decoupled parallel backpropagation algorithm for deep learning optimization with convergence guarantee. Firstly, we decouple the backpropagation algorithm using delayed gradients, and show that the backward locking is removed when we split the networks into multiple modules. Then, we utilize decoupled parallel backpropagation in two stochastic methods and prove that our method guarantees convergence to critical points for the non-convex problem. Finally, we perform experiments for training deep convolutional neural networks on benchmark datasets. The experimental results not only confirm our theoretical analysis, but also demonstrate that the proposed method can achieve significant speedup without loss of accuracy.

cs.LG↗

arXiv2018

Small Animal Multivariate Brain Analysis (SAMBA): A High Throughput Pipeline with a Validation Framework

While many neuroscience questions aim to understand the human brain, much current knowledge has been gained using animal models, which replicate genetic, structural, and connectivity aspects of the human brain. While voxel-based analysis (VBA) of preclinical magnetic resonance images is widely-used, a thorough examination of the statistical robustness, stability, and error rates is hindered by high computational demands of processing large arrays, and the many parameters involved. Thus, workflows are often based on intuition or experience, while preclinical validation studies remain scarce. To increase throughput and reproducibility of quantitative small animal brain studies, we have developed a publicly shared, high throughput VBA pipeline in a high-performance computing environment, called SAMBA. The increased computational efficiency allowed large multidimensional arrays to be processed in 1-3 days, a task that previously took ~1 month. To quantify the variability and reliability of preclinical VBA in rodent models, we propose a validation framework consisting of morphological phantoms, and four metrics. This addresses several sources that impact VBA results, including registration and template construction strategies. We have used this framework to inform the VBA workflow parameters in a VBA study for a mouse model of epilepsy. We also present initial efforts towards standardizing small animal neuroimaging data in a similar fashion with human neuroimaging. We conclude that verifying the accuracy of VBA merits attention, and should be the focus of a broader effort within the community. The proposed framework promotes consistent quality assurance of VBA in preclinical neuroimaging; facilitating the creation and communication of robust results.

q-bio.QM↗

arXiv2017

Accelerated Method for Stochastic Composition Optimization with Nonsmooth Regularization

Stochastic composition optimization draws much attention recently and has been successful in many emerging applications of machine learning, statistical analysis, and reinforcement learning. In this paper, we focus on the composition problem with nonsmooth regularization penalty. Previous works either have slow convergence rate or do not provide complete convergence analysis for the general problem. In this paper, we tackle these two issues by proposing a new stochastic composition optimization method for composition problem with nonsmooth regularization penalty. In our method, we apply variance reduction technique to accelerate the speed of convergence. To the best of our knowledge, our method admits the fastest convergence rate for stochastic composition optimization: for strongly convex composition problem, our algorithm is proved to admit linear convergence; for general composition problem, our algorithm significantly improves the state-of-the-art convergence rate from $O(T^{-1/2})$ to $O((n_1+n_2)^{{2}/{3}}T^{-1})$. Finally, we apply our proposed algorithm to portfolio management and policy evaluation in reinforcement learning. Experimental results verify our theoretical analysis.

cs.LG↗

arXiv2016

Zeroth-order Asynchronous Doubly Stochastic Algorithm with Variance Reduction

Zeroth-order (derivative-free) optimization attracts a lot of attention in machine learning, because explicit gradient calculations may be computationally expensive or infeasible. To handle large scale problems both in volume and dimension, recently asynchronous doubly stochastic zeroth-order algorithms were proposed. The convergence rate of existing asynchronous doubly stochastic zeroth order algorithms is $O(\frac{1}{\sqrt{T}})$ (also for the sequential stochastic zeroth-order optimization algorithms). In this paper, we focus on the finite sums of smooth but not necessarily convex functions, and propose an asynchronous doubly stochastic zeroth-order optimization algorithm using the accelerated technology of variance reduction (AsyDSZOVR). Rigorous theoretical analysis show that the convergence rate can be improved from $O(\frac{1}{\sqrt{T}})$ the best result of existing algorithms to $O(\frac{1}{T})$. Also our theoretical results is an improvement to the ones of the sequential stochastic zeroth-order optimization algorithms.

cs.LG↗

arXiv2016

Asynchronous Stochastic Block Coordinate Descent with Variance Reduction

Asynchronous parallel implementations for stochastic optimization have received huge successes in theory and practice recently. Asynchronous implementations with lock-free are more efficient than the one with writing or reading lock. In this paper, we focus on a composite objective function consisting of a smooth convex function $f$ and a block separable convex function, which widely exists in machine learning and computer vision. We propose an asynchronous stochastic block coordinate descent algorithm with the accelerated technology of variance reduction (AsySBCDVR), which are with lock-free in the implementation and analysis. AsySBCDVR is particularly important because it can scale well with the sample size and dimension simultaneously. We prove that AsySBCDVR achieves a linear convergence rate when the function $f$ is with the optimal strong convexity property, and a sublinear rate when $f$ is with the general convexity. More importantly, a near-linear speedup on a parallel system with shared memory can be obtained.

cs.LG↗

arXiv2016

Decoupled Asynchronous Proximal Stochastic Gradient Descent with Variance Reduction

In the era of big data, optimizing large scale machine learning problems becomes a challenging task and draws significant attention. Asynchronous optimization algorithms come out as a promising solution. Recently, decoupled asynchronous proximal stochastic gradient descent (DAP-SGD) is proposed to minimize a composite function. It is claimed to be able to off-loads the computation bottleneck from server to workers by allowing workers to evaluate the proximal operators, therefore, server just need to do element-wise operations. However, it still suffers from slow convergence rate because of the variance of stochastic gradient is nonzero. In this paper, we propose a faster method, decoupled asynchronous proximal stochastic variance reduced gradient descent method (DAP-SVRG). We prove that our method has linear convergence for strongly convex problem. Large-scale experiments are also conducted in this paper, and results demonstrate our theoretical analysis.

cs.LG↗

arXiv2016

On the identification of time interval threshold in the twin-CME scenario

Recently it has been suggested that the "twin-CME" scenario Li.etal2012 may be a very effective mechanism in causing extreme Solar Energetic Particle (SEP) events and in particular Ground Level Enhancement (GLE) events. Ding.etal2013 performed a statistical examination of the twin-CME scenario with a total of $126$ fast and wide western Coronal Mass Ejections (CMEs). They found that CMEs having a preceding CME with a speed $>$ 300 $km/s$ within $9$ hours from the same active region have larger probability of leading to large SEP events than CMEs that do not have preceding CMEs. The choice of $9$ hours being the time lag $τ$ between the preceding CME and the main CME was based on some crude estimates of the decay time of the turbulence downstream of the shock driven by the preceding CME. In this work, we examine this choice. For the $126$ fast wide CMEs examined in Ding.etal2013, we vary the time lag $τ$ from $1$ hour to $24$ hours with an increment of $1$ hour. By considering three quantities whose values depend on the choice of this time lag $τ$, we show that the choice of $13$ hours for $τ$ is more appropriate. Our study confirms our earlier result that twin CMEs are more likely to lead to large SEP events than single fast CMEs. The results shown here are of great relevance to space weather studies.

astro-ph.SR↗

arXiv2016

Seed population in large Solar Energetic Particle events and the twin-CME scenario

It has been recently suggested that large solar energetic particle (SEP) events are often caused by twin CMEs. In the twin-CME scenario, the preceding CME is to provide both an enhanced turbulence level and enhanced seed population at the main CME-driven shock. In this work, we study the effect of the preceding CMEs on the seed population. We examine event-integrated abundance of iron to oxygen ratio (Fe/O) at energies above 25 MeV/nuc for large SEP events in solar cycle 23. We find that the Fe/O ratio (normalized to the reference coronal value of $0.134$) $\leq2.0$ for almost all single-CME events and these events tend to have smaller peak intensities. In comparison, the Fe/O ratio of twin-CME events scatters in a larger range, reaching as high as $8$, suggesting the presence of flare material from perhaps preceding flares. For extremely large SEP events with peak intensity above $1000$ pfu, the Fe/O drop below $2$, indicating that in these extreme events the seed particles are dominated by coronal material than flare material. The Fe/O ratios of Ground level enhancement (GLE) events, all being twin-CME events, scatter in a broad range. For a given Fe/O ratio, GLE events tend to have larger peak intensities than non-GLE events. Using velocity dispersion analysis (VDA), we find that GLE events have lower solar particle release (SPR) heights than non-GLE events, \red{agreeing with earlier results by Reames 2009b.

astro-ph.SR↗

arXiv2013

MDM: A Mode Diagram Modeling Framework

Periodic control systems used in spacecrafts and automotives are usually period-driven and can be decomposed into different modes with each mode representing a system state observed from outside. Such systems may also involve intensive computing in their modes. Despite the fact that such control systems are widely used in the above-mentioned safety-critical embedded domains, there is lack of domain-specific formal modelling languages for such systems in the relevant industry. To address this problem, we propose a formal visual modeling framework called mode diagram as a concise and precise way to specify and analyze such systems. To capture the temporal properties of periodic control systems, we provide, along with mode diagram, a property specification language based on interval logic for the description of concrete temporal requirements the engineers are concerned with. The statistical model checking technique can then be used to verify the mode diagram models against desired properties. To demonstrate the viability of our approach, we have applied our modelling framework to some real life case studies from industry and helped detect two design defects for some spacecraft control systems.

cs.LO↗

arXiv2012

MDM: A Mode Diagram Modeling Framework for Periodic Control Systems

Periodic control systems used in spacecrafts and automotives are usually period-driven and can be decomposed into different modes with each mode representing a system state observed from outside. Such systems may also involve intensive computing in their modes. Despite the fact that such control systems are widely used in the above-mentioned safety-critical embedded domains, there is lack of domain-specific formal modelling languages for such systems in the relevant industry. To address this problem, we propose a formal visual modeling framework called MDM as a concise and precise way to specify and analyze such systems. To capture the temporal properties of periodic control systems, we provide, along with MDM, a property specification language based on interval logic for the description of concrete temporal requirements the engineers are concerned with. The statistical model checking technique can then be used to verify the MDM models against desired properties. To demonstrate the viability of our approach, we have applied our modelling framework to some real life case studies from industry and helped detect two design defects for some spacecraft control systems.

eess.SY↗