arXiv subjects

Xiaoming Yuan

Publications and source records attributed to Xiaoming Yuan.

At least 55 records · Page 3Linked to original sources

arXiv2021

Extensions of ADMM for Separable Convex Optimization Problems with Linear Equality or Inequality Constraints

The alternating direction method of multipliers (ADMM) proposed by Glowinski and Marrocco is a benchmark algorithm for two-block separable convex optimization problems with linear equality constraints. It has been modified, specified, and generalized from various perspectives to tackle more concrete or complicated application problems. Despite its versatility and phenomenal popularity, it remains unknown whether or not the ADMM can be extended to separable convex optimization problems with linear inequality constraints. In this paper, we lay down the foundation of how to extend the ADMM to two-block and multiple-block (more than two blocks) separable convex optimization problems with linear inequality constraints. From a high-level and methodological perspective, we propose a unified framework of algorithmic design and a roadmap for convergence analysis in the context of variational inequalities, based on which it is possible to design a series of concrete ADMM-based algorithms with provable convergence in the prediction-correction structure. The proposed algorithmic framework and roadmap for convergence analysis are eligible to various convex optimization problems with different degrees of separability, in which both linear equality and linear inequality constraints can be included. The analysis is comprehensive yet can be presented by elementary mathematics, and hence generically understandable.

math.OC↗

arXiv2021

A Value-Function-based Interior-point Method for Non-convex Bi-level Optimization

Bi-level optimization model is able to capture a wide range of complex learning tasks with practical interest. Due to the witnessed efficiency in solving bi-level programs, gradient-based methods have gained popularity in the machine learning community. In this work, we propose a new gradient-based solution scheme, namely, the Bi-level Value-Function-based Interior-point Method (BVFIM). Following the main idea of the log-barrier interior-point scheme, we penalize the regularized value function of the lower level problem into the upper level objective. By further solving a sequence of differentiable unconstrained approximation problems, we consequently derive a sequential programming scheme. The numerical advantage of our scheme relies on the fact that, when gradient methods are applied to solve the approximation problem, we successfully avoid computing any expensive Hessian-vector or Jacobian-vector product. We prove the convergence without requiring any convexity assumption on either the upper level or the lower level objective. Experiments demonstrate the efficiency of the proposed BVFIM on non-convex bi-level problems.

math.OC↗

arXiv2021

Ultralow Threshold, Single-Mode InGaAs/GaAs Multi-Quantum Disk Nanowire Lasers

We present single-mode nanowire (NW) lasers with ultralow threshold in the near-infrared spectral range. To ensure the single-mode operation, the NW diameter and length are reduced specifically to minimize the longitudinal and transverse modes of the NW cavity. Increased optical losses and reduced gain volume by the dimension reduction are compensated by excellent NW morphology and InGaAs/GaAs multi-quantum disks. At 5 K, a threshold low as 1.6 μJ/cm2 per pulse is achieved with a resulting quality factor exceeding 6400. By further passivating the NW with an AlGaAs shell to suppress surface non-radiative recombination, single-mode lasing operation is obtained with a threshold of only 48 μJ/cm2 per pulse at room temperature with a high characteristic temperature of 223 K and power output of ~ 0.9 μW. These single-mode, ultralow threshold, high power output NW lasers are promising for the development of near-infrared nanoscale coherent light sources for integrated photonic circuits, sensing, and spectroscopy.

physics.optics↗

arXiv2021

An ADMM-Newton-CNN Numerical Approach to a TV Model for Identifying Discontinuous Diffusion Coefficients in Elliptic Equations: Convex Case with Gradient Observations

Identifying the discontinuous diffusion coefficient in an elliptic equation with observation data of the gradient of the solution is an important nonlinear and ill-posed inverse problem. Models with total variational (TV) regularization have been widely studied for this problem, while the theoretically required nonsmoothness property of the TV regularization and the hidden convexity of the models are usually sacrificed when numerical schemes are considered in the literature. In this paper, we show that the favorable nonsmoothness and convexity properties can be entirely kept if the well-known alternating direction method of multipliers (ADMM) is applied to the TV-regularized models, hence it is meaningful to consider designing numerical schemes based on the ADMM. Moreover, we show that one of the ADMM subproblems can be well solved by the active-set Newton method along with the Schur complement reduction method, and the other one can be efficiently solved by the deep convolutional neural network (CNN). The resulting ADMM-Newton-CNN approach is demonstrated to be easily implementable and very efficient even for higher-dimensional spaces with fine mesh discretization.

math.NA↗

arXiv2021

Bilinear Optimal Control of an Advection-reaction-diffusion System

We consider the bilinear optimal control of an advection-reaction-diffusion system, where the control arises as the velocity field in the advection term. Such a problem is generally challenging from both theoretical analysis and algorithmic design perspectives mainly because the state variable depends nonlinearly on the control variable and an additional divergence-free constraint on the control is coupled together with the state equation. Mathematically, the proof of the existence of optimal solutions is delicate, and up to now, only some results are known for a few special cases where additional restrictions are imposed on the space dimension and the regularity of the control. We prove the existence of optimal controls and derive the first-order optimality conditions in general settings without any extra assumption. Computationally, the well-known conjugate gradient (CG) method can be applied conceptually. However, due to the additional divergence-free constraint on the control variable and the nonlinear relation between the state and control variables, it is challenging to compute the gradient and the optimal stepsize at each CG iteration, and thus nontrivial to implement the CG method. To address these issues, we advocate a fast inner preconditioned CG method to ensure the divergence-free constraint and an efficient inexactness strategy to determine an appropriate stepsize. An easily implementable nested CG method is thus proposed for solving such a complicated problem. For the numerical discretization, we combine finite difference methods for the time discretization and finite element methods for the space discretization. Efficiency of the proposed nested CG method is promisingly validated by the results of some preliminary numerical experiments.

math.OC↗

arXiv2020

FDNAS: Improving Data Privacy and Model Diversity in AutoML

To prevent the leakage of private information while enabling automated machine intelligence, there is an emerging trend to integrate federated learning and Neural Architecture Search (NAS). Although promising as it may seem, the coupling of difficulties from both two tenets makes the algorithm development quite challenging. In particular, how to efficiently search the optimal neural architecture directly from massive non-iid data of clients in a federated manner remains to be a hard nut to crack. To tackle this challenge, in this paper, by leveraging the advances in proxy-less NAS, we propose a Federated Direct Neural Architecture Search (FDNAS) framework that allows hardware-aware NAS from decentralized non-iid data of clients. To further adapt for various data distributions of clients, inspired by meta-learning, a cluster Federated Direct Neural Architecture Search (CFDNAS) framework is proposed to achieve client-aware NAS, in the sense that each client can learn a tailored deep learning model for its particular data distribution. Extensive experiments on real-world non-iid datasets show state-of-the-art accuracy-efficiency trade-offs for various hardware and data distributions of clients. Our codes will be released publicly upon paper acceptance.

cs.LG↗

arXiv2020

A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singleton

In recent years, a variety of gradient-based first-order methods have been developed to solve bi-level optimization problems for learning applications. However, theoretical guarantees of these existing approaches heavily rely on the simplification that for each fixed upper-level variable, the lower-level solution must be a singleton (a.k.a., Lower-Level Singleton, LLS). In this work, we first design a counter-example to illustrate the invalidation of such LLS condition. Then by formulating BLPs from the view point of optimistic bi-level and aggregating hierarchical objective information, we establish Bi-level Descent Aggregation (BDA), a flexible and modularized algorithmic framework for generic bi-level optimization. Theoretically, we derive a new methodology to prove the convergence of BDA without the LLS condition. Our investigations also demonstrate that BDA is indeed compatible to a verify of particular first-order computation modules. Additionally, as an interesting byproduct, we also improve these conventional first-order bi-level schemes (under the LLS simplification). Particularly, we establish their convergences with weaker assumptions. Extensive experiments justify our theoretical results and demonstrate the superiority of the proposed BDA for different tasks, including hyper-parameter optimization and meta learning.

cs.LG↗

arXiv2020

The flare Package for High Dimensional Linear Regression and Precision Matrix Estimation in R

This paper describes an R package named flare, which implements a family of new high dimensional regression methods (LAD Lasso, SQRT Lasso, $\ell_q$ Lasso, and Dantzig selector) and their extensions to sparse precision matrix estimation (TIGER and CLIME). These methods exploit different nonsmooth loss functions to gain modeling flexibility, estimation robustness, and tuning insensitiveness. The developed solver is based on the alternating direction method of multipliers (ADMM). The package flare is coded in double precision C, and called from R by a user-friendly interface. The memory usage is optimized by using the sparse matrix output. The experiments show that flare is efficient and can scale up to large problems.

stat.ML↗

arXiv2020

Implementation of the ADMM to Parabolic Optimal Control Problems with Control Constraints and Beyond

Parabolic optimal control problems with control constraints are generally challenging, from either theoretical analysis or algorithmic design perspectives. Conceptually, the well-known alternating direction method of multipliers (ADMM) can be directly applied to such a problem. An attractive advantage of this direct ADMM application is that the control constraint can be untied from the parabolic PDE constraint; these two inherently different constraints thus can be treated individually in iterations. At each iteration of the ADMM, the main computation is for solving an unconstrained parabolic optimal control problem. Because of its high dimensionality after discretization, the unconstrained parabolic optimal control problem at each iteration can be solved only inexactly by implementing certain numerical scheme internally and thus a two-layer nested iterative scheme is required. It then becomes important to find an easily implementable and efficient inexactness criterion to execute the internal iterations, and to prove the overall convergence rigorously for the resulting two-layer nested iterative scheme. To implement the ADMM efficiently, we propose an inexactness criterion that is independent of the mesh size of the involved discretization, and it can be executed automatically with no need to set empirically perceived constant accuracy a prior. The inexactness criterion turns out to allow us to solve the resulting unconstrained optimal control problems to medium or even low accuracy and thus saves computation significantly, yet convergence of the overall two-layer nested iterative scheme can be still guaranteed rigorously. Efficiency of this ADMM implementation is promisingly validated by preliminary numerical results. Our methodology can also be extended to a range of optimal control problems constrained by other linear PDEs such as elliptic equations and hyperbolic equations.

math.OC↗

arXiv2019

Strong Hot Carrier Effects Observed in a Single Nanowire Heterostructure

We use Transient Rayleigh Scattering to study the thermalization of hot photoexcited carriers in single GaAsSb/InP nanowire heterostructures. By comparing the energy loss rate in single bare GaAsSb nanowires which do not show substantial hot carrier effects with the core-shell nanowires, we show that the presence of an InP shell substantially suppresses the LO phonon emission rate at low temperatures leading to strong hot carrier effects.

cond-mat.mes-hall↗

arXiv2019

Stability and Optimization Error of Stochastic Gradient Descent for Pairwise Learning

In this paper we study the stability and its trade-off with optimization error for stochastic gradient descent (SGD) algorithms in the pairwise learning setting. Pairwise learning refers to a learning task which involves a loss function depending on pairs of instances among which notable examples are bipartite ranking, metric learning, area under ROC (AUC) maximization and minimum error entropy (MEE) principle. Our contribution is twofold. Firstly, we establish the stability results of SGD for pairwise learning in the convex, strongly convex and non-convex settings, from which generalization bounds can be naturally derived. Secondly, we establish the trade-off between stability and optimization error of SGD algorithms for pairwise learning. This is achieved by lower-bounding the sum of stability and optimization error by the minimax statistical error over a prescribed class of pairwise loss functions. From this fundamental trade-off, we obtain lower bounds for the optimization error of SGD algorithms and the excess expected risk over a class of pairwise losses. In addition, we illustrate our stability results by giving some specific examples of AUC maximization, metric learning and MEE.

cs.LG↗

arXiv2018

Perturbation techniques for convergence analysis of proximal gradient method and other first-order algorithms via variational analysis

We develop new perturbation techniques for conducting convergence analysis of various first-order algorithms for a class of nonsmooth optimization problems. We consider the iteration scheme of an algorithm to construct a perturbed stationary point set-valued map, and define the perturbing parameter by the difference of two consecutive iterates. Then, we show that the calmness condition of the induced set-valued map, together with a local version of the proper separation of stationary value condition, is a sufficient condition to ensure the linear convergence of the algorithm. The equivalence of the calmness condition to the one for the canonically perturbed stationary point set-valued map is proved, and this equivalence allows us to derive some sufficient conditions for calmness by using some recent developments in variational analysis. These sufficient conditions are different from existing results (especially, those error-bound-based ones) in that they can be easily verified for many concrete application models. Our analysis is focused on the fundamental proximal gradient (PG) method, and it enables us to show that any accumulation of the sequence generated by the PG method must be a stationary point in terms of the proximal subdifferential, instead of the limiting subdifferential. This result finds the surprising fact that the solution quality found by the PG method is in general superior. Our analysis also leads to some improvement for the linear convergence results of the PG method in the convex case. The new perturbation technique can be conveniently used to derive linear rate convergence of a number of other first-order methods including the well-known alternating direction method of multipliers and primal-dual hybrid gradient method, under mild assumptions.

math.OC↗

arXiv2018

Generalized Short Circuit Ratio for Multi Power Electronic based Devices Infeed Systems: Defi-nition and Theoretical Analysis

Short circuit ratio (SCR) is widely applied to analyze the strength of AC system and the small signal stability for single power elec-tronic based devices infeed systems (SPEISs). However, there still lacking the theory of short circuit ratio applicable for multi power electronic based devices infeed systems (MPEIS), as the complex coupling among multi power electronic devices (PEDs) leads to difficulties in stability analysis. In this regard, this paper firstly proposes a concept named generalized short circuit ratio (gSCR) to measure the strength of connected AC grid in a multi-infeed system from the small signal stability point of view. Generally, the gSCR is physically and mathematically extended from conven-tional SCR by decomposing the multi-infeed system into n inde-pendent single infeed systems. Then the operation gSCR (OgSCR) is proposed based on gSCR in order to take the variation of op-eration point into consideration. The participation factors and sensitivity are analyzed as well. Finally, simulations are conducted to demonstrate the rationality and effectiveness of the defined gSCR and OgSCR.

eess.SY↗

arXiv2017

Adaptive Consensus ADMM for Distributed Optimization

The alternating direction method of multipliers (ADMM) is commonly used for distributed model fitting problems, but its performance and reliability depend strongly on user-defined penalty parameters. We study distributed ADMM methods that boost performance by using different fine-tuned algorithm parameters on each worker node. We present a O(1/k) convergence rate for adaptive ADMM methods with node-specific parameters, and propose adaptive consensus ADMM (ACADMM), which automatically tunes parameters without user oversight.

cs.LG↗

arXiv2017

Adaptive Relaxed ADMM: Convergence Theory and Practical Implementation

Many modern computer vision and machine learning applications rely on solving difficult optimization problems that involve non-differentiable objective functions and constraints. The alternating direction method of multipliers (ADMM) is a widely used approach to solve such problems. Relaxed ADMM is a generalization of ADMM that often achieves better performance, but its efficiency depends strongly on algorithm parameters that must be chosen by an expert user. We propose an adaptive method that automatically tunes the key algorithm parameters to achieve optimal performance without user oversight. Inspired by recent work on adaptivity, the proposed adaptive relaxed ADMM (ARADMM) is derived by assuming a Barzilai-Borwein style linear gradient. A detailed convergence analysis of ARADMM is provided, and numerical results on several applications demonstrate fast practical convergence.

cs.CV↗

arXiv2017

IEEE 802.15.6-based Prototype System for WBAN: Design and Implementation

Various current trends such as ever growing population and accelerated aging effects have effectively promoted the growth of Wireless Body Area Network (WBAN). Being the specialized standard of WBAN, the burgeoning IEEE 802.15.6 is on its way to refinement and perfection. Since experiments are the foundation of research in WBAN that reproduce system operating patterns, yet existing platforms in WBAN are mainly simulations that could not highly conform to reality with strict requirements to some degree. Thus, a reliable and efficient system based on the IEEE 802.15.6, named WBAN prototype system, is proposed and implemented in this paper. The prototype system is ascertained to be valid, authentic and efficient via analyses of scenario tests. Running in circumstances much closer to reality, the system obtains experimental results that meet rigid requirements preferably. Furthermore, based on the valid platform we created, optimizations of two parameters in the IEEE 802.15.6 by actual measurements are derived. We obtain an optimum number of transmission retries and relationship of payload length within 30 with FER, making a trade-off between waste of resource and high packet loss rate.

cs.NI↗

arXiv2016

Convergence analysis of the direct extension of ADMM for multiple-block separable convex minimization

Recently, the alternating direction method of multipliers (ADMM) has found many efficient applications in various areas; and it has been shown that the convergence is not guaranteed when it is directly extended to the multiple-block case of separable convex minimization problems where there are $m\ge 3$ functions without coupled variables in the objective. This fact has given great impetus to investigate various conditions on both the model and the algorithm's parameter that can ensure the convergence of the direct extension of ADMM (abbreviated as "e-ADMM"). Despite some results under very strong conditions (e.g., at least $(m-1)$ functions should be strongly convex) that are applicable to the generic case with a general $m$, some others concentrate on the special case of $m=3$ under the relatively milder condition that only one function is assumed to be strongly convex. We focus on extending the convergence analysis from the case of $m=3$ to the more general case of $m\ge3$. That is, we show the convergence of e-ADMM for the case of $m\ge 3$ with the assumption of only $(m-2)$ functions being strongly convex; and establish its convergence rates in different scenarios such as the worst-case convergence rates measured by iteration complexity and the asymptotically linear convergence rate under stronger assumptions. Thus the convergence of e-ADMM for the general case of $m\ge 4$ is proved; this result seems to be still unknown even though it is intuitive given the known result of the case of $m=3$. Even for the special case of $m=3$, our convergence results turn out to be more general than the exiting results that are derived specifically for the case of $m=3$.

math.OC↗

arXiv2016

On the Optimal Linear Convergence Rate of a Generalized Proximal Point Algorithm

The proximal point algorithm (PPA) has been well studied in the literature. In particular, its linear convergence rate has been studied by Rockafellar in 1976 under certain condition. We consider a generalized PPA in the generic setting of finding a zero point of a maximal monotone operator, and show that the condition proposed by Rockafellar can also sufficiently ensure the linear convergence rate for this generalized PPA. Indeed we show that these linear convergence rates are optimal. Both the exact and inexact versions of this generalized PPA are discussed. The motivation to consider this generalized PPA is that it includes as special cases the relaxed versions of some splitting methods that are originated from PPA. Thus, linear convergence results of this generalized PPA can be used to better understand the convergence of some widely used algorithms in the literature. We focus on the particular convex minimization context and specify Rockafellar's condition to see how to ensure the linear convergence rate for some efficient numerical schemes, including the classical augmented Lagrangian method proposed by Hensen and Powell in 1969 and its relaxed version, the original alternating direction method of multipliers (ADMM) by Glowinski and Marrocco in 1975 and its relaxed version (i.e., the generalized ADMM by Eckstein and Bertsekas in 1992). Some refined conditions weaker than existing ones are proposed in these particular contexts.

math.OC↗