arXiv subjects

Liwei Zhang

Publications and source records attributed to Liwei Zhang.

At least 55 records · Page 3Linked to original sources

arXiv2022

Regrets of Proximal Method of Multipliers for Online Non-convex Optimization with Long Term Constraints

The online optimization problem with non-convex loss functions over a closed convex set, coupled with a set of inequality (possibly non-convex) constraints is a challenging online learning problem. A proximal method of multipliers with quadratic approximations (named as OPMM) is presented to solve this online non-convex optimization with long term constraints. Regrets of the violation of Karush-Kuhn-Tucker conditions of OPMM for solving online non-convex optimization problems are analyzed. Under mild conditions, it is shown that this algorithm exhibits ${\cO}(T^{-1/8})$ Lagrangian gradient violation regret, ${\cO}(T^{-1/8})$ constraint violation regret and ${\cO}(T^{-1/4})$ complementarity residual regret if parameters in the algorithm are properly chosen, where $T$ denotes the number of time periods. For the case that the objective is a convex quadratic function, we demonstrate that the regret of the objective reduction can be established even the feasible set is non-convex. For the case when the constraint functions are convex, if the solution of the subproblem in OPMM is obtained by solving its dual, OPMM is proved to be an implementable projection method for solving the online non-convex optimization problem.

math.OC↗

arXiv2022

Optimality Conditions and Numerical Algorithms for A Class of Linearly Constrained Minimax Optimization Problems

It is well known that there have been many numerical algorithms for solving nonsmooth minimax problems, numerical algorithms for nonsmooth minimax problems with joint linear constraints are very rare. This paper aims to discuss optimality conditions and develop practical numerical algorithms for minimax problems with joint linear constraints. First of all, we use the properties of proximal mapping and KKT system to establish optimality conditions. Secondly, we propose a framework of alternating coordinate algorithm for the minimax problem and analyze its convergence properties. Thirdly, we develop a proximal gradient multi-step ascent decent method (PGmsAD) as a numerical algorithm and demonstrate that the method can find an $ε$-stationary point for this kind of nonsmooth nonconvex-nonconcave problem in ${\cal O}(ε^{-2}\logε^{-1})$ iterations. Finally, we apply PGmsAD to generalized absolute value equations, generalized linear projection equations and linear regression problems and report the efficiency of PGmsAD on large-scale optimization.

math.OC↗

arXiv2021

The Augmented Lagrangian Method Can Approximately Solve Convex Optimization with Least Constraint Violation

There are many important practical optimization problems whose feasible regions are not known to be nonempty or not, and optimizers of the objective function with the least constraint violation prefer to be found. A natural way for dealing with these problems is to extend the nonlinear optimization problem as the one optimizing the objective function over the set of points with the least constraint violation. This leads to the study of the shifted problem. This paper focuses on the constrained convex optimization problem. The sufficient condition for the closedness of the set of feasible shifts is presented and the continuity properties of the optimal value function and the solution mapping for the shifted problem are studied. Properties of the conjugate dual of the shifted problem are discussed through the relations between the dual function and the optimal value function. The solvability of the dual of the optimization problem with the least constraint violation is investigated. It is shown that, if the least violated shift is in the domain of the subdifferential of the optimal value function, then this dual problem has an unbounded solution set. Under this condition, the optimality conditions for the problem with the least constraint violation are established in term of the augmented Lagrangian. It is shown that the augmented Lagrangian method has the properties that the sequence of shifts converges to the least violated shift and the sequence of multipliers is unbounded. Moreover, it is proved that the augmented Lagrangian method is able to find an approximate solution to the problem with the least constraint violation.

math.OC↗

arXiv2021

Stability for Constrained Minimax Optimization

Minimax optimization problems are an important class of optimization problems arising from both modern machine learning and from traditional research areas. We focus on the stability of constrained minimax optimization problems based on the notion of local minimax point by Dai and Zhang (2020). Firstly, we extend the classical Jacobian uniqueness conditions of nonlinear programming to the constrained minimax problem and prove that this set of properties is stable with respect to small $C^2$ perturbation. Secondly, we provide a set of conditions, called Property A, which does not require the strict complementarity condition for the upper level constraints. Finally, we prove that Property A is a sufficient condition for the strong regularity of the Kurash-Kuhn-Tucker (KKT) system at the KKT point, and it is also a sufficient condition for the local Lipschitzian homeomorphism of the Kojima mapping near the KKT point.

math.OC↗

arXiv2021

Local invertibility and sensitivity of atomic structure-feature mappings

The increasingly common applications of machine-learning schemes to atomic-scale simulations have triggered efforts to better understand the mathematical properties of the mapping between the Cartesian coordinates of the atoms and the variety of representations that can be used to convert them into a finite set of symmetric descriptors or features. Here, we analyze the sensitivity of the mapping to atomic displacements, showing that the combination of symmetry and smoothness leads to mappings that have singular points at which the Jacobian has one or more null singular values (besides those corresponding to infinitesimal translations and rotations). This is in fact desirable, because it enforces physical symmetry constraints on the values predicted by regression models constructed using such representations. However, besides these symmetry-induced singularities, there are also spurious singular points, that we find to be linked to the incompleteness of the mapping, i.e. the fact that, for certain classes of representations, structurally distinct configurations are not guaranteed to be mapped onto different feature vectors. Additional singularities can be introduced by a too aggressive truncation of the infinite basis set that is used to discretize the representations.

physics.chem-ph↗

arXiv2021

The Truth about Power Laws: Theory and Reality

Consensus about the universality of the power law feature in complex networks is experiencing profound challenges. To shine fresh light on this controversy, we propose a generic theoretical framework in order to examine the power law property. First, we study a class of birth-and-death networks that is ubiquitous in the real world, and calculate its degree distributions. Our results show that the tails of its degree distributions exhibits a distinct power law feature, providing robust theoretical support for the ubiquity of the power law feature. Second, we suggest that in the real world two important factors, network size and node disappearance probability, point to the existence of the power law feature in the observed networks. As network size reduces, or as the probability of node disappearance increases, then the power law feature becomes increasingly difficult to observe. Finally, we suggest that an effective way of detecting the power law property is to observe the asymptotic (limiting) behaviour of the degree distribution within its effective intervals.

physics.soc-ph↗

arXiv2021

Modified Poisson-Nernst-Planck model with Coulomb and hard-sphere correlations

We develop a modified Poisson-Nernst-Planck model which includes both the long-range Coulomb and short-range hard-sphere correlations in its free energy functional such that the model can accurately describe the ion transport in complex environment and under a nanoscale confinement. The Coulomb correlation including the dielectric polarization is treated by solving a generalized Debye-Hückel equation which is a Green's function equation with the correlation energy of a test ion described by the self Green's function. The hard-sphere correlation is modeled through the modified fundamental measure theory. The resulting model is available for problems beyond the mean-field theory such as problems with variable dielectric media, multivalent ions, and strong surface charge density. We solve the generalized Debye-Hückel equation by the Wentzel-Kramers-Brillouin approximation, and study the electrolytes between two parallel dielectric surfaces. In comparison to other modified models, the new model is shown more accurate in agreement with particle-based simulations and capturing the physical properties of ionic structures near interfaces.

physics.comp-ph↗

arXiv2021

Common kernel-smoothed proper orthogonal decomposition (CKSPOD): An efficient reduced-order model for emulation of spatiotemporally evolving flow dynamics

In the present study, we propose a new surrogate model, called common kernel-smoothed proper orthogonal decomposition (CKSPOD), to efficiently emulate the spatiotemporal evolution of fluid flow dynamics. The proposed surrogate model integrates and extends recent developments in Gaussian process learning, high-fidelity simulations, projection-based model reduction, uncertainty quantification, and experimental design, rendering a systematic, multidisciplinary framework. The novelty of the CKSPOD emulation lies in the construction of a common Gram matrix, which results from the Hadamard product of Gram matrices of all observed design settings. The Gram matrix is a spatially averaged temporal correlation matrix and contains the temporal dynamics of the corresponding sampling point. The common Gram matrix synthesizes the temporal dynamics by transferring POD modes into spatial functions at each observed design setting, which remedies the phase-difference issue encountered in the kernel-smoothed POD (KSPOD) emulation, a recent fluid flow emulator proposed in Chang et al. (2020). The CKSPOD methodology is demonstrated through a model study of flow dynamics of swirl injectors with three design parameters. A total of 30 training design settings and 8 validation design settings are included. Both qualitative and quantitative results show that the CKSPOD emulation outperforms the KSPOD emulation for all validation cases, and is capable of capturing small-scale wave structures on the liquid-film surface faithfully. The turbulent kinetic energy prediction using CKSPOD reveals lower predictive uncertainty than KSPOD, thereby allowing for more accurate and precise flow predictions. The turnaround time of the CKSPOD emulation is about 5 orders of magnitude faster than the corresponding high-fidelity simulation, which enables an efficient and scalable framework for design exploration and optimization.

physics.flu-dyn↗

arXiv2020

Majorized Semi-proximal Alternating Coordinate Method for Nonsmooth Convex-Concave Minimax Optimization

Minimax optimization problems are an important class of optimization problems arising from modern machine learning and traditional research areas. While there have been many numerical algorithms for solving smooth convex-concave minimax problems, numerical algorithms for nonsmooth convex-concave minimax problems are very rare. This paper aims to develop an efficient numerical algorithm for a structured nonsmooth convex-concave minimax problem. A majorized semi-proximal alternating coordinate method (mspACM) is proposed, in which a majorized quadratic convex-concave function is adopted for approximating the smooth part of the objective function and semi-proximal terms are added in each subproblem. This construction enables the subproblems at each iteration are solvable and even easily solved when the semiproximal terms are cleverly chosen. We prove the global convergence of the algorithm mspACM under mild assumptions, without requiring strong convexity-concavity condition. Under the locally metrical subregularity of the solution mapping, we prove that the algorithm mspACM has the linear rate of convergence. Preliminary numerical results are reported to verify the efficiency of the algorithm mspACM.

math.OC↗

arXiv2020

Optimization with Least Constraint Violation

Study about theory and algorithms for constrained optimization usually assumes that the feasible region of the optimization problem is nonempty. However, there are many important practical optimization problems whose feasible regions are not known to be nonempty or not, and optimizers of the objective function with the least constraint violation prefer to be found. A natural way for dealing with these problems is to extend the constrained optimization problem as the one optimizing the objective function over the set of points with the least constraint violation. Firstly, the minimization problem with least constraint violation is proved to be an Lipschitz equality constrained optimization problem when the original problem is a convex optimization problem with possible inconsistent conic constraints, and it can be reformulated as an MPEC problem. Secondly, for nonlinear programming problems with possible inconsistent constraints, various types of stationary points are presented for the MPCC problem which is equivalent to the minimization problem with least constraint violation, and an elegant necessary optimality condition, named as L-stationary condition, is established from the classical optimality theory of Lipschitz continuous optimization. Finally, the smoothing Fischer-Burmeister function method for nonlinear programming case is constructed for solving the problem minimizing the objective function with the least constraint violation. It is demonstrated that, when the positive smoothing parameter approaches to zero, any point in the outer limit of the KKT-point mapping is an L-stationary point of the equivalent MPCC problem.

math.OC↗

arXiv2020

Stochastic Approximation Proximal Method of Multipliers for Convex Stochastic Programming

This paper considers the problem of minimizing a convex expectation function over a closed convex set, coupled with a set of inequality convex expectation constraints. We present a new stochastic approximation type algorithm, namely the stochastic approximation proximal method of multipliers (PMMSopt) to solve this convex stochastic optimization problem. We analyze regrets of a stochastic approximation proximal method of multipliers for solving convex stochastic optimization problems. Under mild conditions, we show that this algorithm exhibits ${\rm O}(T^{-1/2})$ rate of convergence, in terms of both optimality gap and constraint violation if parameters in the algorithm are properly chosen, when the objective and constraint functions are generally convex, where $T$ denotes the number of iterations. Moreover, we show that, with at least $1-e^{-T^{1/4}}$ probability, the algorithm has no more than ${\rm O}(T^{-1/4})$ objective regret and no more than ${\rm O}(T^{-1/8})$ constraint violation regret. To the best of our knowledge, this is the first time that such a proximal method for solving expectation constrained stochastic optimization is presented in the literature.

math.OC↗

arXiv2020

Statistical Robustness of Empirical Risks in Machine Learning

This paper studies convergence of empirical risks in reproducing kernel Hilbert spaces (RKHS). A conventional assumption in the existing research is that empirical training data do not contain any noise but this may not be satisfied in some practical circumstances. Consequently the existing convergence results do not provide a guarantee as to whether empirical risks based on empirical data are reliable or not when the data contain some noise. In this paper, we fill out the gap in a few steps. First, we derive moderate sufficient conditions under which the expected risk changes stably (continuously) against small perturbation of the probability distribution of the underlying random variables and demonstrate how the cost function and kernel affect the stability. Second, we examine the difference between laws of the statistical estimators of the expected optimal loss based on pure data and contaminated data using Prokhorov metric and Kantorovich metric and derive some qualitative and quantitative statistical robustness results. Third, we identify appropriate metrics under which the statistical estimators are uniformly asymptotically consistent. These results provide theoretical grounding for analysing asymptotic convergence and examining reliability of the statistical estimators in a number of well-known machine learning models.

math.OC↗

arXiv2020

Optimality Conditions for Constrained Minimax Optimization

Minimax optimization problems arises from both modern machine learning including generative adversarial networks, adversarial training and multi-agent reinforcement learning, as well as from tradition research areas such as saddle point problems, numerical partial differential equations and optimality conditions of equality constrained optimization. For the unconstrained continuous nonconvex-nonconcave situation, Jin, Netrapalli and Jordan (2019) carefully considered the very basic question: what is a proper definition of local optima of a minimax optimization problem, and proposed a proper definition of local optimality called local minimax. We shall extend the definition of local minimax point to constrained nonconvex-nonconcave minimax optimization problems. By analyzing Jacobian uniqueness conditions for the lower-level maximization problem and the strong regularity of Karush-Kuhn-Tucker conditions of the maximization problem, we provide both necessary optimality conditions and sufficient optimality conditions for the local minimax points of constrained minimax optimization problems.

math.OC↗

arXiv2020

Adaptive Step Size Strategy for Orthogonality Constrained Line Search Methods

In this paper, we propose an adaptive step size strategy for a class of line search methods for orthogonality constrained minimization problems, which avoids the classic backtracking procedure. We prove the convergence of the line search methods equipped with our adaptive step size strategy under some mild assumptions.We then apply the adaptive algorithm to electronic structure calculations, which show that our strategy is efficient and recommended.

math.OC↗

arXiv2020

Pactical Newton Methods for Electronic Structure Calculations

In this paper, we propose and analyze some practical Newton methods for electronic structure calculations. We show the convergence and the local quadratic convergence rate for the Newton method when the Newton search directions are well-obtained. In particular, we investigate some basic implementation issues in determining the search directions and step sizes which ensures the convergence of the subproblem at each iteration and accelerates the algorithm, respectively. It is shown by our numerical experiments that our Newton methods perform better than the existing conjugate gradient method, and the Newton method with the adaptive step size strategy is even more efficient.

math.OC↗

arXiv2018

Synergistic Drug Combination Prediction by Integrating Multi-omics Data in Deep Learning Models

Drug resistance is still a major challenge in cancer therapy. Drug combination is expected to overcome drug resistance. However, the number of possible drug combinations is enormous, and thus it is infeasible to experimentally screen all effective drug combinations considering the limited resources. Therefore, computational models to predict and prioritize effective drug combinations is important for combinatory therapy discovery in cancer. In this study, we proposed a novel deep learning model, AuDNNsynergy, to prediction drug combinations by integrating multi-omics data and chemical structure data. In specific, three autoencoders were trained using the gene expression, copy number and genetic mutation data of all tumor samples from The Cancer Genome Atlas. Then the physicochemical properties of drugs combined with the output of the three autoencoders, characterizing the individual cancer cell-lines, were used as the input of a deep neural network that predicts the synergy value of given pair-wise drug combinations against the specific cancer cell-lines. The comparison results showed the proposed AuDNNsynergy model outperforms four state-of-art approaches, namely DeepSynergy, Gradient Boosting Machines, Random Forests, and Elastic Nets. Moreover, we conducted the interpretation analysis of the deep learning model to investigate potential vital genetic predictors and the underlying mechanism of synergistic drug combinations on specific cancer cell-lines.

q-bio.GN↗

arXiv2018

Kernel-smoothed proper orthogonal decomposition (KSPOD)-based emulation for prediction of spatiotemporally evolving flow dynamics

This interdisciplinary study, which combines machine learning, statistical methodologies, high-fidelity simulations, and flow physics, demonstrates a new process for building an efficient surrogate model for predicting spatiotemporally evolving flow dynamics. In our previous work, a common-grid proper-orthogonal-decomposition (CPOD) technique was developed to establish a physics-based surrogate (emulation) model for prediction of mean flowfields and design exploration over a wide parameter space. The CPOD technique is substantially improved upon here using a kernel-smoothed POD (KSPOD) technique, which leverages kriging-based weighted functions from the design matrix. The resultant emulation model is then trained using a dataset obtained through high-fidelity simulations. As an example, the flow evolution in a swirl injector is considered for a wide range of design parameters and operating conditions. The KSPOD-based emulation model performs well, and can faithfully capture the spatiotemporal flow dynamics. The model enables effective design surveys utilizing high-fidelity simulation data, achieving a turnaround time for evaluating new design points that is 42,000 times faster than the original simulation.

cs.CE↗

arXiv2018

A Linearly Convergent Majorized ADMM with Indefinite Proximal Terms for Convex Composite Programming and Its Applications

This paper aims to study a majorized alternating direction method of multipliers with indefinite proximal terms (iPADMM) for convex composite optimization problems. We show that the majorized iPADMM for 2-block convex optimization problems converges globally under weaker conditions than those used in the literature and exhibits a linear convergence rate under a local error bound condition. Based on these, we establish the linear rate convergence results for a symmetric Gaussian-Seidel based majorized iPADMM, which is designed for multi-block composite convex optimization problems. Moreover, we apply the majorized iPADMM to solve different types of regularized logistic regression problems. The numerical results on both synthetic and real datasets demonstrate the efficiency of the majorized iPADMM and also illustrate the effectiveness of the introduced indefinite proximal terms.

math.OC↗