arXiv subjects

Sungho Shin

Publications and source records attributed to Sungho Shin.

At least 55 records · Page 3Linked to original sources

arXiv2022

On the Convergence of Overlapping Schwarz Decomposition for Nonlinear Optimal Control

We study the convergence properties of an overlapping Schwarz decomposition algorithm for solving nonlinear optimal control problems (OCPs). The algorithm decomposes the time domain into a set of overlapping subdomains, and solves all subproblems defined over subdomains in parallel. The convergence is attained by updating primal-dual information at the boundaries of overlapping subdomains. We show that the algorithm exhibits local linear convergence, and that the convergence rate improves exponentially with the overlap size. We also establish global convergence results for a general quadratic programming, which enables the application of the Schwarz scheme inside second-order optimization algorithms (e.g., sequential quadratic programming). The theoretical foundation of our convergence analysis is a sensitivity result of nonlinear OCPs, which we call "exponential decay of sensitivity" (EDS). Intuitively, EDS states that the impact of perturbations at domain boundaries (i.e. initial and terminal time) on the solution decays exponentially as one moves into the domain. Here, we expand a previous analysis available in the literature by showing that EDS holds for both primal and dual solutions of nonlinear OCPs, under uniform second-order sufficient condition, controllability condition, and boundedness condition. We conduct experiments with a quadrotor motion planning problem and a PDE control problem to validate our theory; and show that the approach is significantly more efficient than ADMM and as efficient as the centralized solver Ipopt.

math.OC↗

arXiv2021

Exponential Decay of Sensitivity in Graph-Structured Nonlinear Programs

We study solution sensitivity for nonlinear programs (NLPs) whose structures are induced by graphs. These NLPs arise in many applications such as dynamic optimization, stochastic optimization, optimization with partial differential equations, and network optimization. We show that for a given pair of nodes, the sensitivity of the primal-dual solution at one node against a data perturbation at the other node decays exponentially with respect to the distance between these two nodes on the graph. In other words, the solution sensitivity decays as one moves away from the perturbation point. This result, which we call exponential decay of sensitivity, holds under the strong second-order sufficiency condition and the linear independence constraint qualification. We also present conditions under which the decay rate remains uniformly bounded; this allows us to characterize the sensitivity behavior of NLPs defined over subgraphs of infinite graphs. The theoretical developments are illustrated with numerical examples.

math.OC↗

arXiv2021

A Feasible Reduced Space Method for Real-Time Optimal Power Flow

We propose a novel feasible-path algorithm to solve the optimal power flow (OPF) problem for real-time use cases. The method augments the seminal work of Dommel and Tinney with second-order derivatives to work directly in the reduced space induced by the power flow equations. In the reduced space, the optimization problem includes only inequality constraints corresponding to the operational constraints. While the reduced formulation directly enforces the physical constraints, the operational constraints are softly enforced through Augmented Lagrangian penalty terms. In contrast to interior-point algorithms (state-of-the art for solving OPF), our algorithm maintains feasibility at each iteration, which makes it suitable for real-time application. By exploiting accelerator hardware (Graphic Processing Units) to compute the reduced Hessian, we show that the second-order method is numerically tractable and is effective to solve both static and real-time OPF problems.

math.OC↗

arXiv2021

Controllability and Observability Imply Exponential Decay of Sensitivity in Dynamic Optimization

We study a property of dynamic optimization (DO) problems (as those encountered in model predictive control and moving horizon estimation) that is known as exponential decay of sensitivity (EDS). This property indicates that the sensitivity of the solution at stage $i$ against a data perturbation at stage $j$ decays exponentially with $|i-j|$. {Building upon our previous results, we show that EDS holds under uniform boundedness of the Lagrangian Hessian, a uniform second order sufficiency condition (uSOSC), and a uniform linear independence constraint qualification (uLICQ). Furthermore, we prove that uSOSC and uLICQ can be obtained under uniform controllability and observability. Hence, we have that uniform controllability and observability imply EDS.} These results provide insights into how perturbations propagate along the horizon and enable the development of approximation and solution schemes. We illustrate the developments with numerical examples.

math.OC↗

arXiv2021

Graph-Based Modeling and Decomposition of Energy Infrastructures

Nonlinear optimization problems are found at the heart of real-time operations of critical infrastructures. These problems are computationally challenging because they embed complex physical models that exhibit space-time dynamics. We propose modeling these problems as graph-structured optimization problems, and illustrate how their structure can be exploited at the modeling level (for parallelizing function/derivative computations) and at the solver level (for parallelizing linear algebra operations). Specifically, we present a restricted additive Schwarz scheme that enables flexible decomposition of complex graph structures within an interior-point algorithm. The proposed approach is implemented as a general-purpose nonlinear programming solver that we call MadNLP.jl; this Julia-based solver is interfaced to the graph-based modeling package Plasmo.jl. The efficiency of this framework is demonstrated via problems arising in transient gas network optimization and multi-period AC optimal power flow. We show that our framework accelerates the solution (compared to off-the-shelf tools) by over 300%; specifically, solution times are reduced from 72.36 sec to 23.84 sec for the gas problem and from 515.81 sec to 149.45 sec for the power flow problem.

math.OC↗

arXiv2021

Object Detection for Understanding Assembly Instruction Using Context-aware Data Augmentation and Cascade Mask R-CNN

Understanding assembly instruction has the potential to enhance the robot s task planning ability and enables advanced robotic applications. To recognize the key components from the 2D assembly instruction image, We mainly focus on segmenting the speech bubble area, which contains lots of information about instructions. For this, We applied Cascade Mask R-CNN and developed a context-aware data augmentation scheme for speech bubble segmentation, which randomly combines images cuts by considering the context of assembly instructions. We showed that the proposed augmentation scheme achieves a better segmentation performance compared to the existing augmentation algorithm by increasing the diversity of trainable data while considering the distribution of components locations. Also, we showed that deep learning can be useful to understand assembly instruction by detecting the essential objects in the assembly instruction, such as tools and parts.

cs.RO↗

arXiv2020

Diffusing-Horizon Model Predictive Control

We analyze a time-coarsening strategy for model predictive control (MPC) that we call diffusing-horizon MPC. This strategy seeks to overcome the computational challenges associated with optimal control problems that span multiple timescales. The coarsening approach uses a time discretization grid that becomes exponentially more sparse as one moves forward in time. This design is motivated by a recently established property of optimal control problems that is known as exponential decay of sensitivity. This property states that the impact of a parametric perturbation at a future time decays exponentially as one moves backward in time. We establish conditions under which this property holds for a constrained MPC formulation with linear dynamics and costs. Moreover, we show that the proposed coarsening scheme can be cast as a parametric perturbation of the MPC problem and thus the exponential decay condition holds. We use a heating, ventilation, and air conditioning plant case study with real data to demonstrate the proposed approach. Specifically, we show that computational times can be reduced by two orders of magnitude while increasing the closed-loop cost by only 3%.

math.OC↗

arXiv2020

SBML2Julia: interfacing SBML with efficient nonlinear Julia modelling and solution tools for parameter optimization

Motivation: Estimating model parameters from experimental observations is one of the key challenges in systems biology and can be computationally very expensive. While the Julia programming language was recently developed as a high-level and high-performance language for scientific computing, systems biologists have only started to realise its potential. For instance, we have recently used Julia to cut down the optimization time of a microbial community model by a factor of 140. To facilitate access of the systems biology community to the efficient nonlinear solvers used for this optimisation, we developed SBML2Julia. SBML2Julia translates optimisation problems specified in SBML and TSV files (PEtab format) into Julia for Mathematical Programming (JuMP), executes the optimization and returns the results in tabular format. Availability and implementation: SBML2Julia is freely available under the MIT license. It comes with a command line interface and Python API. Internally, SBML2Julia calls the Julia LTS release v1.0.5 for optimisation. All necessary dependencies can be pulled from Docker Hub (https://hub.docker.com/repository/docker/paulflang/sbml2julia). Source code and documentation are available at https://github.com/paulflang/SBML2Julia.

q-bio.QM↗

arXiv2020

Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks

The quantization of deep neural networks (QDNNs) has been actively studied for deployment in edge devices. Recent studies employ the knowledge distillation (KD) method to improve the performance of quantized networks. In this study, we propose stochastic precision ensemble training for QDNNs (SPEQ). SPEQ is a knowledge distillation training scheme; however, the teacher is formed by sharing the model parameters of the student network. We obtain the soft labels of the teacher by changing the bit precision of the activation stochastically at each layer of the forward-pass computation. The student model is trained with these soft labels to reduce the activation quantization noise. The cosine similarity loss is employed, instead of the KL-divergence, for KD training. As the teacher model changes continuously by random bit-precision assignment, it exploits the effect of stochastic ensemble KD. SPEQ outperforms the existing quantization training methods in various tasks, such as image classification, question-answering, and transfer learning without the need for cumbersome teacher networks.

cs.LG↗

arXiv2020

Multiple Classification with Split Learning

Privacy issues were raised in the process of training deep learning in medical, mobility, and other fields. To solve this problem, we present privacy-preserving distributed deep learning method that allow clients to learn a variety of data without direct exposure. We divided a single deep learning architecture into a common extractor, a cloud model and a local classifier for the distributed learning. First, the common extractor, which is used by local clients, extracts secure features from the input data. The secure features also take the role that the cloud model can employ various task and diverse types of data. The feature contain the most important information that helps to proceed various task. Second, the cloud model including most parts of the whole training model gets the embedded features from the massive local clients, and performs most of deep learning operations which takes severe computing cost. After the operations in cloud model finished, outputs of the cloud model send back to local clients. Finally, the local classifier determined classification results and delivers the results to local clients. When clients train models, our model does not directly expose sensitive information to exterior network. During the test, the average performance improvement was 2.63% over the existing local training model. However, in a distributed environment, there is a possibility of inversion attack due to exposed features. For this reason, we experimented with the common extractor to prevent data restoration. The quality of restoration of the original image was tested by adjusting the depth of the common extractor. As a result, we found that the deeper the common extractor, the restoration score decreased to 89.74.

cs.LG↗

arXiv2020

S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima

The stochastic gradient descent (SGD) method is most widely used for deep neural network (DNN) training. However, the method does not always converge to a flat minimum of the loss surface that can demonstrate high generalization capability. Weight noise injection has been extensively studied for finding flat minima using the SGD method. We devise a new weight-noise injection-based SGD method that adds symmetrical noises to the DNN weights. The training with symmetrical noise evaluates the loss surface at two adjacent points, by which convergence to sharp minima can be avoided. Fixed-magnitude symmetric noises are added to minimize training instability. The proposed method is compared with the conventional SGD method and previous weight-noise injection algorithms using convolutional neural networks for image classification. Particularly, performance improvements in large batch training are demonstrated. This method shows superior performance compared with conventional SGD and weight-noise injection methods regardless of the batch-size and learning rate scheduling algorithms.

cs.LG↗

arXiv2020

Overlapping Schwarz Decomposition for Constrained Quadratic Programs

We present an overlapping Schwarz decomposition algorithm for constrained quadratic programs (QPs). Schwarz algorithms have been traditionally used to solve linear algebra systems arising from partial differential equations, but we have recently shown that they are also effective at solving structured optimization problems. In the proposed scheme, we consider QPs whose algebraic structure can be represented by graphs. The graph domain is partitioned into overlapping subdomains (yielding a set of coupled subproblems), solutions for the subproblems are computed in parallel, and convergence is enforced by updating primal-dual information in the overlapping regions. We show that convergence is guaranteed if the overlap is sufficiently large and that the convergence rate improves exponentially with the size of the overlap. Convergence results rely on a key property of graph-structured problems that is known as exponential decay of sensitivity. Here, we establish conditions under which this property holds for constrained QPs (as those found in network optimization and optimal control), thus extending existing work that addresses unconstrained QPs. The numerical behavior of the Schwarz scheme is demonstrated by using a DC optimal power flow problem defined over a network with 9,241 nodes.

math.OC↗

arXiv2020

A Graph-Based Modeling Abstraction for Optimization: Concepts and Implementation in Plasmo.jl

We present a general graph-based modeling abstraction for optimization that we call an OptiGraph. Under this abstraction, any optimization problem is treated as a hierarchical hypergraph in which nodes represent optimization subproblems and edges represent connectivity between such subproblems. The abstraction enables the modular construction of highly complex models in an intuitive manner, facilitates the use of graph analysis tools (to perform partitioning, aggregation, and visualization tasks), and facilitates communication of structures to decomposition algorithms. We provide an open-source implementation of the abstraction in the Julia-based package Plasmo.jl. We provide tutorial examples and large application case studies to illustrate the capabilities.

math.OC↗

arXiv2020

Quantized Neural Networks: Characterization and Holistic Optimization

Quantized deep neural networks (QDNNs) are necessary for low-power, high throughput, and embedded applications. Previous studies mostly focused on developing optimization methods for the quantization of given models. However, quantization sensitivity depends on the model architecture. Therefore, the model selection needs to be a part of the QDNN design process. Also, the characteristics of weight and activation quantization are quite different. This study proposes a holistic approach for the optimization of QDNNs, which contains QDNN training methods as well as quantization-friendly architecture design. Synthesized data is used to visualize the effects of weight and activation quantization. The results indicate that deeper models are more prone to activation quantization, while wider models improve the resiliency to both weight and activation quantization. This study can provide insight into better optimization of QDNNs.

cs.LG↗

arXiv2020

Characterizing the Predictive Accuracy of Dynamic Mode Decomposition for Data-Driven Control

Dynamic mode decomposition (DMD) is a versatile approach that enables the construction of low-order models from data. Controller design tasks based on such models require estimates and guarantees on predictive accuracy. In this work, we provide a theoretical analysis of DMD model errors that reveals impact of model order and data availability. The analysis also establishes conditions under which DMD models can be made asymptotically exact. We verify our results using a 2D diffusion system.

eess.SY↗

arXiv2020

Unifying Theorems for Subspace Identification and Dynamic Mode Decomposition

This paper presents unifying results for subspace identification (SID) and dynamic mode decomposition (DMD) for autonomous dynamical systems. We observe that SID seeks to solve an optimization problem to estimate an extended observability matrix and a state sequence that minimizes the prediction error for the state-space model. Moreover, we observe that DMD seeks to solve a rank-constrained matrix regression problem that minimizes the prediction error of an extended autoregressive model. We prove that existence conditions for perfect (error-free) state-space and low-rank extended autoregressive models are equivalent and that the SID and DMD optimization problems are equivalent. We exploit these results to propose a SID-DMD algorithm that delivers a provably optimal model and that is easy to implement. We demonstrate our developments using a case study that aims to build dynamical models directly from video data.

eess.SY↗

arXiv2020

On the Convergence of the Dynamic Inner PCA Algorithm

Dynamic inner principal component analysis (DiPCA) is a powerful method for the analysis of time-dependent multivariate data. DiPCA extracts dynamic latent variables that capture the most dominant temporal trends by solving a large-scale, dense, and nonconvex nonlinear program (NLP). A scalable decomposition algorithm has been recently proposed in the literature to solve these challenging NLPs. The decomposition algorithm performs well in practice but its convergence properties are not well understood. In this work, we show that this algorithm is a specialized variant of a coordinate maximization algorithm. This observation allows us to explain why the decomposition algorithm might work (or not) in practice and can guide improvements. We compare the performance of the decomposition strategies with that of the off-the-shelf solver Ipopt. The results show that decomposition is more scalable and, surprisingly, delivers higher quality solutions.

eess.SY↗

arXiv2020

Multi-Grid Schemes for Multi-Scale Coordination of Energy Systems

We discuss how multi-grid computing schemes can be used to design hierarchical coordination architectures for energy systems. These hierarchical architectures can be used to manage multiple temporal and spatial scales and mitigate fundamental limitations of centralized and decentralized architectures. We present the basic elements of a multi-grid scheme, which includes a smoothing operator (a high-resolution decentralized coordination layer that targets phenomena at high frequencies) and a coarsening operator (a low-resolution centralized coordination layer that targets phenomena at low frequencies). For smoothing, we extend existing convergence results for Gauss-Seidel schemes by applying them to systems that cover unstructured domains. This allows us to target problems with multiple timescales and arbitrary networks. The proposed coordination schemes can be used to guide transactions in decentralized electricity markets. We present a storage control example and a power flow diffusion example to illustrate the developments.

math.OC↗