arXiv subjects

Yankai Cao

Publications and source records attributed to Yankai Cao.

14 recordsLinked to original sources

arXiv2026

On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) based on low-rank decomposition, such as LoRA, has become a standard for adapting large pretrained models. However, its behavior in sequential learning -- specifically regarding catastrophic forgetting -- remains insufficiently understood. In this work, we present an empirical study showing that forgetting is strongly influenced by the geometry and parameterization of the update subspace. While methods that restrict updates to small, shared matrix subspaces often suffer from task interference, tensor-based decompositions (e.g., LoRETTA) mitigate forgetting by capturing richer structural information within ultra-compact budgets, and structurally aligned parameterizations (e.g., WeGeFT) preserve pretrained representations. Our findings highlight update subspace design as a key factor in continual learning and offer practical guidance for selecting efficient adaptation strategies in sequential settings.

cs.LG↗

arXiv2026

A Global Optimization Algorithm for K-Center Clustering of One Billion Samples

This paper presents a practical global optimization algorithm for the K-center clustering problem, which aims to select K samples as the cluster centers to minimize the maximum within-cluster distance. This algorithm is based on a reduced-space branch and bound scheme and guarantees convergence to the global optimum in a finite number of steps by only branching on the regions of centers. To improve efficiency, we have designed a two-stage decomposable lower bound, the solution of which can be derived in a closed form. In addition, we also propose several acceleration techniques to narrow down the region of centers, including bounds tightening, sample reduction, and parallelization. Extensive studies on synthetic and real-world datasets have demonstrated that our algorithm can solve the K-center problems to global optimal within 4 hours for ten million samples in the serial mode and one billion samples in the parallel mode. Moreover, compared with the state-of-the-art heuristic methods, the global optimum obtained by our algorithm can averagely reduce the objective function by 25.8% on all the synthetic and real-world datasets.

math.OC↗

arXiv2025

Exact Learning of Linear Model Predictive Control Laws using Oblique Decision Trees with Linear Predictions

Model Predictive Control (MPC) is a powerful strategy for constrained multivariable systems but faces computational challenges in real-time deployment due to its online optimization requirements. While explicit MPC and neural network approximations mitigate this burden, they suffer from scalability issues or lack interpretability, limiting their applicability in safety-critical systems. This work introduces a data-driven framework that directly learns the Linear MPC control law from sampled state-action pairs using Oblique Decision Trees with Linear Predictions (ODT-LP), achieving both computational efficiency and interpretability. By leveraging the piecewise affine structure of Linear MPC, we prove that the Linear MPC control law can be replicated by finite-depth ODT-LP models. We develop a gradient-based training algorithm using smooth approximations of tree routing functions to learn this structure from grid-sampled Linear MPC solutions, enabling end-to-end optimization. Input-to-state stability is established under bounded approximation errors, with explicit error decomposition into learning inaccuracies and sampling errors to inform model design. Numerical experiments demonstrate that ODT-LP controllers match MPC's closed-loop performance while reducing online evaluation time by orders of magnitude compared to MPC, explicit MPC, neural network, and random forest counterparts. The transparent tree structure enables formal verification of control logic, bridging the gap between computational efficiency and certifiable reliability for safety-critical systems.

math.OC↗

arXiv2025

Power System Transition Planning: An Industry-Aligned Framework for Long-Term Optimization

This work introduces the category of Power System Transition Planning optimization problem. It aims to shift power systems to emissions-free networks efficiently. Unlike comparable work, the framework presented here broadly applies to the industry's decision-making process. It defines a field-appropriate functional boundary focused on the economic efficiency of power systems. Namely, while imposing a wide range of planning factors in the decision space, the model maintains the structure and depth of conventional power system planning under uncertainty, which leads to a large-scale multistage stochastic programming formulation that encounters intractability in real-life cases. Thus, the framework simultaneously invokes high-performance computing defaultism. In this comprehensive exposition, we present a guideline model, comparing its scope to existing formulations, supported by a fully detailed example problem, showcasing the analytical value of the solution gained in a small test case. Then, the framework's viability for realistic applications is demonstrated by solving an extensive test case based on a realistic planning construct consistent with Alberta's power system practices for long-term planning studies. The framework resorts to Stochastic Dual Dynamic Programming as a decomposition method to achieve tractability, leveraging High-Performance Computing and parallel computation.

eess.SY↗

arXiv2024

Adaptive Principal Components Allocation with the $\ell_{2,g}$-regularized Gaussian Graphical Model for Efficient Fine-Tuning Large Models

In this work, we propose a novel Parameter-Efficient Fine-Tuning (PEFT) approach based on Gaussian Graphical Models (GGMs), marking the first application of GGMs to PEFT tasks, to the best of our knowledge. The proposed method utilizes the $\ell_{2,g}$-norm to effectively select critical parameters and capture global dependencies. The resulting non-convex optimization problem is efficiently solved using a Block Coordinate Descent (BCD) algorithm. Experimental results on the GLUE benchmark [24] for fine-tuning RoBERTa-Base [18] demonstrate the effectiveness of the proposed approach, achieving competitive performance with significantly fewer trainable parameters. The code for this work is available at: https://github.com/jzheng20/Course projects.git.

cs.LG↗

arXiv2024

Can a Single Tree Outperform an Entire Forest?

The prevailing mindset is that a single decision tree underperforms classic random forests in testing accuracy, despite its advantages in interpretability and lightweight structure. This study challenges such a mindset by significantly improving the testing accuracy of an oblique regression tree through our gradient-based entire tree optimization framework, making its performance comparable to the classic random forest. Our approach reformulates tree training as a differentiable unconstrained optimization task, employing a scaled sigmoid approximation strategy. To ameliorate numerical instability, we propose an algorithmic scheme that solves a sequence of increasingly accurate approximations. Additionally, a subtree polish strategy is implemented to reduce approximation errors accumulated across the tree. Extensive experiments on 16 datasets demonstrate that our optimized tree outperforms the classic random forest by an average of $2.03\%$ improvements in testing accuracy.

cs.LG↗

arXiv2024

Handling The Non-Smooth Challenge in Tensor SVD: A Multi-Objective Tensor Recovery Framework

Recently, numerous tensor singular value decomposition (t-SVD)-based tensor recovery methods have shown promise in processing visual data, such as color images and videos. However, these methods often suffer from severe performance degradation when confronted with tensor data exhibiting non-smooth changes. It has been commonly observed in real-world scenarios but ignored by the traditional t-SVD-based methods. In this work, we introduce a novel tensor recovery model with a learnable tensor nuclear norm to address such a challenge. We develop a new optimization algorithm named the Alternating Proximal Multiplier Method (APMM) to iteratively solve the proposed tensor completion model. Theoretical analysis demonstrates the convergence of the proposed APMM to the Karush-Kuhn-Tucker (KKT) point of the optimization problem. In addition, we propose a multi-objective tensor recovery framework based on APMM to efficiently explore the correlations of tensor data across its various dimensions, providing a new perspective on extending the t-SVD-based method to higher-order tensor cases. Numerical experiments demonstrated the effectiveness of the proposed method in tensor completion.

stat.ML↗

arXiv2024

Causality-inspired Latent Feature Augmentation for Single Domain Generalization

Single domain generalization (Single-DG) intends to develop a generalizable model with only one single training domain to perform well on other unknown target domains. Under the domain-hungry configuration, how to expand the coverage of source domain and find intrinsic causal features across different distributions is the key to enhancing the models' generalization ability. Existing methods mainly depend on the meticulous design of finite image-level transformation techniques and learning invariant features across domains based on statistical correlation between samples and labels in source domain. This makes it difficult to capture stable semantics between source and target domains, which hinders the improvement of the model's generalization performance. In this paper, we propose a novel causality-inspired latent feature augmentation method for Single-DG by learning the meta-knowledge of feature-level transformation based on causal learning and interventions. Instead of strongly relying on the finite image-level transformation, with the learned meta-knowledge, we can generate diverse implicit feature-level transformations in latent space based on the consistency of causal features and diversity of non-causal features, which can better compensate for the domain-hungry defect and reduce the strong reliance on initial finite image-level transformations and capture more stable domain-invariant causal features for generalization. Extensive experiments on several open-access benchmarks demonstrate the outstanding performance of our model over other state-of-the-art single domain generalization and also multi-source domain generalization methods.

cs.CV↗

arXiv2023

A GPU-Accelerated Moving-Horizon Algorithm for Training Deep Classification Trees on Large Datasets

Decision trees are essential yet NP-complete to train, prompting the widespread use of heuristic methods such as CART, which suffers from sub-optimal performance due to its greedy nature. Recently, breakthroughs in finding optimal decision trees have emerged; however, these methods still face significant computational costs and struggle with continuous features in large-scale datasets and deep trees. To address these limitations, we introduce a moving-horizon differential evolution algorithm for classification trees with continuous features (MH-DEOCT). Our approach consists of a discrete tree decoding method that eliminates duplicated searches between adjacent samples, a GPU-accelerated implementation that significantly reduces running time, and a moving-horizon strategy that iteratively trains shallow subtrees at each node to balance the vision and optimizer capability. Comprehensive studies on 68 UCI datasets demonstrate that our approach outperforms the heuristic method CART on training and testing accuracy by an average of 3.44% and 1.71%, respectively. Moreover, these numerical studies empirically demonstrate that MH-DEOCT achieves near-optimal performance (only 0.38% and 0.06% worse than the global optimal method on training and testing, respectively), while it offers remarkable scalability for deep trees (e.g., depth=8) and large-scale datasets (e.g., ten million samples).

cs.LG↗

arXiv2022

A Review of High-Performance Computing and Parallel Techniques Applied to Power Systems Optimization

The accelerating technological landscape and drive towards net-zero emission made the power system grow in scale and complexity. Serial computational approaches for grid planning and operation struggle to execute necessary calculations within reasonable times. Resorting to high-performance and parallel computing approaches has become paramount. Moreover, the ambitious plans for the future grid and IoT integration make a shift towards utilizing Cloud computing inevitable. This article recounts the dawn of parallel computation and its appearance in power system studies, reviewing the most recent literature and research on exploiting the available computational resources and technologies today. The article starts with a brief introduction to the field. The relevant hardware and paradigms are then explained thoroughly in this article providing a base for the reader to understand the literature. Later, parallel power system studies are reviewed, reciting the study development from older papers up to the 21st century, emphasizing the most impactful work of the last decade. The studies included system stability studies, state estimation and power system operation, and market optimization. The reviews are split into \ac{CPU} based,\ac{GPU} based, and Cloud-based studies. Finally, the state-of-the-art is discussed, highlighting the issue of standardization and the future of computation in power system studies.

eess.SY↗

arXiv2022

Deep Learning-based Predictive Control of Battery Management for Frequency Regulation

This paper proposes a deep learning-based optimal battery management scheme for frequency regulation (FR) by integrating model predictive control (MPC), supervised learning (SL), reinforcement learning (RL), and high-fidelity battery models. By taking advantage of deep neural networks (DNNs), the derived DNN-approximated policy is computationally efficient in online implementation. The design procedure of the proposed scheme consists of two sequential processes: (1) the SL process, in which we first run a simulation with an MPC embedding a low-fidelity battery model to generate a training data set, and then, based on the generated data set, we optimize a DNN-approximated policy using SL algorithms; and (2) the RL process, in which we utilize RL algorithms to improve the performance of the DNN-approximated policy by balancing short-term economic incentives and long-term battery degradation. The SL process speeds up the subsequent RL process by providing a good initialization. By utilizing RL algorithms, one prominent property of the proposed scheme is that it can learn from the data generated by simulating the FR policy on the high-fidelity battery simulator to adjust the DNN-approximated policy, which is originally based on low-fidelity battery model. A case study using real-world data of FR signals and prices is performed. Simulation results show that, compared to conventional MPC schemes, the proposed deep learning-based scheme can effectively achieve higher economic benefits of FR participation while maintaining lower online computational cost.

eess.SY↗

arXiv2020

Multiscale Model Predictive Control of Battery Systems for Frequency Regulation Markets using Physics-Based Models

We propose a multiscale model predictive control (MPC) framework for stationary battery systems that exploits high-fidelity models to trade-off short-term economic incentives provided by energy and frequency regulation (FR) markets and long-term degradation effects. We find that the MPC framework can drastically reduce long-term degradation while properly responding to FR and energy market signals (compared to MPC formulations that use low-fidelity models). Our results also provide evidence that sophisticated battery models can be embedded within closedloop MPC simulations by using modern nonlinear programming solvers (we provide an efficient and easy-to-use implementation in Julia). We use insights obtained with our simulations to design a low-complexity MPC formulation that matches the behavior obtained with high-fidelity models. This is done by designing a suitable terminal penalty term that implicitly captures longterm degradation. The results suggest that complex degradation behavior can be accounted for in low-complexity MPC formulations by properly designing the cost function. We believe that our proof-of-concept results can be of industrial relevance, as battery vendors are seeking to participate in fast-changing electricity markets while maintaining asset integrity.

math.OC↗

arXiv2020

A Sigmoidal Approximation for Chance-Constrained Nonlinear Programs

We propose a sigmoidal approximation for the value-at-risk (that we call SigVaR) and we use this approximation to tackle nonlinear programs (NLPs) with chance constraints. We prove that the approximation is conservative and that the level of conservatism can be made arbitrarily small for limiting parameter values. The SigVar approximation brings scalability benefits over exact mixed-integer reformulations because its sample average approximation can be cast as a standard NLP. We also establish explicit connections between SigVaR and other smooth sigmoidal approximations recently reported in the literature. We show that a key benefit of SigVaR over such approximations is that one can establish an explicit connection with the conditional value at risk (CVaR) approximation and exploit this connection to obtain initial guesses for the approximation parameters. We present small- and large-scale numerical studies to illustrate the developments.

math.OC↗

arXiv2018

Graph-Based Modeling and Simulation of Complex Systems

We present graph-based modeling abstractions to represent cyber-physical dependencies arising in complex systems. Specifically, we propose an algebraic graph abstraction to capture physical connectivity in complex optimization models and a computing graph abstraction to capture communication connectivity in computing architectures. The proposed abstractions are scalable and are used as the backbone of a Julia -based software package that we call Plasmo.jl . We show how the algebraic graph abstraction facilitates the implementation, analysis, and decomposition of optimization problems and we show how the computing graph abstraction facilitates the implementation of optimization and control algorithms and their simulation in virtual environments that involve distributed, centralized, and hierarchical computing architectures.

math.OC↗