arXiv subjects

Kibaek Kim

Publications and source records attributed to Kibaek Kim.

At least 37 records · Page 2Linked to original sources

arXiv2025

Advances in APPFL: A Comprehensive and Extensible Federated Learning Framework

Federated learning (FL) is a distributed machine learning paradigm enabling collaborative model training while preserving data privacy. In today's landscape, where most data is proprietary, confidential, and distributed, FL has become a promising approach to leverage such data effectively, particularly in sensitive domains such as medicine and the electric grid. Heterogeneity and security are the key challenges in FL, however, most existing FL frameworks either fail to address these challenges adequately or lack the flexibility to incorporate new solutions. To this end, we present the recent advances in developing APPFL, an extensible framework and benchmarking suite for federated learning, which offers comprehensive solutions for heterogeneity and security concerns, as well as user-friendly interfaces for integrating new algorithms or adapting to new applications. We demonstrate the capabilities of APPFL through extensive experiments evaluating various aspects of FL, including communication efficiency, privacy preservation, computational performance, and resource utilization. We further highlight the extensibility of APPFL through case studies in vertical, hierarchical, and decentralized FL. APPFL is fully open-sourced on GitHub at https://github.com/APPFL/APPFL.

cs.LG↗

arXiv2025

FedSpaLLM: Federated Pruning of Large Language Models

Large Language Models (LLMs) achieve state-of-the-art performance but are challenging to deploy due to their high computational and storage demands. Pruning can reduce model size, yet existing methods assume public access to calibration data, which is impractical for privacy-sensitive applications. To address the challenge of pruning LLMs in privacy-preserving settings, we propose FedSpaLLM, the first federated learning framework designed specifically for pruning LLMs. FedSpaLLM enables clients to prune their models locally based on private data while accounting for system heterogeneity and maintaining communication efficiency. Our framework introduces several key innovations: (1) a novel $\ell_0$-norm aggregation function that ensures only non-zero weights are averaged across clients, preserving important model parameters; (2) an adaptive mask expansion technique that meets global sparsity targets while accommodating client-specific pruning decisions; and (3) a layer sampling strategy that reduces communication overhead and personalizes the pruning process based on client resources. Extensive experiments show that FedSpaLLM improves pruning performance in diverse federated settings.

cs.LG↗

arXiv2025

Federated Low-Rank Tensor Estimation for Multimodal Image Reconstruction

Low-rank tensor estimation offers a powerful approach to addressing high-dimensional data challenges and can substantially improve solutions to ill-posed inverse problems, such as image reconstruction under noisy or undersampled conditions. Meanwhile, tensor decomposition has gained prominence in federated learning (FL) due to its effectiveness in exploiting latent space structure and its capacity to enhance communication efficiency. In this paper, we present a federated image reconstruction method that applies Tucker decomposition, incorporating joint factorization and randomized sketching to manage large-scale, multimodal data. Our approach avoids reconstructing full-size tensors and supports heterogeneous ranks, allowing clients to select personalized decomposition ranks based on prior knowledge or communication capacity. Numerical results demonstrate that our method achieves superior reconstruction quality and communication compression compared to existing approaches, thereby highlighting its potential for multimodal inverse problems in the FL setting.

cs.LG↗

arXiv2025

A GPU-Accelerated Distributed Algorithm for Optimal Power Flow in Distribution Systems

We propose a GPU-accelerated distributed optimization algorithm for controlling multi-phase optimal power flow in active distribution systems with dynamically changing topologies. To handle varying network configurations and enable adaptable decomposition, we advocate a componentwise decomposition strategy. However, this approach can lead to a prolonged computation time mainly due to the excessive iterations required for achieving consensus among a large number of fine-grained components. To overcome this, we introduce a technique that segregates equality constraints from inequality constraints, enabling GPU parallelism to reduce per-iteration time by orders of magnitude, thereby significantly accelerating the overall computation. Numerical experiments on IEEE test systems ranging from 13 to 8500 buses demonstrate the superior scalability of the proposed approach compared to its CPU-based counterparts.

cs.DC↗

arXiv2025

FIRM: Federated Image Reconstruction using Multimodal Tomographic Data

We propose a federated algorithm for reconstructing images using multimodal tomographic data sourced from dispersed locations, addressing the challenges of traditional unimodal approaches that are prone to noise and reduced image quality. Our approach formulates a joint inverse optimization problem incorporating multimodality constraints and solves it in a federated framework through local gradient computations complemented by lightweight central operations, ensuring data decentralization. Leveraging the connection between our federated algorithm and the quadratic penalty method, we introduce an adaptive step-size rule with guaranteed sublinear convergence and further suggest its extension to augmented Lagrangian framework. Numerical results demonstrate its superior computational efficiency and improved image reconstruction quality.

math.OC↗

arXiv2024

From RNNs to Foundation Models: An Empirical Study on Commercial Building Energy Consumption

Accurate short-term energy consumption forecasting for commercial buildings is crucial for smart grid operations. While smart meters and deep learning models enable forecasting using past data from multiple buildings, data heterogeneity from diverse buildings can reduce model performance. The impact of increasing dataset heterogeneity in time series forecasting, while keeping size and model constant, is understudied. We tackle this issue using the ComStock dataset, which provides synthetic energy consumption data for U.S. commercial buildings. Two curated subsets, identical in size and region but differing in building type diversity, are used to assess the performance of various time series forecasting models, including fine-tuned open-source foundation models (FMs). The results show that dataset heterogeneity and model architecture have a greater impact on post-training forecasting performance than the parameter count. Moreover, despite the higher computational cost, fine-tuned FMs demonstrate competitive performance compared to base models trained from scratch.

cs.LG↗

arXiv2024

Foundation Models for the Electric Power Grid

Foundation models (FMs) currently dominate news headlines. They employ advanced deep learning architectures to extract structural information autonomously from vast datasets through self-supervision. The resulting rich representations of complex systems and dynamics can be applied to many downstream applications. Therefore, FMs can find uses in electric power grids, challenged by the energy transition and climate change. In this paper, we call for the development of, and state why we believe in, the potential of FMs for electric grids. We highlight their strengths and weaknesses amidst the challenges of a changing grid. We argue that an FM learning from diverse grid data and topologies could unlock transformative capabilities, pioneering a new approach in leveraging AI to redefine how we manage complexity and uncertainty in the electric grid. Finally, we discuss a power grid FM concept, namely GridFM, based on graph neural networks and show how different downstream tasks benefit.

cs.LG↗

arXiv2024

SparseLLM: Towards Global Pruning for Pre-trained Language Models

The transformative impact of large language models (LLMs) like LLaMA and GPT on natural language processing is countered by their prohibitive computational demands. Pruning has emerged as a pivotal compression strategy, introducing sparsity to enhance both memory and computational efficiency. Yet, traditional global pruning is impractical for LLMs due to scalability issues, while local pruning, despite its efficiency, leads to suboptimal solutions. Addressing these challenges, we propose SparseLLM, a novel framework that redefines the global pruning process into manageable, coordinated subproblems, allowing for resource-efficient optimization with global optimality. SparseLLM's approach, which conceptualizes LLMs as a chain of modular functions and leverages auxiliary variables for problem decomposition, not only facilitates a pragmatic application on LLMs but also demonstrates significant performance improvements, particularly in high-sparsity regimes where it surpasses current state-of-the-art methods.

cs.CL↗

arXiv2024

Advances in Privacy Preserving Federated Learning to Realize a Truly Learning Healthcare System

The concept of a learning healthcare system (LHS) envisions a self-improving network where multimodal data from patient care are continuously analyzed to enhance future healthcare outcomes. However, realizing this vision faces significant challenges in data sharing and privacy protection. Privacy-Preserving Federated Learning (PPFL) is a transformative and promising approach that has the potential to address these challenges by enabling collaborative learning from decentralized data while safeguarding patient privacy. This paper proposes a vision for integrating PPFL into the healthcare ecosystem to achieve a truly LHS as defined by the Institute of Medicine (IOM) Roundtable.

cs.CR↗

arXiv2024

Modeling and solving cascading failures across interdependent infrastructure systems

Physical infrastructure systems supply crucial resources to residential, commercial, and industrial activities. These infrastructure systems generally consist of multiple types of infrastructure assets that are interdependent. In the event of a disaster, some of the infrastructure assets can be damaged and disabled, creating failures that propagate to other assets that depend on the disabled assets and cause a cascade of failures that may lead to a potential system collapse. We present a bilevel interdiction model in this paper to study this problem of cascading failures in a system of interdependent infrastructure systems with a nondeterministic dependency graph. We also propose a computationally tractable reformulation of the proposed bilevel model and utilize a Benders-type decomposition algorithm to solve the resulting formulation. Computational experiments are performed using infrastructure networks generated from anonymized real-world data to validate the performance of this algorithm.

math.OC↗

arXiv2024

FedSZ: Leveraging Error-Bounded Lossy Compression for Federated Learning Communications

With the promise of federated learning (FL) to allow for geographically-distributed and highly personalized services, the efficient exchange of model updates between clients and servers becomes crucial. FL, though decentralized, often faces communication bottlenecks, especially in resource-constrained scenarios. Existing data compression techniques like gradient sparsification, quantization, and pruning offer some solutions, but may compromise model performance or necessitate expensive retraining. In this paper, we introduce FedSZ, a specialized lossy-compression algorithm designed to minimize the size of client model updates in FL. FedSZ incorporates a comprehensive compression pipeline featuring data partitioning, lossy and lossless compression of model parameters and metadata, and serialization. We evaluate FedSZ using a suite of error-bounded lossy compressors, ultimately finding SZ2 to be the most effective across various model architectures and datasets including AlexNet, MobileNetV2, ResNet50, CIFAR-10, Caltech101, and Fashion-MNIST. Our study reveals that a relative error bound 1E-2 achieves an optimal tradeoff, compressing model states between 5.55-12.61x while maintaining inference accuracy within <0.5% of uncompressed results. Additionally, the runtime overhead of FedSZ is <4.7% or between of the wall-clock communication-round time, a worthwhile trade-off for reducing network transfer times by an order of magnitude for networks bandwidths <500Mbps. Intriguingly, we also find that the error introduced by FedSZ could potentially serve as a source of differentially private noise, opening up new avenues for privacy-preserving FL.

cs.DC↗

arXiv2024

Linearized Optimal Power Flow for Multiphase Radial Networks with Delta Connections

This paper proposes a linear approximation of the alternating current optimal power flow problem for multiphase distribution networks with voltage-dependent loads connected in both wye and delta configurations. We establish a set of linear equations that exactly describes the relationship between power changes at a bus and the corresponding values from a delta-connected device under specific assumptions necessary for a widely accepted linear model. Numerical studies on IEEE test feeders demonstrate that the proposed linear model provides solutions with reasonable error bounds efficiently, as compared with an exact nonconvex formulation and a convex conic relaxation. Our experiments reveal that modeling delta-connected, voltage-dependent loads as if they are wye-connected can lead to significantly different voltage profiles. We also investigate the limitations of the proposed linear approximation stemming from the underlying assumptions, while emphasizing its robust performance in practical situations.

math.OC↗

arXiv2024

Addressing Heterogeneity in Federated Load Forecasting with Personalization Layers

The advent of smart meters has enabled pervasive collection of energy consumption data for training short-term load forecasting models. In response to privacy concerns, federated learning (FL) has been proposed as a privacy-preserving approach for training, but the quality of trained models degrades as client data becomes heterogeneous. In this paper we propose the use of personalization layers for load forecasting in a general framework called PL-FL. We show that PL-FL outperforms FL and purely local training, while requiring lower communication bandwidth than FL. This is done through extensive simulations on three different datasets from the NREL ComStock repository.

cs.LG↗

arXiv2024

FedCompass: Efficient Cross-Silo Federated Learning on Heterogeneous Client Devices using a Computing Power Aware Scheduler

Cross-silo federated learning offers a promising solution to collaboratively train robust and generalized AI models without compromising the privacy of local datasets, e.g., healthcare, financial, as well as scientific projects that lack a centralized data facility. Nonetheless, because of the disparity of computing resources among different clients (i.e., device heterogeneity), synchronous federated learning algorithms suffer from degraded efficiency when waiting for straggler clients. Similarly, asynchronous federated learning algorithms experience degradation in the convergence rate and final model accuracy on non-identically and independently distributed (non-IID) heterogeneous datasets due to stale local models and client drift. To address these limitations in cross-silo federated learning with heterogeneous clients and data, we propose FedCompass, an innovative semi-asynchronous federated learning algorithm with a computing power-aware scheduler on the server side, which adaptively assigns varying amounts of training tasks to different clients using the knowledge of the computing power of individual clients. FedCompass ensures that multiple locally trained models from clients are received almost simultaneously as a group for aggregation, effectively reducing the staleness of local models. At the same time, the overall training process remains asynchronous, eliminating prolonged waiting periods from straggler clients. Using diverse non-IID heterogeneous distributed datasets, we demonstrate that FedCompass achieves faster convergence and higher accuracy than other asynchronous algorithms while remaining more efficient than synchronous algorithms when performing federated learning on heterogeneous clients. The source code for FedCompass is available at https://github.com/APPFL/FedCompass.

cs.LG↗

arXiv2024

Secure Federated Learning Across Heterogeneous Cloud and High-Performance Computing Resources -- A Case Study on Federated Fine-tuning of LLaMA 2

Federated learning enables multiple data owners to collaboratively train robust machine learning models without transferring large or sensitive local datasets by only sharing the parameters of the locally trained models. In this paper, we elaborate on the design of our Advanced Privacy-Preserving Federated Learning (APPFL) framework, which streamlines end-to-end secure and reliable federated learning experiments across cloud computing facilities and high-performance computing resources by leveraging Globus Compute, a distributed function as a service platform, and Amazon Web Services. We further demonstrate the use case of APPFL in fine-tuning a LLaMA 2 7B model using several cloud resources and supercomputers.

cs.DC↗

arXiv2024

Two-Stage Distributionally Robust Conic Linear Programming over 1-Wasserstein Balls

This paper studies two-stage distributionally robust conic linear programming under constraint uncertainty over type-1 Wasserstein balls. We present optimality conditions for the dual of the worst-case expectation problem, which characterizes worst-case uncertain parameters for its inner maximization problem. This condition offers an alternative proof, a counter-example, and an extension to previous works. Additionally, the condition highlights the potential advantage of a specific distance metric for out-of-sample performance, as exemplified in a numerical study on a facility location problem with demand uncertainty. A cutting-plane-based algorithm and a variety of algorithmic enhancements are proposed with a finite convergence proof under less stringent assumptions.

math.OC↗

arXiv2024

QCQP-Net: Reliably Learning Feasible Alternating Current Optimal Power Flow Solutions Under Constraints

At the heart of power system operations, alternating current optimal power flow (ACOPF) studies the generation of electric power in the most economical way under network-wide load requirement, and can be formulated as a highly structured non-convex quadratically constrained quadratic program (QCQP). Optimization-based solutions to ACOPF (such as ADMM or interior-point method), as the classic approach, require large amount of computation and cannot meet the need to repeatedly solve the problem as load requirement frequently changes. On the other hand, learning-based methods that directly predict the ACOPF solution given the load input incur little computational cost but often generates infeasible solutions (i.e. violate the constraints of ACOPF). In this work, we combine the best of both worlds -- we propose an innovated framework for learning ACOPF, where the input load is mapped to the ACOPF solution through a neural network in a computationally efficient and reliable manner. Key to our innovation is a specific-purpose "activation function" defined implicitly by a QCQP and a novel loss, which enforce constraint satisfaction. We show through numerical simulations that our proposed method achieves superior feasibility rate and generation cost in situations where the existing learning-based approaches fail.

math.OC↗

arXiv2024

Distributionally Robust Decentralized Volt-Var Control with Network Reconfiguration

This paper presents a decentralized volt-var optimization (VVO) and network reconfiguration strategy to address the challenges arising from the growing integration of distributed energy resources, particularly photovoltaic (PV) generation units, in active distribution networks. To reconcile control measures with different time resolutions and empower local control centers to handle intermittency locally, the proposed approach leverages a two-stage distributionally robust optimization; decisions on slow-responding control measures and set points that link neighboring subnetworks are made in advance while considering all plausible distributions of uncertain PV outputs. We present a decomposition algorithm with an acceleration scheme for solving the proposed model. Numerical experiments on the IEEE 123 bus distribution system are given to demonstrate its outstanding out-of-sample performance and computational efficiency, which suggests that the proposed method can effectively localize uncertainty via risk-informed proactive timely decisions.

math.OC↗