arXiv subjects

Deokjae Lee

Publications and source records attributed to Deokjae Lee.

At least 19 recordsLinked to original sources

arXiv2026

Q-Strata: Hierarchical Bit Allocation for Mixed-Precision Quantization of Mixture-of-Experts LLMs

Mixed-precision quantization (MPQ) assigns a different bitwidth to each linear layer of a large language model (LLM) to minimize the quantization-induced quality loss under a fixed budget, but Mixture-of-Experts (MoE) models contain these layers in every expert of every MoE block, so the allocation space grows far larger than in a dense model. Existing methods either allocate within each block under a uniform per-block budget, or allocate across blocks through an additive proxy, and neither directly optimizes a model-level objective over the choices that couple the blocks. We propose Q-Strata, a bi-level allocator that ranks within-block assignments with a cheap proxy and allocates across blocks with a model-level objective evaluated on the assembled quantized model. Its inner stage caches a Pareto frontier of candidates per block over finely spaced budgets, leaving the outer stage to set one budget per block instead of a bitwidth for every linear layer. With the search reduced to one budget per block, the outer stage optimizes this model-level objective directly, capturing the inter-block coupling that additive proxies miss. On Mixtral-8x7B-Instruct, Qwen1.5-MoE-A2.7B, and DeepSeek-V2-Lite, Q-Strata consistently achieves lower WikiText2 perplexity than uniform-bitwidth GPTQ and the state-of-the-art MoE MPQ methods MxMoE and GEMQ in the low-bit regime. The code is available at https://github.com/snu-mllab/Q-Strata/tree/main.

cs.LG↗

arXiv2025

Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment

We study weight-only post-training quantization (PTQ), which quantizes the weights of a large language model (LLM) without retraining, using little or no calibration data. Weight-only PTQ is crucial for reducing the memory footprint and latency of LLM inference, especially in memory-bound, small-batch inference scenarios, such as personalized inference on edge devices. Despite its importance, irregular weight distributions with heavy-tailed outliers in LLMs complicate quantization, recently motivating rotation-based methods that transform weights into near-Gaussian distributions, which are more regular with fewer outliers, thereby reducing quantization error. In this work, we first derive the information-theoretically optimal bit allocation for Gaussianized weights under given bit budgets, revealing that fine-grained fractional-bit quantizers approaching the Gaussian distortion-rate bound are essential to achieve near-optimal quantization performance. To bridge this theoretical insight and practical implementation, we introduce Q-Palette, a versatile collection of fractional-bit quantizers that range from trellis-coded quantizers offering near-optimal distortion to simpler vector and scalar quantizers optimized for faster inference, all efficiently implemented with optimized CUDA kernels across various bitwidths. Furthermore, leveraging Q-Palette as a foundational component, we propose a novel mixed-scheme quantization framework, jointly optimizing quantizer choices and layer fusion decisions given resource constraints. The code is available at https://github.com/snu-mllab/Q-Palette.

cs.LG↗

arXiv2025

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance

Post-training quantization is a key technique for reducing the memory and inference latency of large language models by quantizing weights and activations without requiring retraining. However, existing methods either (1) fail to account for the varying importance of hidden features to the end loss or, when incorporating end loss, (2) neglect the critical interactions between model weights. To address these limitations, we propose GuidedQuant, a novel quantization approach that integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies within output channels. GuidedQuant consistently boosts the performance of state-of-the-art quantization methods across weight-only scalar, weight-only vector, and weight-and-activation quantization. Additionally, we introduce a novel non-uniform scalar quantization algorithm, which is guaranteed to monotonically decrease the quantization objective value, and outperforms existing methods in this category. We release the code at https://github.com/snu-mllab/GuidedQuant.

cs.LG↗

arXiv2024

Training Greedy Policy for Proposal Batch Selection in Expensive Multi-Objective Combinatorial Optimization

Active learning is increasingly adopted for expensive multi-objective combinatorial optimization problems, but it involves a challenging subset selection problem, optimizing the batch acquisition score that quantifies the goodness of a batch for evaluation. Due to the excessively large search space of the subset selection problem, prior methods optimize the batch acquisition on the latent space, which has discrepancies with the actual space, or optimize individual acquisition scores without considering the dependencies among candidates in a batch instead of directly optimizing the batch acquisition. To manage the vast search space, a simple and effective approach is the greedy method, which decomposes the problem into smaller subproblems, yet it has difficulty in parallelization since each subproblem depends on the outcome from the previous ones. To this end, we introduce a novel greedy-style subset selection algorithm that optimizes batch acquisition directly on the combinatorial space by sequential greedy sampling from the greedy policy, specifically trained to address all greedy subproblems concurrently. Notably, our experiments on the red fluorescent proteins design task show that our proposed method achieves the baseline performance in 1.69x fewer queries, demonstrating its efficiency.

cs.LG↗

arXiv2023

Efficient Latency-Aware CNN Depth Compression via Two-Stage Dynamic Programming

Recent works on neural network pruning advocate that reducing the depth of the network is more effective in reducing run-time memory usage and accelerating inference latency than reducing the width of the network through channel pruning. In this regard, some recent works propose depth compression algorithms that merge convolution layers. However, the existing algorithms have a constricted search space and rely on human-engineered heuristics. In this paper, we propose a novel depth compression algorithm which targets general convolution operations. We propose a subset selection problem that replaces inefficient activation layers with identity functions and optimally merges consecutive convolution operations into shallow equivalent convolution operations for efficient end-to-end inference latency. Since the proposed subset selection problem is NP-hard, we formulate a surrogate optimization problem that can be solved exactly via two-stage dynamic programming within a few seconds. We evaluate our methods and baselines by TensorRT for a fair inference latency comparison. Our method outperforms the baseline method with higher accuracy and faster inference speed in MobileNetV2 on the ImageNet dataset. Specifically, we achieve $1.41\times$ speed-up with $0.11$\%p accuracy gain in MobileNetV2-1.0 on the ImageNet.

cs.LG↗

arXiv2023

Query-Efficient Black-Box Red Teaming via Bayesian Optimization

The deployment of large-scale generative models is often restricted by their potential risk of causing harm to users in unpredictable ways. We focus on the problem of black-box red teaming, where a red team generates test cases and interacts with the victim model to discover a diverse set of failures with limited query access. Existing red teaming methods construct test cases based on human supervision or language model (LM) and query all test cases in a brute-force manner without incorporating any information from past evaluations, resulting in a prohibitively large number of queries. To this end, we propose Bayesian red teaming (BRT), novel query-efficient black-box red teaming methods based on Bayesian optimization, which iteratively identify diverse positive test cases leading to model failures by utilizing the pre-defined user input pool and the past evaluations. Experimental results on various user input pools demonstrate that our method consistently finds a significantly larger number of diverse positive test cases under the limited query budget than the baseline methods. The source code is available at https://github.com/snu-mllab/Bayesian-Red-Teaming.

cs.AI↗

arXiv2022

Query-Efficient and Scalable Black-Box Adversarial Attacks on Discrete Sequential Data via Bayesian Optimization

We focus on the problem of adversarial attacks against models on discrete sequential data in the black-box setting where the attacker aims to craft adversarial examples with limited query access to the victim model. Existing black-box attacks, mostly based on greedy algorithms, find adversarial examples using pre-computed key positions to perturb, which severely limits the search space and might result in suboptimal solutions. To this end, we propose a query-efficient black-box attack using Bayesian optimization, which dynamically computes important positions using an automatic relevance determination (ARD) categorical kernel. We introduce block decomposition and history subsampling techniques to improve the scalability of Bayesian optimization when an input sequence becomes long. Moreover, we develop a post-optimization algorithm that finds adversarial examples with smaller perturbation size. Experiments on natural language and protein classification tasks demonstrate that our method consistently achieves higher attack success rate with significant reduction in query count and modification rate compared to the previous state-of-the-art methods.

cs.LG↗

arXiv2022

Optimal channel selection with discrete QCQP

Reducing the high computational cost of large convolutional neural networks is crucial when deploying the networks to resource-constrained environments. We first show the greedy approach of recent channel pruning methods ignores the inherent quadratic coupling between channels in the neighboring layers and cannot safely remove inactive weights during the pruning procedure. Furthermore, due to these inactive weights, the greedy methods cannot guarantee to satisfy the given resource constraints and deviate with the true objective. In this regard, we propose a novel channel selection method that optimally selects channels via discrete QCQP, which provably prevents any inactive weights and guarantees to meet the resource constraints tightly in terms of FLOPs, memory usage, and network size. We also propose a quadratic model that accurately estimates the actual inference time of the pruned network, which allows us to adopt inference time as a resource constraint option. Furthermore, we generalize our method to extend the selection granularity beyond channels and handle non-sequential connections. Our experiments on CIFAR-10 and ImageNet show our proposed pruning method outperforms other fixed-importance channel pruning methods on various network architectures.

cs.CV↗

arXiv2021

Homological percolation transitions in growing simplicial complexes

Simplicial complex (SC) representation is an elegant mathematical framework for representing the effect of complexes or groups with higher-order interactions in a variety of complex systems ranging from brain networks to social relationships. Here, we explore the homological percolation transitions (HPTs) of growing SCs using empirical datasets and a model proposed. The HPTs are determined by the first and second Betti numbers, which indicate the appearance of one- and two-dimensional macroscopic-scale homological cycles and cavities, respectively. A minimal SC model with two essential factors, namely, growth and preferential attachment, is proposed to model social coauthorship relationships. This model successfully reproduces the HPTs and determines the transition types as infinite order (the Berezinskii--Kosterlitz--Thouless type) with different critical exponents. In contrast to the Kahle localization observed in static random SCs, the first Betti number continues to increase even after the second Betti number appears. This delocalization is found to stem from the two aforementioned factors and arises when the merging rate of two-dimensional simplexes is less than the birth rate of isolated simplexes. Our results can provide topological insight into the maturing steps of complex networks such as social and biological networks.

physics.soc-ph↗

arXiv2019

Entropy production and fluctuation theorems on complex networks

Entropy production (EP) is known as a fundamental quantity for measuring the irreversibility of processes in thermal equilibrium and states far from equilibrium. In stochastic thermodynamics, the EP becomes more visible in terms of the probability density functions of the trajectories of a particle in the state space. Inspired by a previous result that complex networks can serve as state spaces, we consider a data packet transport problem on complex networks. Entropy is produced owing to the complexity of pathways as the packet travels back and forth between two nodes. The EPs are exactly enumerated along the shortest paths between every pair of nodes, and the functional form of the EP distribution is determined by extreme value analysis. The asymptote of the accumulated EP distribution is found to follow the Gumbel distribution.

cond-mat.stat-mech↗

arXiv2019

Interevent time distribution, burst, and hybrid percolation transition

Critical phenomena of a second-order percolation transition are known to be independent of cluster merging or pruning process. However, those of a hybrid percolation transition (HPT), mixed properties of both first-order and second-order transitions, depend on the processes. The HPT induced by cluster merging is more intrigue and little understood than the other. Here, we construct a theoretical framework using the so-called restricted percolation model. In this model, clusters are ranked by size and partitioned into small- and large-cluster sets. As the cluster rankings are updated by cluster coalescence, clusters may move back and forth across the set boundary. The inter-event time (IET) between two consecutive crossing times have two distributions with power-law decays, which in turn characterize the criticality of the HPT. A burst of such crossing events occurs and signals the upcoming transition. We discuss a related phenomenon to this critical dynamics.

cond-mat.stat-mech↗

arXiv2018

Recent advances of percolation theory in complex networks

During the past two decades, percolation has long served as a basic paradigm for network resilience, community formation and so on in complex systems. While the percolation transition is known as one of the most robust continuous transitions, the percolation transitions occurring in complex systems are often of different types such as discontinuous, hybrid, and infinite-order phase transitions. Thus, percolation has received considerable attention in network science community. Here we present a very brief review of percolation theory recently developed, which includes those types of phase transitions, critical phenomena, and finite-size scaling theory. Moreover, we discuss potential applications of theoretical results and several open questions including universal behaviors.

physics.soc-ph↗

arXiv2018

Two golden times in two-step contagion models

The two-step contagion model is a simple toy model for understanding pandemic outbreaks that occur in the real world. The model takes into account that a susceptible person either gets immediately infected or weakened when getting into contact with an infectious one. As the number of weakened people increases, they eventually can become infected in a short time period and a pandemic outbreak occurs. The time required to reach such a pandemic outbreak allows for intervention and is often called golden time. Understanding the size-dependence of the golden time is useful for controlling pandemic outbreak. Here we find that there exist two types of golden times in the two-step contagion model, which scale as $O(N^{1/3})$ and $O(N^ζ)$ with the system size $N$ on Erdős-Rényi networks, where the measured $ζ$ is slightly larger than $1/4$. They are distinguished by the initial number of infected nodes, $o(N)$ and $O(N)$, respectively. While the exponent $1/3$ of the $N$-dependence of the golden time is universal even in other models showing discontinuous transitions induced by cascading dynamics, the measured $ζ$ exponents are all close to $1/4$ but show model-dependence. It remains open whether or not $ζ$ reduces to $1/4$ in the asymptotically large-$N$ limit.

q-bio.PE↗

arXiv2017

Critical phenomena of a hybrid phase transition in cluster merging dynamics

Recently, a hybrid percolation transitions (HPT) that exhibits both a discontinuous transition and critical behavior at the same transition point has been observed in diverse complex systems. In spite of considerable effort to develop the theory of HPT, it is still incomplete, particularly when the transition is induced by cluster merging dynamics. Here, we aim to develop a theoretical framework of the HPT induced by such dynamics. We find that two correlation-length exponents are necessary for characterizing the giant cluster and finite clusters, respectively. Finite-size scaling method for the HPT is also introduced. The conventional formula of the fractal dimension in terms of the critical exponents is not valid. Neither the giant nor finite clusters are fractals but they have fractal boundaries.

cond-mat.stat-mech↗

arXiv2017

Critical behavior of a two-step contagion model with multiple seeds

A two-step contagion model with a single seed serves as a cornerstone for understanding the critical behaviors and underlying mechanism of discontinuous percolation transitions induced by cascade dynamics. When the contagion spreads from a single seed, a cluster of infected and recovered nodes grows without any cluster merging process. However, when the contagion starts from multiple seeds of $O(N)$ where $N$ is the system size, a node weakened by a seed can be infected more easily when it is in contact with another node infected by a different pathogen seed. This contagion process can be viewed as a cluster merging process in a percolation model. Here, we show analytically and numerically that when the density of infectious seeds is relatively small but $O(1)$, the epidemic transition is hybrid, exhibiting both continuous and discontinuous behavior, whereas when it is sufficiently large and reaches a critical point, the transition becomes continuous. We determine the full set of critical exponents describing the hybrid and the continuous transitions. Their critical behaviors differ from those in the single-seed case.

cond-mat.stat-mech↗

arXiv2017

Universal mechanism for hybrid percolation transitions

Hybrid percolation transitions (HPTs) induced by cascading processes have been observed in diverse complex systems such as $k$-core percolation, breakdown on interdependent networks and cooperative epidemic spreading models. Much effort has been devoted to describe the properties of HPTs of individual systems. Yet the fundamental question about the possible universal mechanism underlying those HPTs has not been investigated at a microscopic level. Here, we find that the discontinuity in the order parameter in such HPTs results from two steps: a durable critical branching (CB) and an explosive, supercritical (SC) process. In a random network of $N$ nodes at the transition the CB process persists for $O(N^{1/3})$ time and the remaining nodes become vulnerable. Those vulnerable nodes are activated then in the short SC process. This crossover mechanism and scaling behavior are universal for different HPT systems.

physics.soc-ph↗

arXiv2016

Diverse types of percolation transitions

Percolation has long served as a model for diverse phenomena and systems. The percolation transition, that is, the formation of a giant cluster on a macroscopic scale, is known as one of the most robust continuous transitions. Recently, however, many abrupt percolation transitions have been observed in complex systems. To illustrate such phenomena, considerable effort has been made to introduce models and construct theoretical frameworks for explosive, discontinuous, and hybrid percolation transitions. Experimental results have also been reported. In this review article, we describe such percolation models, their critical behaviors and universal features, and real-world phenomena.

cond-mat.stat-mech↗

arXiv2016

Critical behavior of $k$-core percolation: Numerical studies

$k$-Core percolation has served as a paradigmatic model of discontinuous percolation for a long time. Recently it was revealed that the order parameter of $k$-core percolation of random networks additionally exhibits critical behavior. Thus $k$-core percolation exhibits a hybrid phase transition. Unlike the critical behaviors of ordinary percolation that are well understood, those of hybrid percolation transitions have not been thoroughly understood yet. Here, we investigate the critical behavior of $k$-core percolation of Erdős-Rényi networks. We find numerically that the fluctuations of the order parameter and the mean avalanche size diverge in different ways. Thus, we classify the critical exponents into two types: those associated with the order parameter and those with finite avalanches. The conventional scaling relations hold within each set, however, these two critical exponents are coupled. Finally we discuss some universal features of the critical behaviors of $k$-core percolation and the cascade failure model on multiplex networks.

cond-mat.stat-mech↗