arXiv subjects

Donghyun Lee

Publications and source records attributed to Donghyun Lee.

At least 55 records · Page 3Linked to original sources

arXiv2025

Spiking Transformer with Spatial-Temporal Attention

Spike-based Transformer presents a compelling and energy-efficient alternative to traditional Artificial Neural Network (ANN)-based Transformers, achieving impressive results through sparse binary computations. However, existing spike-based transformers predominantly focus on spatial attention while neglecting crucial temporal dependencies inherent in spike-based processing, leading to suboptimal feature representation and limited performance. To address this limitation, we propose Spiking Transformer with Spatial-Temporal Attention (STAtten), a simple and straightforward architecture that efficiently integrates both spatial and temporal information in the self-attention mechanism. STAtten introduces a block-wise computation strategy that processes information in spatial-temporal chunks, enabling comprehensive feature capture while maintaining the same computational complexity as previous spatial-only approaches. Our method can be seamlessly integrated into existing spike-based transformers without architectural overhaul. Extensive experiments demonstrate that STAtten significantly improves the performance of existing spike-based transformers across both static and neuromorphic datasets, including CIFAR10/100, ImageNet, CIFAR10-DVS, and N-Caltech101. The code is available at https://github.com/Intelligent-Computing-Lab-Yale/STAtten

cs.NE↗

arXiv2024

CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models

Large Language Models (LLMs) have dramatically advanced AI applications, yet their deployment remains challenging due to their immense inference costs. Recent studies ameliorate the computational costs of LLMs by increasing their activation sparsity but suffer from significant performance degradation on downstream tasks. In this work, we introduce a new framework for sparsifying the activations of base LLMs and reducing inference costs, dubbed Contextually Aware Thresholding for Sparsity (CATS). CATS is relatively simple, easy to implement, and highly effective. At the heart of our framework is a new non-linear activation function. We demonstrate that CATS can be applied to various base models, including Mistral-7B and Llama2-7B, and outperforms existing sparsification techniques in downstream task performance. More precisely, CATS-based models often achieve downstream task performance within 1-2% of their base models without any fine-tuning and even at activation sparsity levels of 50%. Furthermore, CATS-based models converge faster and display better task performance than competing techniques when fine-tuning is applied. Finally, we develop a custom GPU kernel for efficient implementation of CATS that translates the activation of sparsity of CATS to real wall-clock time speedups. Our custom kernel implementation of CATS results in a ~15% improvement in wall-clock inference latency of token generation on both Llama-7B and Mistral-7B.

cs.LG↗

arXiv2024

Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems

As Large Language Models (LLMs) grow increasingly powerful, multi-agent systems are becoming more prevalent in modern AI applications. Most safety research, however, has focused on vulnerabilities in single-agent LLMs. These include prompt injection attacks, where malicious prompts embedded in external content trick the LLM into executing unintended or harmful actions, compromising the victim's application. In this paper, we reveal a more dangerous vector: LLM-to-LLM prompt injection within multi-agent systems. We introduce Prompt Infection, a novel attack where malicious prompts self-replicate across interconnected agents, behaving much like a computer virus. This attack poses severe threats, including data theft, scams, misinformation, and system-wide disruption, all while propagating silently through the system. Our extensive experiments demonstrate that multi-agent systems are highly susceptible, even when agents do not publicly share all communications. To address this, we propose LLM Tagging, a defense mechanism that, when combined with existing safeguards, significantly mitigates infection spread. This work underscores the urgent need for advanced security measures as multi-agent LLM systems become more widely adopted.

cs.MA↗

arXiv2024

GenQ: Quantization in Low Data Regimes with Generative Synthetic Data

In the realm of deep neural network deployment, low-bit quantization presents a promising avenue for enhancing computational efficiency. However, it often hinges on the availability of training data to mitigate quantization errors, a significant challenge when data availability is scarce or restricted due to privacy or copyright concerns. Addressing this, we introduce GenQ, a novel approach employing an advanced Generative AI model to generate photorealistic, high-resolution synthetic data, overcoming the limitations of traditional methods that struggle to accurately mimic complex objects in extensive datasets like ImageNet. Our methodology is underscored by two robust filtering mechanisms designed to ensure the synthetic data closely aligns with the intrinsic characteristics of the actual training data. In case of limited data availability, the actual data is used to guide the synthetic data generation process, enhancing fidelity through the inversion of learnable token embeddings. Through rigorous experimentation, GenQ establishes new benchmarks in data-free and data-scarce quantization, significantly outperforming existing methods in accuracy and efficiency, thereby setting a new standard for quantization in low data regimes. Code is released at \url{https://github.com/Intelligent-Computing-Lab-Yale/GenQ}.

cs.CV↗

arXiv2024

ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition

Spiking Neural Networks (SNNs) have emerged as a compelling, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) for static image tasks such as image classification and segmentation. However, in the more complex video classification domain, SNN-based methods fall considerably short of ANN-based benchmarks due to the challenges in processing dense frame sequences. To bridge this gap, we propose ReSpike, a hybrid framework that synergizes the strengths of ANNs and SNNs to tackle action recognition tasks with high accuracy and low energy cost. By decomposing film clips into spatial and temporal components, i.e., RGB image Key Frames and event-like Residual Frames, ReSpike leverages ANN for learning spatial information and SNN for learning temporal information. In addition, we propose a multi-scale cross-attention mechanism for effective feature fusion. Compared to state-of-the-art SNN baselines, our ReSpike hybrid architecture demonstrates significant performance improvements (e.g., >30% absolute accuracy improvement on HMDB-51, UCF-101, and Kinetics-400). Furthermore, ReSpike achieves comparable performance with prior ANN approaches while bringing better accuracy-energy tradeoff.

cs.CV↗

arXiv2024

Decoupled Marked Temporal Point Process using Neural Ordinary Differential Equations

A Marked Temporal Point Process (MTPP) is a stochastic process whose realization is a set of event-time data. MTPP is often used to understand complex dynamics of asynchronous temporal events such as money transaction, social media, healthcare, etc. Recent studies have utilized deep neural networks to capture complex temporal dependencies of events and generate embedding that aptly represent the observed events. While most previous studies focus on the inter-event dependencies and their representations, how individual events influence the overall dynamics over time has been under-explored. In this regime, we propose a Decoupled MTPP framework that disentangles characterization of a stochastic process into a set of evolving influences from different events. Our approach employs Neural Ordinary Differential Equations (Neural ODEs) to learn flexible continuous dynamics of these influences while simultaneously addressing multiple inference problems, such as density estimation and survival rate computation. We emphasize the significance of disentangling the influences by comparing our framework with state-of-the-art methods on real-life datasets, and provide analysis on the model behavior for potential applications.

cs.LG↗

arXiv2024

The Mixed Convex-Concave effect on the regularity of a Boltzmann solution

The geometric properties of domains are well-known to be crucial factors influencing the regularity of Boltzmann boundary problems. In the presence of non-convex physical boundaries, highly singular behaviors are investigated including the propagation of discontinuities \cite{Kim11} and Hölder regularity \cite{CD2022}. In this paper, we focus on studying the $C^{0,\frac{1}{4}-}$ Hölder regularity of the Boltzmann equation within concentric cylinders where both convex and concave features are present on the boundaries. Our findings extend the previous results of \cite{CD2022}, which primarily addressed concavity while considering the exterior of convex objects.

math.AP↗

arXiv2023

BanditPAM++: Faster $k$-medoids Clustering

Clustering is a fundamental task in data science with wide-ranging applications. In $k$-medoids clustering, cluster centers must be actual datapoints and arbitrary distance metrics may be used; these features allow for greater interpretability of the cluster centers and the clustering of exotic objects in $k$-medoids clustering, respectively. $k$-medoids clustering has recently grown in popularity due to the discovery of more efficient $k$-medoids algorithms. In particular, recent research has proposed BanditPAM, a randomized $k$-medoids algorithm with state-of-the-art complexity and clustering accuracy. In this paper, we present BanditPAM++, which accelerates BanditPAM via two algorithmic improvements, and is $O(k)$ faster than BanditPAM in complexity and substantially faster than BanditPAM in wall-clock runtime. First, we demonstrate that BanditPAM has a special structure that allows the reuse of clustering information $\textit{within}$ each iteration. Second, we demonstrate that BanditPAM has additional structure that permits the reuse of information $\textit{across}$ different iterations. These observations inspire our proposed algorithm, BanditPAM++, which returns the same clustering solutions as BanditPAM but often several times faster. For example, on the CIFAR10 dataset, BanditPAM++ returns the same results as BanditPAM but runs over 10$\times$ faster. Finally, we provide a high-performance C++ implementation of BanditPAM++, callable from Python and R, that may be of interest to practitioners at https://github.com/motiwari/BanditPAM. Auxiliary code to reproduce all of our experiments via a one-line script is available at https://github.com/ThrunGroup/BanditPAM_plusplus_experiments.

cs.LG↗

arXiv2023

Dynamical Billiard and a long-time behavior of the Boltzmann equation in general 3D toroidal domains

Establishing global well-posedness and convergence toward equilibrium of the Boltzmann equation with specular reflection boundary condition has been one of the central questions in the subject of kinetic theory. Despite recent significant progress in this question when domains are strictly convex, as shown by Guo and Kim-Lee, the same question without the strict convexity of domains is still totally open in 3D. The major difficulty arises when a billiard map has an infinite number of bounces in a finite time interval or when the map fails to be Lipschitz continuous, both of which happen generically when the domain is non-convex. In this paper, we develop a new method to control a billiard map on a surface of revolution generated by revolving any planar analytic convex closed curve (e.g., typical shape of tokamak reactors' chamber). In particular, we classify and measure the size (to be small) of a pullback set (along the billiard trajectory) of the infinite-bouncing and singular-bouncing cases. As a consequence, we solve the open question affirmatively in such domains. To the best of our knowledge, this work is the first construction of global solutions to the hard-sphere Boltzmann equation in generic non-convex 3-dimensional domains. In Appendix, we introduce a novel method for constructive coercivity of a linearized collision operator $L$ when the specular boundary condition is imposed. In particular, this method works for a periodic cylindrical domain with an annulus cross-section.

math.AP↗

arXiv2023

Faster Maximum Inner Product Search in High Dimensions

Maximum Inner Product Search (MIPS) is a ubiquitous task in machine learning applications such as recommendation systems. Given a query vector and $n$ atom vectors in $d$-dimensional space, the goal of MIPS is to find the atom that has the highest inner product with the query vector. Existing MIPS algorithms scale at least as $O(\sqrt{d})$, which becomes computationally prohibitive in high-dimensional settings. In this work, we present BanditMIPS, a novel randomized MIPS algorithm whose complexity is independent of $d$. BanditMIPS estimates the inner product for each atom by subsampling coordinates and adaptively evaluates more coordinates for more promising atoms. The specific adaptive sampling strategy is motivated by multi-armed bandits. We provide theoretical guarantees that BanditMIPS returns the correct answer with high probability, while improving the complexity in $d$ from $O(\sqrt{d})$ to $O(1)$. We also perform experiments on four synthetic and real-world datasets and demonstrate that BanditMIPS outperforms prior state-of-the-art algorithms. For example, in the Movie Lens dataset ($n$=4,000, $d$=6,000), BanditMIPS is 20$\times$ faster than the next best algorithm while returning the same answer. BanditMIPS requires no preprocessing of the data and includes a hyperparameter that practitioners may use to trade off accuracy and runtime. We also propose a variant of our algorithm, named BanditMIPS-$α$, which achieves further speedups by employing non-uniform sampling across coordinates. Finally, we demonstrate how known preprocessing techniques can be used to further accelerate BanditMIPS, and discuss applications to Matching Pursuit and Fourier analysis.

cs.LG↗

arXiv2023

Large amplitude problem of BGK model: Relaxation to quadratic nonlinearity

Bhatnagar-Gross-Krook (BGK) equation is a relaxation model of the Boltzmann equation which is widely used in place of the Boltzmann equation for the simulation of various kinetic flow problems. In this work, we study the asymptotic stability of the BGK model when the initial data is not necessarily close to the global equilibrium pointwisely. Due to the highly nonlinear structure of the relaxation operator, the argument developed to derive the bootstrap estimate for the Boltzmann equation leads to a weaker estimate in the case of the BGK model, which does not exclude the possible blow-up of the perturbation. To overcome this issue, we carry out a refined analysis of the macroscopic fields to guarantee that the system transits from a highly nonlinear regime into a quadratic nonlinear regime after a long but finite time, in which the highly nonlinear perturbative term relaxes to essentially quadratic nonlinearity.

math.AP↗

arXiv2022

Hölder regularity of the Boltzmann equation past an obstacle

Regularity and singularity of the solutions according to the shape of domains is a challenging research theme in the Boltzmann theory (\cite{Kim11,GKTT1}). In this paper, we prove an Hölder regularity in $C^{0,\frac{1}{2}-}_{x,v}$ for the Boltzmann equation of the hard-sphere molecule, which undergoes the elastic reflection in the intermolecular collision and the contact with the boundary of a convex obstacle. In particular, this Hölder regularity result is a stark contrast to the case of other physical boundary conditions (such as the diffuse reflection boundary condition and in-flow boundary condition), for which the solutions of the Boltzmann equation develop discontinuity in a codimension 1 subset (\cite{Kim11}), and therefore the best possible regularity is BV, which has been proved in \cite{GKTT2}.

math.AP↗

arXiv2022

High-velocity tails of the inelastic and the multi-species mixture Boltzmann equations

We study high-velocity tails of some homogeneous Boltzmann equations on $v \in \mathbb{R}_{v}^d$. First, we consider spatially homogeneous inelastic Boltzmann equation with noncutoff collision kernel, in the case of moderately soft potentials. We also study spatially homogeneous mixture Boltzmann equations : for both noncutoff collision kernel with moderately soft potentials and cutoff collision kernel with hard potentials. In the case of noncutoff inelastic Boltzmann, we obtain \[ f(t,v) \geq a(t) e^{-b(t)|v|^p}, \quad 2 < p < 6.213 \] by extending Cancellation lemma and spreading lemma and assuming $f\in C^{\infty}$. For the Mixture type Boltzmann equations, we prove Maxwellian $p=2$.

math.AP↗

arXiv2022

On $C^2$ solution of the free-transport equation in a disk

The free transport operator of probability density function $f(t,x,v)$ is one the most fundamental operator which is widely used in many areas of PDE theory including kinetic theory, in particular. When it comes to general boundary problems in kinetic theory, however, it is well-known that high order regularity is very hard to obtain in general. In this paper, we study the free transport equation in a disk with the specular boundary condition. We obtain initial-boundary compatibility conditions for $C^1_{t,x,v}$ and $C^2_{t,x,v}$ regularity of the solution. We also provide regularity estimates.

math.AP↗

arXiv2021

The large amplitude solution of the Boltzmann equation with soft potential

In this paper, we deal with (angular cut-off) Boltzmann equation with soft potential ($-3<γ<0$). In particular, we construct a unique global solution in $L^\infty_{x,v}$ which converges to global equilibrium asymptotically provided that initial data has a large amplitude but with sufficiently small relative entropy. Because frequency multiplier is not uniformly positive anymore, unlike hard potential case, time-involved velocity weight will be used to derive sub-exponential decay of the solution. Motivated by recent development of $L^2\mbox{-}L^\infty$ approach also, we introduce some modified estimates of quadratic nonlinear terms. Linearized collision kernel will be treated in a subtle manner to control singularity of soft potential kernel.

math.AP↗

arXiv2021

Data-free mixed-precision quantization using novel sensitivity metric

Post-training quantization is a representative technique for compressing neural networks, making them smaller and more efficient for deployment on edge devices. However, an inaccessible user dataset often makes it difficult to ensure the quality of the quantized neural network in practice. In addition, existing approaches may use a single uniform bit-width across the network, resulting in significant accuracy degradation at extremely low bit-widths. To utilize multiple bit-width, sensitivity metric plays a key role in balancing accuracy and compression. In this paper, we propose a novel sensitivity metric that considers the effect of quantization error on task loss and interaction with other layers. Moreover, we develop labeled data generation methods that are not dependent on a specific operation of the neural network. Our experiments show that the proposed metric better represents quantization sensitivity, and generated data are more feasible to be applied to mixed-precision quantization.

cs.LG↗

arXiv2020

The Boltzmann equation with large-amplitude initial data and specular reflection boundary condition

For the Boltzmann equation with cutoff hard potentials, we construct the unique global solution converging with an exponential rate in large time to global Maxwellians not only for the specular reflection boundary condition with the bounded convex C^3 domain but also for a class of large amplitude initial data where the L^infty norm with a suitable velocity weight can be arbitrarily large but the relative entropy need to be small. A key point in the proof is to introduce a delicate nonlinear iterative process of estimating the gain term basing on the triple Duhamel iteration along the linearized dynamics.

math.AP↗

arXiv2019

Global strong solutions of the Vlasov-Poisson-Boltzmann system in bounded domains

When dilute charged particles are confined in a bounded domain, boundary effects are crucial in the global dynamics. We construct a unique global-in-time solution to the Vlasov-Poisson-Boltzmann system in convex domains with the diffuse boundary condition. The construction is based on an $L^{2}$-$L^{\infty}$ framework with a novel \textit{nonlinear-normed energy estimate} of a distribution function in weighted $W^{1,p}$-spaces and a $C^{2,δ}$-estimate of the self-consistent electric potential. Moreover we prove an exponential convergence of the distribution function toward the global Maxwellian.

math.AP↗