Searcharxiv⌕ Search

arXiv subjects

Minjae Park

Publications and source records attributed to Minjae Park.

12 recordsLinked to original sources

One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation

Reordering multiple-choice options can change a model's answer, an effect studied as selection bias (a preference for certain option letters) and as order sensitivity (changes in answers across option orderings). We connect these perspectives through the pooled selection-rate standard deviation (pooled SD), which measures how unevenly option letters are selected across questions and orderings. When every option of each question appears equally often at each position, the pooled SD is exactly the product of two label-free factors: the flip rate and the positional skew. The flip rate is the fraction of questions whose chosen option changes across orderings; the positional skew is the standard deviation of option-letter selection rates among those questions. Across 41 checkpoints and 12 benchmarks, the flip rate tracks accuracy and output uncertainty, while the positional skew tracks the estimated option-symbol prior. Prior correction for selection bias mainly reduces the positional skew. Aggregating each ordering with its reversal, a mitigation for order sensitivity, mainly reduces the flip rate. Both reduce the range of accuracy across correct-answer letters by similar amounts, but aggregation leaves fewer model-rank reversals. Applying prior correction before aggregation reduces both factors and further narrows this accuracy range. However, aggregation lowers the flip rate partly by settling questions on the same wrong option. We therefore recommend reporting both factors to distinguish mitigation effects, alongside accuracy to distinguish stability from correctness.

cs.CL↗

When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation

Data pruning reduces the training cost of knowledge distillation (KD). However, the preferred teacher capacity changes with the data budget: smaller teachers can outperform larger ones when limited training data are available. Understanding what drives this shift is important not only for teacher choice but also for identifying which samples are useful for distillation. We analyze teacher supervision by decomposing it into relational ordering---the ranking of classes---and score geometry---the magnitudes and margins of class probabilities---and show that the small-teacher advantage in the low-data regime arises not only from score geometry but also from relational ordering. Beyond understanding teacher capacity, our analysis reveals two properties of effective subsets: samples should match the difficulty appropriate for the available budget, and their relational signals should be diverse rather than redundant. Based on these findings, we propose DVA (Difficulty- and Volume-Aware data selection for KD), a training-dynamics-free method, which uses a small teacher as a proxy for budget-aware difficulty filtering and class-conditional relational volume maximization. Despite requiring no training dynamics statistics, our method remains competitive with training-dynamics-based methods while consistently outperforming training-dynamics-free baselines.

cs.LG↗

GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache

The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory. Vector Quantization (VQ), particularly Residual Quantization (RQ), is a promising approach for pushing KV cache storage toward the sub-1-bit regime by progressively encoding residuals with small codebooks. However, most VQ methods still rely on standard $\ell_2$ $K$-means as the core codebook-learning primitive. We identify a subtle high-dimensional issue of this primitive: Euclidean centroid averaging can induce centroid shrinkage, which weakens the angular alignment term in the $\ell_2$ distortion and makes directional preservation harder. To address this issue, we propose Gain-Shape $K$-means (GSKM), a drop-in replacement for $K$-means that improves directional fidelity while matching, and in some regimes improving, $\ell_2$ distortion. We then build Gain-Shape Residual Quantization (GSRQ) by incorporating a weighted extension of GSKM into an RQ pipeline. On LLaMA-3-8B, GSRQ substantially improves over strong KV cache quantization baselines across bit rates. At 1-bit, it improves the average accuracy across LongBench tasks from 11.34 to 33.54, a gain of 22.20 percentage points over VQLLM.

cs.LG↗

Random surfaces and lattice Yang-Mills

We study Wilson loop expectations in lattice Yang-Mills models with a compact Lie group $G$. Using tools recently introduced in a companion paper, we provide alternate derivations, interpretations, and generalizations of several recent theorems about Brownian motion limits (Dahlqvist), lattice string trajectories (Chatterjee and Jafarov) and surface sums (Magee and Puder). We show further that one can express Wilson loop expectations as sums over embedded planar maps in a manner that applies to any matrix dimension $N \geq 1$, any inverse temperature $β>0$, and any lattice dimension $d \geq 2$. When $G=\mathrm{U}(N)$, the embedded maps we consider are pairs $(\mathcal M, ϕ)$ where $\mathcal M$ is a planar (or higher genus) map and $ϕ$ is a graph homomorphism from $\mathcal M$ to a lattice such as $\mathbb Z^d$. The faces of $\mathcal M$ come in two partite classes: $\textit{edge-faces}$ (each mapped by $ϕ$ onto a single edge) and $\textit{plaquette-faces}$ (each mapped by $ϕ$ onto a single plaquette). The weight of a lattice edge $e$ is the Weingarten function applied to the partition whose parts are given by half the boundary lengths of the faces in $ϕ^{-1}(e)$. (The Weingarten function becomes quite simple in the $N\to \infty$ limit.) The overall weight of an embedded map is proportional to $N^χ$ (where $χ$ is the Euler characteristic) times the product of the edge weights. We establish analogous results for $\mathrm{SU}(N)$, $\mathrm{O}(N)$, $\mathrm{SO}(N)$, and $\mathrm{Sp}(N/2)$, where the embedded surfaces and weights take a different form. There are several variants of these constructions. In this context, we present a list of relevant open problems spanning several disciplines: random matrix theory, representation theory, statistical physics, and the theory of random surfaces, including random planar maps and Liouville quantum gravity.

math.PR↗

Prompt-based Depth Pruning of Large Language Models

Depth pruning aims to reduce the inference cost of a large language model without any hardware-specific complications, by simply removing several less important transformer blocks. However, our empirical findings suggest that the importance of a transformer block may be highly task-dependent -- a block that is crucial for a task can be removed without degrading the accuracy on another task. Based on this observation, we develop a dynamic depth pruning algorithm, coined PuDDing (Prompt-routed Dynamic Depth Pruning), which determines which blocks to omit from the model based on the input prompt. PuDDing operates by training a lightweight router to predict the best omission set among a set of options, where this option set has also been constructed in a data-driven manner. Empirical results on commonsense reasoning benchmarks demonstrate that PuDDing effectively accelerates the inference language models, and achieves better on-task performance than static depth pruning baselines.

cs.CL↗

Wilson loop expectations as sums over surfaces on the plane

Although lattice Yang-Mills theory on finite subgraphs of $\mathbb Z^d$ is easy to rigorously define, the construction of a satisfactory continuum theory on $\mathbb R^d$ is a major open problem when $d \geq 3$. Such a theory should in some sense assign a Wilson loop expectation to each suitable finite collection $\mathcal L$ of loops in $\mathbb R^d$. One classical approach is to try to represent this expectation as a sum over surfaces with boundary $\mathcal L$. There are some formal/heuristic ways to make sense of this notion, but they typically yield an ill-defined difference of infinities. In this paper, we show how to make sense of Yang-Mills integrals as surface sums for $d=2$, where the continuum theory is more accessible. Applications include several new explicit calculations, a new combinatorial interpretation of the master field, and a new probabilistic proof of the Makeenko-Migdal equation.

math.PR↗

On the geometry of uniform meandric systems

A meandric system of size $n$ is the set of loops formed from two arc diagrams (non-crossing perfect matchings) on $\{1,\dots,2n\}$, one drawn above the real line and the other below the real line. A uniform random meandric system can be viewed as a random planar map decorated by a Hamiltonian path (corresponding to the real line) and a collection of loops (formed by the arcs). Based on physics heuristics and numerical evidence, we conjecture that the scaling limit of this decorated random planar map is given by an independent triple consisting of a Liouville quantum gravity (LQG) sphere with parameter $γ=\sqrt 2$, a Schramm-Loewner evolution (SLE) curve with parameter $κ=8$, and a conformal loop ensemble (CLE) with parameter $κ=6$. We prove several rigorous results which are consistent with this conjecture. In particular, a uniform meandric system admits loops of nearly macroscopic graph-distance diameter with high probability. Furthermore, a.s., the uniform infinite meandric system with boundary has no infinite path. But, a.s., its boundary-modified version has a unique infinite path whose scaling limit is conjectured to be chordal SLE$_6$.

math.PR↗

Brownian loops on non-smooth surfaces and the Polyakov-Alvarez formula

Let $ρ$ be compactly supported on $D \subset \mathbb R^2$. Endow $\mathbb R^2$ with the metric $e^ρ(dx_1^2 + dx_2^2)$. As $δ\to 0$ the set of Brownian loops centered in $D$ with length at least $δ$ has measure $$\frac{\text{area}(D)}{2πδ} + \frac{1}{48π}(ρ,ρ)_{\nabla}+ o(1).$$ When $ρ$ is smooth, this follows from the classical Polyakov-Alvarez formula. We show that the above also holds if $ρ$ is not smooth, e.g. if $ρ$ is only Lipschitz. This fact can alternatively be expressed in terms of heat kernel traces, eigenvalue asymptotics, or zeta regularized determinants. Variants of this statement apply to more general non-smooth manifolds on which one considers all loops (not only those centered in a domain $D$). We also show that the $o(1)$ error is uniform for any family of $ρ$ satisfying certain conditions. This implies that if we weight a measure $ν$ on this family by the ($δ$-truncated) Brownian loop soup partition function, and take the vague $δ\to 0$ limit, we obtain a measure whose Radon-Nikodym derivative with respect to $ν$ is $\exp\bigl( \frac{1}{48π}(ρ,ρ)_{\nabla}\bigr)$. When the measure is a certain regularized Liouville quantum gravity measure, a companion work [APPS20] shows that this weighting has the effect of changing the so-called central charge of the surface.

math.PR↗

Heterogeneous Graph Neural Network for Personalized Session-Based Recommendation with User-Session Constraints

The recommendation system provides users with an appropriate limit of recent online large amounts of information. Session-based recommendation, a sub-area of recommender systems, attempts to recommend items by interpreting sessions that consist of sequences of items. Recently, research to include user information in these sessions is progress. However, it is difficult to generate high-quality user representation that includes session representations generated by user. In this paper, we consider various relationships in graph created by sessions through Heterogeneous attention network. Constraints also force user representations to consider the user's preferences presented in the session. It seeks to increase performance through additional optimization in the training process. The proposed model outperformed other methods on various real-world datasets.

cs.IR↗

Cross-view Self-Supervised Learning on Heterogeneous Graph Neural Network via Bootstrapping

Heterogeneous graph neural networks can represent information of heterogeneous graphs with excellent ability. Recently, self-supervised learning manner is researched which learns the unique expression of a graph through a contrastive learning method. In the absence of labels, this learning methods show great potential. However, contrastive learning relies heavily on positive and negative pairs, and generating high-quality pairs from heterogeneous graphs is difficult. In this paper, in line with recent innovations in self-supervised learning called BYOL or bootstrapping, we introduce a that can generate good representations without generating large number of pairs. In addition, paying attention to the fact that heterogeneous graphs can be viewed from two perspectives, network schema and meta-path views, high-level expressions in the graphs are captured and expressed. The proposed model showed state-of-the-art performance than other methods in various real world datasets.

cs.LG↗

Large deviations of radial SLE$_{\infty}$

We derive the large deviation principle for radial Schramm-Loewner evolution ($\operatorname{SLE}$) on the unit disk with parameter $κ\rightarrow \infty$. Restricting to the time interval $[0,1]$, the good rate function is finite only on a certain family of Loewner chains driven by absolutely continuous probability measures $\{ϕ_t^2 (ζ)\, dζ\}_{t \in [0,1]}$ on the unit circle and equals $\int_0^1 \int_{S^1} |ϕ_t'|^2/2\,dζ\,dt$. Our proof relies on the large deviation principle for the long-time average of the Brownian occupation measure by Donsker and Varadhan.

math.PR↗

Brownian loops and the central charge of a Liouville random surface

We explore the geometric meaning of the so-called zeta-regularized determinant of the Laplace-Beltrami operator on a compact surface, with or without boundary. We relate the $(-c/2)$-th power of the determinant of the Laplacian to the appropriately regularized partition function of a Brownian loop soup of intensity $c$ on the surface. This means that, in a certain sense, decorating a random surface by a Brownian loop soup of intensity $c$ corresponds to weighting the law of the surface by the $(-c/2)$-th power of the determinant of the Laplacian. Next, we introduce a method of regularizing a Liouville quantum gravity (LQG) surface (with some matter central charge parameter $\mathbf{c}$) to produce a smooth surface. And we show that weighting the law of this random surface by the $( -\mathbf{c}'/ 2)$-th power of the Laplacian determinant has precisely the effect of changing the matter central charge from $\mathbf{c}$ to $\mathbf{c} + \mathbf{c}'$. Taken together with the earlier results, this provides a way of interpreting an LQG surface of matter central charge $\mathbf{c}$ as a pure LQG surface decorated by a Brownian loop soup of intensity $\mathbf{c}$. Building on this idea, we present several open problems about random planar maps and their continuum analogs. Although the original construction of LQG is well-defined only for $\mathbf{c}\leq 1$, some of the constructions and questions also make sense when $\mathbf{c}>1$.

math.PR↗