arXiv subjects

Adam D. Jozefiak

Publications and source records attributed to Adam D. Jozefiak.

3 recordsLinked to original sources

arXiv2026

The Value of Covariance Matching in Gaussian DDPMs and the Lanczos Sampler

A central error measure in Gaussian DDPMs is the path-space KL divergence between the exact reverse chain and the learned Gaussian reverse process. This quantity is especially relevant for procedures such as classifier guidance, which perturb the entire reverse trajectory rather than only the terminal sample. Prior analyses show that standard isotropic reverse covariances suffer an unavoidable $\Omega(1/T)$ path-KL error as the number of denoising steps $T$ grows. We show that matching the full posterior covariance breaks this barrier, yielding an order-wise improvement that reduces the path KL to $O(1/T^2)$. To make full covariance matching practical, we introduce the Lanczos Gaussian sampler (LGS), a training-free, matrix-free method for sampling from the optimal reverse covariance using only covariance-vector products, which are available through Jacobian-vector products of the posterior mean. LGS avoids dense covariance storage and auxiliary covariance models. We prove that LGS approximation error decays exponentially in the number of Lanczos steps, where each Lanczos step requires a single Jacobian-vector product. Empirically, using only just three such steps improves sample quality over strong diagonal-covariance baselines, including OCM-DDPM, across standard image benchmarks. This identifies full covariance matching as both theoretically valuable and practically accessible for fast DDPM sampling.

cs.LG↗

arXiv2024

Self-Normalized Resets for Plasticity in Continual Learning

Plasticity Loss is an increasingly important phenomenon that refers to the empirical observation that as a neural network is continually trained on a sequence of changing tasks, its ability to adapt to a new task diminishes over time. We introduce Self-Normalized Resets (SNR), a simple adaptive algorithm that mitigates plasticity loss by resetting a neuron's weights when evidence suggests its firing rate has effectively dropped to zero. Across a battery of continual learning problems and network architectures, we demonstrate that SNR consistently attains superior performance compared to its competitor algorithms. We also demonstrate that SNR is robust to its sole hyperparameter, its rejection percentile threshold, while competitor algorithms show significant sensitivity. SNR's threshold-based reset mechanism is motivated by a simple hypothesis test that we derive. Seen through the lens of this hypothesis test, competing reset proposals yield suboptimal error rates in correctly detecting inactive neurons, potentially explaining our experimental observations. We also conduct a theoretical investigation of the optimization landscape for the problem of learning a single ReLU. We show that even when initialized adversarially, an idealized version of SNR learns the target ReLU, while regularization based approaches can fail to learn.

cs.LG↗

arXiv2023

Diversity Embeddings and the Hypergraph Sparsest Cut

Good approximations have been attained for the sparsest cut problem by rounding solutions to convex relaxations via low-distortion metric embeddings. Recently, Bryant and Tupper showed that this approach extends to the hypergraph setting by formulating a linear program whose solutions are so-called diversities which are rounded via diversity embeddings into $\ell_1$. Diversities are a generalization of metric spaces in which the nonnegative function is defined on all subsets as opposed to only on pairs of elements. We show that this approach yields a polytime $O(\log{n})$-approximation when either the supply or demands are given by a graph. This result improves upon Plotkin et al.'s $O(\log{(kn)}\log{n})$-approximation, where $k$ is the number of demands, for the setting where the supply is given by a graph and the demands are given by a hypergraph. Additionally, we provide a polytime $O(\min{\{r_G,r_H\}}\log{r_H}\log{n})$-approximation for when the supply and demands are given by hypergraphs whose hyperedges are bounded in cardinality by $r_G$ and $r_H$ respectively. To establish these results we provide an $O(\log{n})$-distortion $\ell_1$ embedding for the class of diversities known as diameter diversities. This improves upon Bryant and Tupper's $O(\log\^2{n})$-distortion embedding. The smallest known distortion with which an arbitrary diversity can be embedded into $\ell_1$ is $O(n)$. We show that for any $\epsilon > 0$ and any $p>0$, there is a family of diversities which cannot be embedded into $\ell_1$ in polynomial time with distortion smaller than $O(n^{1-\epsilon})$ based on querying the diversities on sets of cardinality at most $O(\log^p{n})$, unless $P=NP$. This disproves (an algorithmic refinement of) Bryant and Tupper's conjecture that there exists an $O(\sqrt{n})$-distortion $\ell_1$ embedding based off a diversity's induced metric.

cs.DS↗