Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2611.03445

Error Accumulation in Second-Order Sequential Machine Unlearning

Chen Shang·Haolin Zou·Binghe Zhu·Arian Maleki
View original source ↗Read source document ↗Export metadata ↓

Abstract

Machine unlearning seeks to remove the influence of deleted training data without full retraining. In the sequential setting, deletion requests arrive over time, and repeated updates can accumulate approximation errors. In this paper, we study error accumulation in high dimensions, where parameter count $p$ is proportional to sample size $n$, for second-order unlearning methods under two privacy settings: (i) \emph{partial unlearning}, which allows unpublished internal states to be retained while certified versions of the model are released; and (ii) \emph{perfect unlearning}, which retains only published models and no hidden states. We also distinguish pointwise and pathwise certification, depending on whether the certification guarantee applies to each released model or the whole released sequence. We analyze conjugate gradient (CG) as an efficient solver for the inverse Hessian-vector product (IHVP) in Newton iterations. Our theory shows that a maximum of 1/2/2/3 Newton--CG updates suffice for successful certification and vanishing population-risk degradation of the published model, for pointwise partial, pathwise partial, pointwise perfect and pathwise perfect certification, respectively. Numerical experiments demonstrate accurate sequential tracking and substantial computational gains of Newton-based methods combined with conjugate gradient, over gradient-based updates, especially for ill-conditioned problems.

Explore related subjects

stat.MLcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Chen Shang, Haolin Zou, Binghe Zhu, Arian Maleki. 2026-11-02. Error Accumulation in Second-Order Sequential Machine Unlearning. https://arxiv.org/abs/2611.03445

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Recovering Geodesic Distances from Noisy Observations

Let $(M,d,μ)$ be a nice geodesic probability space (such as a manifold) and $X=\{x_i\}_{i\in [N]} \subset M$ a finite $μ$-i.i.d. sample. Suppose we observe noisy distance random variables $d'(x_i,x_j)$. When can the true distances be recovered to small additive error? This question has attracted interest in two disparate fields: latent-space graph models and boundary-value inverse problems. We first show that, taking $N \asymp ε^{-2d-2}|\logε|$ and using runtime $O(N^{2.5+o_d (1)})$, we can recover all distances in $X$ to additive error $O(ε|\logε|)$ with high probability, assuming $\mathbb{E} d'(x_i, x_j) = F(d(x_i,x_j))$ for an unknown, increasing bilipschitz $F$ and with the $d'(x_i,x_j)$ allowed to be arbitrarily heterogeneous in their distribution as long as they are bounded in subgaussian norm. This generalizes existing work, which required $F$ to be known and made strict assumptions on the distribution of $d'(x_i, x_j)$. Achieving this generalization required a novel ``ruler-building'' procedure to learn $F$. We extend our results to a missing data case where we randomly lose access to noisy distances. Our second result uses ideas from the first to tackle a partial observability case: $X = Y \cup Z$ with $Z$ sampled from $M$ and $Y$ sampled from an observer subset satisfying visibility conditions that model the assumptions made on the boundary in boundary-value inverse problems. In this setting, we recover all pairwise distances in $Z$ to additive error $O(ε)$ with $|Z| \asymp ε^{-2d-2}|\logε|$, $|Y| \asymp ε^{-\max\{d_ν,4\}}$ with $d_ν$ determined by the volume-growth properties of the probability measure on the observer set, and runtime $O(|Z|^3)$. This is, to our knowledge, the first quantitative guarantee for this observer/observed split under noise, which complements existing stability results for boundary data.

stat.ML↗
arXiv2026

Emergence of Distortions in High-Dimensional Guided Diffusion Models

Classifier-free guidance (CFG) is the de facto standard for conditional sampling in diffusion models, yet it often reduces sample diversity. Using tools from statistical physics, we analyze the emergence of generative distortions induced by CFG, namely the mismatch between the CFG sampling distribution and the true conditional distribution. We study this phenomenon in analytically tractable settings with exact score functions, characterizing its dependence on data dimensionality and the number of classes. For high-dimensional Gaussian mixtures, we use dynamic mean-field theory to show that distortions persist when the number of classes scales exponentially with the data dimension, whereas they vanish in the sub-exponential regime due to a dynamical phase transition. Moreover we found that, in the infinite-class limit, distortions remain unavoidable regardless of dimensionality because of the increasing density of classes. Finally, we show that standard CFG schedules cannot prevent variance shrinkage, and we propose a theoretically grounded guidance schedule incorporating a negative-guidance window that improves both class separability and sample diversity in real-world latent diffusion models.

stat.ML↗
arXiv2026

Deep Learning of Compositional Targets with Hierarchical Spectral Methods

Why depth yields a computational advantage over shallow methods remains a central open question in learning theory. We study this question in a controlled high-dimensional Gaussian setting, focusing on compositional target functions. We analyze their learnability using an explicit three-layer fitting model trained via layer-wise spectral estimators. The target combines polynomial latent features with a scalar readout, and its compositional structure allows learning to proceed in stages. Recovering an intermediate representation reduces the problem to lower-degree spectral estimation tasks, as in multi-index models. Our analysis establishes Gaussian comparison bounds for projected moments when $0<\varepsilon<1/3$, and motivates separations in sample complexity between two- and three-layer learning strategies, conditional on a conjectured spectral recovery step.

stat.ML↗