arXiv subjects

Nikita P. Kalinin

Publications and source records attributed to Nikita P. Kalinin.

16 recordsLinked to original sources

arXiv2026

Dithered Gaussian Mechanism for Randomness-Efficient Differential Privacy

We present the dithered Gaussian mechanism, an alternative to the discrete Gaussian mechanism for differential privacy that discretizes the private output rather than the noise distribution itself. By interpreting this discretization as post-processing of the Gaussian mechanism, our construction directly inherits the privacy guarantees of the standard Gaussian mechanism while avoiding vulnerabilities caused by finite-precision floating-point outputs. In addition, the mechanism is provably randomness-efficient: by sampling the discretized output values directly, the number of high-quality random bits required for privacy can be reduced significantly and made independent of the noise level. This is achieved by separating the randomness into two sources: a high-quality source used for the privacy-critical sampling step, and a high-performance public source, possibly known to the adversary, that supplies the additional randomness needed for randomized discretization. This separation enables the use of cryptographically secure randomness without substantial performance loss. As an application, we study model training with DP-SGD and show that cryptographically secure noise generation with reduced exposure to floating-point vulnerabilities can be achieved with modest practical overhead.

cs.CR↗

arXiv2026

DP-λCGD: Efficient Noise Correlation for Differentially Private Model Training

Differentially private stochastic gradient descent (DP-SGD) is the gold standard for training machine learning models with formal differential privacy guarantees. Several recent extensions improve its accuracy by introducing correlated noise across training iterations. Matrix factorization mechanisms are a prominent example, but they can require storing previously added noise vectors, leading to substantial memory overhead. In this work, we propose a new algorithm for correlated-noise private training that correlates noise only with the immediately preceding iteration and cancels a tunable portion of it. By regenerating the previous noise using a pseudorandom number generator rather than storing it, our method has essentially the same memory footprint as DP-SGD. We show that the computational overhead is minimal and empirically demonstrate improved accuracy over DP-SGD.

cs.LG↗

arXiv2026

Beyond Square Roots: A Memory-Efficient Explicit Factorization for Multi-Epoch Private Learning

Correlated-noise mechanisms are among the most promising approaches for improving the utility of differentially private model training, but rigorous guarantees require explicit, analyzable factorizations, and practical deployment requires memory efficiency. Recent works have developed banded inverse factorizations, which address both requirements by exploiting a banded structure in the correlation matrix. The bandwidth controls the size of the noise buffer used to correlate noise across iterations, and thus governs the tradeoff between utility and memory cost. At the two ends of this tradeoff, DP-$λ$CGD achieves high memory efficiency by using only a one-step noise buffer, but this limits its utility gains, while the banded inverse square root (BISR) factorization exploits larger correlation windows and is asymptotically optimal for large bandwidths but performs poorly at low bandwidths. To bridge this gap, we introduce $γ$-BIFR, a unified generalization of both factorizations. In the low-memory, low-bandwidth regime, $γ$-BIFR improves RMSE, amplified RMSE, and private training performance, while yielding tighter theoretical guarantees for multi-participation error in multi-epoch training.

cs.LG↗

arXiv2026

The divisor function for matrices

We introduce a matrix divisor function $τ_n(T,M)$, counting factorisations $AB=M$ for $n\times n$ integer matrices $A,B$ of height at most $T$. For a fixed non-singular $M$, or for the zero matrix $M=O_n$, we prove an asymptotic formula for $τ_n(T,M)$, as $T\to \infty$, using lattice point counting. We also prove an essentially sharp uniform upper bound for $τ_n(T,M)$, for an arbitrary non-singular matrix $M$.

math.NT↗

arXiv2026

Improved Error Bounds for Pure Differentially Private Continual Counting via Matrix Factorization

Continual counting under pure differential privacy is one of the simplest and most well-studied problems in the continual observation model. Nevertheless, an asymptotic gap remains between the best known upper and lower bounds for maximum squared error and mean squared error: the upper bound is $O(ε^{-2}\log^3 n)$, while the lower bound is $Ω(ε^{-2}\log^2 n)$, for both error metrics. The best known constant in the upper bound is achieved by the $k$-ary tree mechanism with the subtraction trick, due to Andersson, Pagh, Steiner, and Torkamani (FORC 2025). In this work, we improve the leading constant in the maximum squared error and the mean squared error. Our approach uses a general matrix factorization mechanism, yielding an improved bound for pure-DP continual counting that does not rely on a tree-based construction. The mechanism starts from a good-quality low-dimensional factorization, obtained via gradient-based optimization, and gives an explicit matrix construction that lifts this factorization to arbitrarily large dimensions, further improving its error guarantees. We offer an efficient algorithmic implementation of our mechanism. On the lower-bound side, we prove an $Ω(ε^{-2}\log^3 n)$ lower bound for the class of factorizations whose matrices have entries in $\{0,1\}$, matching the upper-bound asymptotics for this class. This class includes the binary tree mechanism and $k$-ary tree mechanisms without the subtraction trick. Extending this lower bound to arbitrary matrix factorizations, and beyond the matrix mechanism altogether, remains an open problem.

cs.DS↗

arXiv2026

Matrix Factorization for Practical Continual Mean Estimation Under User-Level Differential Privacy

We study continual mean estimation, where data vectors arrive sequentially and the goal is to maintain accurate estimates of the running mean. We address this problem under user-level differential privacy, which protects each user's entire dataset even when they contribute multiple data points. Previous work on this problem has focused on pure differential privacy. While important, this approach limits applicability, as it leads to overly noisy estimates. In contrast, we analyze the problem under approximate differential privacy, adopting recent advances in the Matrix Factorization mechanism. We introduce a novel mean estimation specific factorization, which is both efficient and accurate, achieving asymptotically lower mean-squared error bounds in continual mean estimation under user-level differential privacy.

cs.LG↗

arXiv2026

Learning Rate Scheduling with Matrix Factorization for Private Training

We study differentially private model training with stochastic gradient descent under learning rate scheduling and correlated noise. Although correlated noise, in particular via matrix factorizations, has been shown to improve accuracy, prior theoretical work focused primarily on the prefix-sum workload. That workload assumes a constant learning rate, whereas in practice learning rate schedules are widely used to accelerate training and improve convergence. We close this gap by deriving general upper and lower bounds for a broad class of learning rate schedules in both single- and multi-epoch settings. Building on these results, we propose a learning-rate-aware factorization that achieves improvements over prefix-sum factorizations under both MaxSE and MeanSE error metrics. Our theoretical analysis yields memory-efficient constructions suitable for practical deployment, and experiments on CIFAR-10 and IMDB datasets confirm that schedule-aware factorizations improve accuracy in private training.

cs.LG↗

arXiv2026

Back to Square Roots: An Optimal Bound on the Matrix Factorization Error for Multi-Epoch Differentially Private SGD

Matrix factorization mechanisms for differentially private training have emerged as a promising approach to improve model utility under privacy constraints. In practical settings, models are typically trained over multiple epochs, requiring matrix factorizations that account for repeated participation. Existing theoretical upper and lower bounds on multi-epoch factorization error leave a significant gap. In this work, we introduce a new explicit factorization method, Banded Inverse Square Root (BISR), which imposes a banded structure on the inverse correlation matrix. This factorization enables us to derive an explicit and tight characterization of the multi-epoch error. We further prove that BISR achieves asymptotically optimal error by matching the upper and lower bounds. Empirically, BISR performs on par with state-of-the-art factorization methods, while being simpler to implement, computationally efficient, and easier to analyze.

cs.CR↗

arXiv2025

DP-MicroAdam: Private and Frugal Algorithm for Training and Fine-tuning

Adaptive optimizers are the de facto standard in non-private training as they often enable faster convergence and improved performance. In contrast, differentially private (DP) training is still predominantly performed with DP-SGD, typically requiring extensive compute and hyperparameter tuning. We propose DP-MicroAdam, a memory-efficient and sparsity-aware adaptive DP optimizer. We prove that DP-MicroAdam converges in stochastic non-convex optimization at the optimal $\mathcal{O}(1/\sqrt{T})$ rate, up to privacy-dependent constants. Empirically, DP-MicroAdam outperforms existing adaptive DP optimizers and achieves competitive or superior accuracy compared to DP-SGD across a range of benchmarks, including CIFAR-10, large-scale ImageNet training, and private fine-tuning of pretrained transformers. These results demonstrate that adaptive optimization can improve both performance and stability under differential privacy.

cs.LG↗

arXiv2025

Normalized Square Root: Sharper Matrix Factorization Bounds for Differentially Private Continual Counting

The factorization norms of the lower-triangular all-ones $n \times n$ matrix, $γ_2(M_{count})$ and $γ_{F}(M_{count})$, play a central role in differential privacy as they are used to give theoretical justification of the accuracy of the only known production-level private training algorithm of deep neural networks by Google. Prior to this work, the best known upper bound on $γ_2(M_{count})$ was $1 + \frac{\log n}π$ by Mathias (Linear Algebra and Applications, 1993), and the best known lower bound was $\frac{1}π(2 + \log(\frac{2n+1}{3})) \approx 0.507 + \frac{\log n}π$ (Matoušek, Nikolov, Talwar, IMRN 2020), where $\log$ denotes the natural logarithm. Recently, Henzinger and Upadhyay (SODA 2025) gave the first explicit factorization that meets the bound of Mathias (1993) and asked whether there exists an explicit factorization that improves on Mathias' bound. We answer this question in the affirmative. Additionally, we improve the lower bound significantly. More specifically, we show that $$ 0.701 + \frac{\log n}π + o(1) \;\leq\; γ_2(M_{count}) \;\leq\; 0.846 + \frac{\log n}π + o(1). $$ That is, we reduce the gap between the upper and lower bound to $0.14 + o(1)$. We also show that our factors achieve a better upper bound for $γ_{F}(M_{count})$ compared to prior work, and we establish an improved lower bound: $$ 0.701 + \frac{\log n}π + o(1) \;\leq\; γ_{F}(M_{count}) \;\leq\; 0.748 + \frac{\log n}π + o(1). $$ That is, the gap between the lower and upper bound provided by our explicit factorization is $0.047 + o(1)$.

cs.DS↗

arXiv2025

Banded Square Root Matrix Factorization for Differentially Private Model Training

Current state-of-the-art methods for differentially private model training are based on matrix factorization techniques. However, these methods suffer from high computational overhead because they require numerically solving a demanding optimization problem to determine an approximately optimal factorization prior to the actual model training. In this work, we present a new matrix factorization approach, BSR, which overcomes this computational bottleneck. By exploiting properties of the standard matrix square root, BSR allows to efficiently handle also large-scale problems. For the key scenario of stochastic gradient descent with momentum and weight decay, we even derive analytical expressions for BSR that render the computational overhead negligible. We prove bounds on the approximation quality that hold both in the centralized and in the federated learning setting. Our numerical experiments demonstrate that models trained using BSR perform on par with the best existing methods, while completely avoiding their computational overhead.

cs.LG↗

arXiv2025

Continual Release Moment Estimation with Differential Privacy

We propose Joint Moment Estimation (JME), a method for continually and privately estimating both the first and second moments of data with reduced noise compared to naive approaches. JME uses the matrix mechanism and a joint sensitivity analysis to allow the second moment estimation with no additional privacy cost, thereby improving accuracy while maintaining privacy. We demonstrate JME's effectiveness in two applications: estimating the running mean and covariance matrix for Gaussian density estimation, and model training with DP-Adam on CIFAR-10.

cs.LG↗

arXiv2025

Binned Group Algebra Factorization for Differentially Private Continual Counting

We study memory-efficient matrix factorization for differentially private counting under continual observation. While recent work by Henzinger and Upadhyay 2024 introduced a factorization method with reduced error based on group algebra, its practicality in streaming settings remains limited by computational constraints. We present new structural properties of the group algebra factorization, enabling the use of a binning technique from Andersson and Pagh (2024). By grouping similar values in rows, the binning method reduces memory usage and running time to $\tilde O(\sqrt{n})$, where $n$ is the length of the input stream, while maintaining a low error. Our work bridges the gap between theoretical improvements in factorization accuracy and practical efficiency in large-scale private learning systems.

cs.DS↗

arXiv2025

Efficient Estimation of a Gaussian Mean with Local Differential Privacy

In this paper we study the problem of estimating the unknown mean $θ$ of a unit variance Gaussian distribution in a locally differentially private (LDP) way. In the high-privacy regime ($ε\le 1$), we identify an optimal privacy mechanism that minimizes the variance of the estimator asymptotically. Our main technical contribution is the maximization of the Fisher-Information of the sanitized data with respect to the local privacy mechanism $Q$. We find that the exact solution $Q_{θ,ε}$ of this maximization is the sign mechanism that applies randomized response to the sign of $X_i-θ$, where $X_1,\dots, X_n$ are the confidential iid original samples. However, since this optimal local mechanism depends on the unknown mean $θ$, we employ a two-stage LDP parameter estimation procedure which requires splitting agents into two groups. The first $n_1$ observations are used to consistently but not necessarily efficiently estimate the parameter $θ$ by $\tildeθ_{n_1}$. Then this estimate is updated by applying the sign mechanism with $\tildeθ_{n_1}$ instead of $θ$ to the remaining $n-n_1$ observations, to obtain an LDP and efficient estimator of the unknown mean.

math.ST↗

arXiv2024

Notes on Sampled Gaussian Mechanism

In these notes, we prove a recent conjecture posed in the paper by Räisä, O. et al. [Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimization (2024)]. Theorem 6.2 of the paper asserts that for the Sampled Gaussian Mechanism - a composition of subsampling and additive Gaussian noise, the effective noise level, $σ_{\text{eff}} = \frac{σ(q)}{q}$, decreases as a function of the subsampling rate $q$. Consequently, larger subsampling rates are preferred for better privacy-utility trade-offs. Our notes provide a rigorous proof of Conjecture 6.3, which was left unresolved in the original paper, thereby completing the proof of Theorem 6.2.

cs.LG↗

arXiv2024

DP-KAN: Differentially Private Kolmogorov-Arnold Networks

We study the Kolmogorov-Arnold Network (KAN), recently proposed as an alternative to the classical Multilayer Perceptron (MLP), in the application for differentially private model training. Using the DP-SGD algorithm, we demonstrate that KAN can be made private in a straightforward manner and evaluated its performance across several datasets. Our results indicate that the accuracy of KAN is not only comparable with MLP but also experiences similar deterioration due to privacy constraints, making it suitable for differentially private model training.

cs.LG↗