arXiv subjects

Ayaka Sakata

Publications and source records attributed to Ayaka Sakata.

At least 19 recordsLinked to original sources

arXiv2026

Sequential Retrieval in Dense Associative Memory: Asymptotic Dynamics and Storage Capacity

We study the retrieval dynamics of an $n$-body asymmetric dense associative memory model, in which sequential retrieval is implemented by shifting one pattern index in the higher-order Hebbian interaction. The model exhibits sequential retrieval of the stored patterns, and the retrieval state corresponds to a stable limit-cycle attractor. The storage capacity is defined as the maximum loading rate for which this limit cycle remains stable. Using generating functional analysis, we derive the dynamical equations describing the sequential retrieval state. The generating functional analysis reveals that the retarded self-interaction term vanishes in the asymmetric model, and the resulting overlap dynamics is reproduced by an Amari-Maginu-type signal-to-noise analysis. Also, we find that for the interaction order $n\ge 3$, the normalized storage capacity decreases monotonically with increasing interaction order, in contrast to symmetric Modern Hopfield models.

cond-mat.dis-nn↗

arXiv2026

Stabilizing Private LASSO under Heterogeneous Covariates via Anisotropic Objective Perturbation

We study high-dimensional LASSO under differential privacy via objective perturbation with heterogeneous covariate scales. In practical scenarios, covariates often exhibit diverse scales; however, standard preprocessing is problematic under privacy constraints, as it consumes additional privacy budget. This heterogeneity induces effective anisotropy in the objective perturbation via the inverse Gram matrix of covariates, which can degrade the stability and accuracy of algorithms. To address this, we propose a Gram-based anisotropic objective perturbation, a ``pre-distortion" strategy that counteracts the distortion from the covariate structure to restore isotropy in the estimation process. Using an Approximate Message Passing (AMP) framework and state evolution analysis, we demonstrate that our proposed perturbation significantly stabilizes convergence and improves both statistical efficiency and privacy performance compared to standard uniform noise injection. Our results provide theoretical insights into designing stable and efficient private estimators without relying on data-dependent preprocessing.

stat.ML↗

arXiv2026

A Reexamination of Noise-Driven Robustness Evolution in Gene Regulatory Networks

Gene regulatory network (GRN) models are widely used to investigate gene-expression dynamics in biological systems. In this study, we re-examine an evolutionary GRN model proposed by Kaneko, which was used to investigate the relationship between robustness to nongenetic phenotypic noise and robustness to mutation. We explicitly reconstruct the simulation protocol by specifying computational details omitted from the original study and reproduce its main qualitative findings, including the noise-dependent suppression of low-fitness individuals, the relationship between isogenic phenotypic variance and genetic variance, and the increased robustness of evolved networks under certain expression noise. We further analyze the robustness of evolved populations to variations in initial conditions. While the representative-network analysis shows quantitative differences from the original study, the population-level analysis supports the qualitative conclusion that evolution under higher phenotypic noise leads to broader basins of attraction. We provide a reusable implementation and publicly release the source code to support reproducibility and facilitate further studies on robustness in dynamical GRN models.

q-bio.MN↗

arXiv2026

Perfect reconstruction of sparse signals using nonconvexity control and one-step RSB message passing

We consider sparse signal reconstruction via minimization of the smoothly clipped absolute deviation (SCAD) penalty, and develop one-step replica-symmetry-breaking (1RSB) extensions of approximate message passing (AMP), termed 1RSB-AMP. Starting from the 1RSB formulation of belief propagation, we derive explicit update rules of 1RSB-AMP together with the corresponding state evolution (1RSB-SE) equations. A detailed comparison shows that 1RSB-AMP and 1RSB-SE agree remarkably well at the macroscopic level, even in parameter regions where replica-symmetric (RS) AMP, termed RS-AMP, diverges and where the 1RSB description itself is not expected to be thermodynamically exact. Fixed-point analysis of 1RSB-SE reveals a phase diagram consisting of success, failure, and diverging phases, as in the RS case. However, the diverging-region boundary now depends on the Parisi parameter due to the 1RSB ansatz, and we propose a new criterion---minimizing the size of the diverging region---rather than the conventional zero-complexity condition, to determine its value. Combining this criterion with the nonconvexity-control (NCC) protocol proposed in a previous RS study improves the algorithmic limit of perfect reconstruction compared with RS-AMP. Numerical solutions of 1RSB-SE and experiments with 1RSB-AMP confirm that this improved limit is achieved in practice, though the gain is modest and remains slightly inferior to the Bayes-optimal threshold. We also report the behavior of thermodynamic quantities---overlaps, free entropy, complexity, and the non-self-averaging susceptibility---that characterize the 1RSB phase in this problem.

stat.ML↗

arXiv2026

Privacy-Accuracy Trade-offs in High-Dimensional LASSO under Perturbation Mechanisms

We study privacy-preserving sparse linear regression in the high-dimensional regime, focusing on the LASSO estimator. We analyze two widely used mechanisms for differential privacy: output perturbation, which injects noise into the estimator, and objective perturbation, which adds a random linear term to the loss function. Using approximate message passing (AMP), we characterize the typical behavior of these estimators under random design and privacy noise. To quantify privacy, we adopt typical-case measures, including the on-average KL divergence, which admits a hypothesis-testing interpretation in terms of distinguishability between neighboring datasets. Our analysis reveals that sparsity plays a central role in shaping the privacy-accuracy trade-off: stronger regularization can improve privacy by stabilizing the estimator against single-point data changes. We further show that the two mechanisms exhibit qualitatively different behaviors. In particular, for objective perturbation, increasing the noise level can have non-monotonic effects, and excessive noise may destabilize the estimator, leading to increased sensitivity to data perturbations. Our results demonstrate that AMP provides a powerful framework for analyzing privacy-accuracy trade-offs in high-dimensional sparse models.

stat.ML↗

arXiv2025

The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model

Self-distillation (SD), a technique where a model improves itself using its own predictions, has attracted attention as a simple yet powerful approach in machine learning. Despite its widespread use, the mechanisms underlying its effectiveness remain unclear. In this study, we investigate the efficacy of hyperparameter-tuned multi-stage SD with a linear classifier for binary classification on noisy Gaussian mixture data. For the analysis, we employ the replica method from statistical physics. Our findings reveal that the primary driver of SD's performance improvement is denoising through hard pseudo-labels, with the most notable gains observed in moderately sized datasets. We also identify two practical heuristics to enhance SD: early stopping that limits the number of stages, which is broadly effective, and bias parameter fixing, which helps under label imbalance. To empirically validate our theoretical findings derived from our toy model, we conduct additional experiments on CIFAR-10 classification using pretrained ResNet backbone. These results provide both theoretical and practical insights, advancing our understanding and application of SD in noisy settings.

stat.ML↗

arXiv2025

High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator

Quantized neural network training optimizes a discrete, non-differentiable objective. The straight-through estimator (STE) enables backpropagation through surrogate gradients and is widely used. While previous studies have primarily focused on the properties of surrogate gradients and their convergence, the influence of quantization hyperparameters, such as bit width and quantization range, on learning dynamics remains largely unexplored. We theoretically show that in the high-dimensional limit, STE dynamics converge to a deterministic ordinary differential equation. This reveals that STE training exhibits a plateau followed by a sharp drop in generalization error, with plateau length depending on the quantization range. A fixed-point analysis quantifies the asymptotic deviation from the unquantized linear model. We also extend analytical techniques for stochastic gradient descent to nonlinear transformations of weights and inputs.

stat.ML↗

arXiv2024

Effect of Weight Quantization on Learning Models by Typical Case Analysis

This paper examines the quantization methods used in large-scale data analysis models and their hyperparameter choices. The recent surge in data analysis scale has significantly increased computational resource requirements. To address this, quantizing model weights has become a prevalent practice in data analysis applications such as deep learning. Quantization is particularly vital for deploying large models on devices with limited computational resources. However, the selection of quantization hyperparameters, like the number of bits and value range for weight quantization, remains an underexplored area. In this study, we employ the typical case analysis from statistical physics, specifically the replica method, to explore the impact of hyperparameters on the quantization of simple learning models. Our analysis yields three key findings: (i) an unstable hyperparameter phase, known as replica symmetry breaking, occurs with a small number of bits and a large quantization width; (ii) there is an optimal quantization width that minimizes error; and (iii) quantization delays the onset of overparameterization, helping to mitigate overfitting as indicated by the double descent phenomenon. We also discover that non-uniform quantization can enhance stability. Additionally, we develop an approximate message-passing algorithm to validate our theoretical results.

stat.ML↗

arXiv2023

Evolutionary Shaping of Low-Dimensional Path Facilitates Robust and Plastic Switching Between Phenotypes

Biological systems must be robust for stable function against perturbations, but robustness alone is not sufficient. The ability to switch between appropriate states (phenotypes) in response to different conditions is essential for biological functions. How are robustness and plasticity simultaneously acquired through evolution? We examine the evolution of genotypes that realize plastic switching between two phenotypes upon external inputs, as well as stationary expressions of phenotypes. We introduce a statistical physics model consisting of spins, with active and regulatory sites, which are distinct from each other. We represent the phenotype and genotype as spin configurations and the spin-spin interactions, respectively. The fitness for selection is given so that it takes a higher value as more of the active sites take two requested spin configurations depending on the regulation. We numerically evolve the interaction matrix by changing them with mutations and selection of those with higher fitness. Our numerical simulations show that characteristic genotypes evolve slightly above the transition temperature between replica symmetric and replica symmetry breaking phase. These genotypes shape two spin configurations separately depending on the regulation, where the two phenotypes are dominantly represented by the genotypes' first and second eigenmodes, and smooth switching of two phenotypes are achieved by following a one-dimensional path connecting the two phenotypes. Upon changes in regulations, spin configurations are attracted to this path, which allows for robust and plastic switching between the two phenotypes. The statistical-physics analysis show that the free energy landscape has a valley along the switching path. Our finding indicates that the compatibility of the robustness and plasticity is acquired by the evolution of the low-dimensionality in the phenotype space.

q-bio.PE↗

arXiv2023

Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing

We discuss the prediction accuracy of assumed statistical models in terms of prediction errors for the generalized linear model and penalized maximum likelihood methods. We derive the forms of estimators for the prediction errors, such as $C_p$ criterion, information criteria, and leave-one-out cross validation (LOOCV) error, using the generalized approximate message passing (GAMP) algorithm and replica method. These estimators coincide with each other when the number of model parameters is sufficiently small; however, there is a discrepancy between them in particular in the parameter region where the number of model parameters is larger than the data dimension. In this paper, we review the prediction errors and corresponding estimators, and discuss their differences. In the framework of GAMP, we show that the information criteria can be expressed by using the variance of the estimates. Further, we demonstrate how to approach LOOCV error from the information criteria by utilizing the expression provided by GAMP.

stat.ML↗

arXiv2021

Decision Theoretic Cutoff and ROC Analysis for Bayesian Optimal Group Testing

We study the inference problem in the group testing to identify defective items from the perspective of the decision theory. We introduce Bayesian inference and consider the Bayesian optimal setting in which the true generative process of the test results is known. We demonstrate the adequacy of the posterior marginal probability in the Bayesian optimal setting as a diagnostic variable based on the area under the curve (AUC). Using the posterior marginal probability, we derive the general expression of the optimal cutoff value that yields the minimum expected risk function. Furthermore, we evaluate the performance of the Bayesian group testing without knowing the true states of the items: defective or non-defective. By introducing an analytical method from statistical physics, we derive the receiver operating characteristics curve, and quantify the corresponding AUC under the Bayesian optimal setting. The obtained analytical results precisely describes the actual performance of the belief propagation algorithm defined for single samples when the number of items is sufficiently large.

cs.IT↗

arXiv2021

Perfect reconstruction of sparse signals with piecewise continuous nonconvex penalties and nonconvexity control

We consider compressed sensing formulated as a minimization problem of nonconvex sparse penalties, Smoothly Clipped Absolute deviation (SCAD) and Minimax Concave Penalty (MCP). The nonconvexity of these penalties is controlled by nonconvexity parameters, and L1 penalty is contained as a limit with respect to these parameters. The analytically derived reconstruction limit overcomes that of L1 and the algorithmic limit in the Bayes-optimal setting, when the nonconvexity parameters have suitable values. However, for small nonconvexity parameters, where the reconstruction of the relatively dense signals is theoretically guaranteed, the corresponding approximate message passing (AMP) cannot achieve perfect reconstruction. We identify that the shrinks in the basin of attraction to the perfect reconstruction causes the discrepancy between the AMP and corresponding theory using state evolution. A part of the discrepancy is resolved by introducing the control of the nonconvexity parameters to guide the AMP trajectory to the basin of the attraction.

stat.ML↗

arXiv2020

Active pooling design in group testing based on Bayesian posterior prediction

In identifying infected patients in a population, group testing is an effective method to reduce the number of tests and correct the test errors. In the group testing procedure, tests are performed on pools of specimens collected from patients, where the number of pools is lower than that of patients. The performance of group testing heavily depends on the design of pools and algorithms that are used in inferring the infected patients from the test outcomes. In this paper, an adaptive design method of pools based on the predictive distribution is proposed in the framework of Bayesian inference. The proposed method executed using the belief propagation algorithm results in more accurate identification of the infected patients, as compared to the group testing performed on random pools determined in advance.

stat.ML↗

arXiv2020

Bayesian inference of infected patients in group testing with prevalence estimation

Group testing is a method of identifying infected patients by performing tests on a pool of specimens collected from patients. For the case in which the test returns a false result with finite probability, we propose Bayesian inference and a corresponding belief propagation (BP) algorithm to identify the infected patients from the results of tests performed on the pool. We show that the true-positive rate is improved by taking into account the credible interval of a point estimate of each patient. Further, the prevalence and the error probability in the test are estimated by combining an expectation-maximization method with the BP algorithm. As another approach, we introduce a hierarchical Bayes model to identify the infected patients and estimate the prevalence. By comparing these methods, we formulate a guide for practical usage.

stat.ML↗

arXiv2020

Dimensional reduction in evolving spin-glass model: correlation of phenotypic responses to environmental and mutational changes

The evolution of high-dimensional phenotypes is investigated using a statistical physics model consists of interacting spins, in which genotypes, phenotypes, and environments are represented by spin configurations, interaction matrices, and external fields, respectively. We found that phenotypic changes upon diverse environmental change and genetic variation are highly correlated across all spins, consistent with recent experimental observations of biological systems. The dimension reduction in phenotypic changes is shown to be a result of the evolution of the robustness to thermal noise, achieved at the replica symmetric phase.

physics.bio-ph↗

arXiv2019

Cross validation in sparse linear regression with piecewise continuous nonconvex penalties and its acceleration

We investigate the signal reconstruction performance of sparse linear regression in the presence of noise when piecewise continuous nonconvex penalties are used. Among such penalties, we focus on the SCAD penalty. The contributions of this study are three-fold: We first present a theoretical analysis of a typical reconstruction performance, using the replica method, under the assumption that each component of the design matrix is given as an independent and identically distributed (i.i.d.) Gaussian variable. This clarifies the superiority of the SCAD estimator compared with $\ell_1$ in a wide parameter range, although the nonconvex nature of the penalty tends to lead to solution multiplicity in certain regions. This multiplicity is shown to be connected to replica symmetry breaking in the spin-glass theory. We also show that the global minimum of the mean square error between the estimator and the true signal is located in the replica symmetric phase. Second, we develop an approximate formula efficiently computing the cross-validation error without actually conducting the cross-validation, which is also applicable to the non-i.i.d. design matrices. It is shown that this formula is only applicable to the unique solution region and tends to be unstable in the multiple solution region. We implement instability detection procedures, which allows the approximate formula to stand alone and resultantly enables us to draw phase diagrams for any specific dataset. Third, we propose an annealing procedure, called nonconvexity annealing, to obtain the solution path efficiently. Numerical simulations are conducted on simulated datasets to examine these results to verify the theoretical results consistency and the approximate formula efficiency. Another numerical experiment on a real-world dataset is conducted; its results are consistent with those of earlier studies using the $\ell_0$ formulation.

stat.ML↗

arXiv2018

Estimator of Prediction Error Based on Approximate Message Passing for Penalized Linear Regression

We propose an estimator of prediction error using an approximate message passing (AMP) algorithm that can be applied to a broad range of sparse penalties. Following Stein's lemma, the estimator of the generalized degrees of freedom, which is a key quantity for the construction of the estimator of the prediction error, is calculated at the AMP fixed point. The resulting form of the AMP-based estimator does not depend on the penalty function, and its value can be further improved by considering the correlation between predictors. The proposed estimator is asymptotically unbiased when the components of the predictors and response variables are independently generated according to a Gaussian distribution. We examine the behaviour of the estimator for real data under nonconvex sparse penalties, where Akaike's information criterion does not correspond to an unbiased estimator of the prediction error. The model selected by the proposed estimator is close to that which minimizes the true prediction error.

stat.ML↗

arXiv2018

Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis

We analyse a linear regression problem with nonconvex regularization called smoothly clipped absolute deviation (SCAD) under an overcomplete Gaussian basis for Gaussian random data. We propose an approximate message passing (AMP) algorithm considering nonconvex regularization, namely SCAD-AMP, and analytically show that the stability condition corresponds to the de Almeida--Thouless condition in spin glass literature. Through asymptotic analysis, we show the correspondence between the density evolution of SCAD-AMP and the replica symmetric solution. Numerical experiments confirm that for a sufficiently large system size, SCAD-AMP achieves the optimal performance predicted by the replica method. Through replica analysis, a phase transition between replica symmetric (RS) and replica symmetry breaking (RSB) region is found in the parameter space of SCAD. The appearance of the RS region for a nonconvex penalty is a significant advantage that indicates the region of smooth landscape of the optimization problem. Furthermore, we analytically show that the statistical representation performance of the SCAD penalty is better than that of L1-based methods, and the minimum representation error under RS assumption is obtained at the edge of the RS/RSB phase. The correspondence between the convergence of the existing coordinate descent algorithm and RS/RSB transition is also indicated.

stat.ML↗