arXiv subjects

Illia Horenko

Publications and source records attributed to Illia Horenko.

15 recordsLinked to original sources

arXiv2026

On a joint simultaneous learning of relevant feature subsets and subspaces in regression-like problems

We extend a recently introduced Entropy-Optimal Manifold Clustering (EOMC) to allow for a joint simultaneous identification of subsets and subspaces of relevant features in nonstationary and nonlinear regression problems. It is shown that the proposed extension - that we coin as Entropy-Optimal Manifold Regression (EOMR) - allows a robust learning with linearly-scaling iteration and memory complexities. EOMR is compared to the most complete set of state-of-the-art tools from the Artificial Intelligence (AI) and Machine Learning (ML) that is available to the author, on the very challenging problems from chaotic and fluid dynamics: (i) on predicting the Lorenz-96 systems dynamics in strongly- and very-strongly chaotic regimes (with forcing parameter being $F=8$ and $F=12$, respectively); and, (ii) on a data from the Hasegawa-Wakatani model on the edge of the tokamak plasma. It is demonstrated that the proposed benchmarks (i) and (ii), indeed, are the very challenging problems for the state of the art ML and AI tools - since both the general-purpose gradient boosted random forests and deep neuronal networks, as well as transformer-based AI tools like TabPFN v.03 (more spezialised for large-dimensional small data learning problems) - result in orders of magnitude inferior root mean squared prediction errors, and orders of magnitude larger model complexities, when compared to the EOMR. For a Hasegawa-Wakatani example, EOMR distills a very simple entropy-optimal and skilful description of the leading Essential Orthogonal Function (EOF) dynamics, given by linear, causal and weakly-stationary autoregressive process described by just 8 parameters.

stat.ML↗

arXiv2026

Inferring Magnetic Material Parameters from Statistical Measures in Strongly Fluctuating Magnetization Dynamics

Magnetic material parameters such as the exchange stiffness and magnetic anisotropy govern the behavior and functionality of magnetic systems, yet their local inference from magnetization data remains challenging, particularly in strongly fluctuating regimes with polycrystalline or multiphase microstructure, where conventional texture-based methods become unreliable. We introduce a magnetization-only framework for inferring material parameters from thermally driven magnetization dynamics. Using micromagnetic simulations, we extract statistical quantities such as temporal mean and latent entropy from the magnetization dynamics, fit models to these descriptors, and invert the models to infer material parameters. We show that this framework enables material-parameter inference as well as grain-boundary detection in a heterogeneous sample. Among the descriptors considered, latent entropy yields more accurate parameter estimates than the temporal mean. Our results establish latent entropy as an efficient descriptor for inferring magnetic material parameters from dynamical magnetization data and point toward its use for experimental parameter extraction at high temperatures and, more broadly, under strongly fluctuating conditions.

cond-mat.mtrl-sci↗

arXiv2026

Linearly-scalable and entropy-optimal learning of nonstationary and nonlinear manifolds

We propose an Entropy-Optimal Manifold Clustering (EOMC) - and show that it mitigates the cost scaling and robustness issues of the existing dimensionality reduction and manifold learning tools in nonstationary and nonlinear situations, while pertaining the favourable O(T) iteration complexity scaling in the statistics size T, and allowing explicit computation of input data reliability. Application to the Lorenz-96 dynamical system in chaotic regime, as well as to a modified Hasegawa-Wakatani (mHW) model of drift-wave turbulence in the edge of a tokamak plasma reveals that for both of the models their essential dynamics is best described as a metastable regime-switching process, making infrequent transitions between the very persistent low-dimensional manifolds. At the same time, the Markovian mean exit times and relaxation times (that bound the predictability horizons for the identified regime-switching process) appear to decrease only very slowly with the growing external forcing - indicating approximately two-fold longer prediction horizons then is currently anticipated based on analysis of positive Lyapunov exponents, even in very chaotic model regimes. It is also demonstrated that when applied for a lossy compression of the Lorenz-96 and mHW output data in various forcing regimes, EOMC achieves several orders of magnitude smaller compression loss - when compared to the common PCA-related linear compression approaches that build a backbone of the state-of-the-art lossy data compression tools (like JPEG, MP3, and others). These findings open new exciting opportunities for EOMC and transfer operator theory, by offering new possibilities to significantly improve predictive skills and performance of data-driven tools in fluid mechanics and geosciences applications.

nlin.CD↗

arXiv2026

Fast, close, non-singular and property-preserving approximations of entropic measures

Entropic measures like Shannon entropy (SE), its quantum mechanical analogue von Neumann entropy, and Kullback-Leibler divergence (KL) are key components in many tools used in physics, information theory, machine learning (ML) and quantum computing. Besides of the significant amounts of SE and KL computations required in these fields, the singularity of their gradients near zero is one of the central mathematical reason inducing the high cost, frequently low robustness and slow convergence of computational tools that rely on these concepts. Here we propose the Fast Entropic Approximations (FEA) - non-singular rational approximations of SE and symmetrized KL, that preserve their main mathematical properties and achieve a mean absolute errors of around $10^-3$ ($10-20$ times better than comparable state-of-the-art computational approximations). We show that FEA allows up to around 2 times faster computation of SE and up to 37 times faster computation of symmetrized KL: it requires only $5$ to $7$ elementary computational operations, as compared to the tens of elementary operations behind SE and KL evaluations based on approximate logarithm schemes with table look-ups, bitshifts, or series approximations. On a set of common benchmarks for the feature selection problem in machine learning, we show that the combined effect of fewer elementary operations, low approximation error, preservation of main mathematical properties, and non-singular gradients allows much faster training of significantly-better models. We demonstrate that FEA enables ML feature extraction that is three orders of magnitude faster, and better in quality then the very popular LASSO feature extraction.

cs.LG↗

arXiv2026

Distillation and Interpretability of Ensemble Forecasts of ENSO Phase using Entropic Learning

This paper introduces a distillation framework for an ensemble of entropy-optimal Sparse Probabilistic Approximation (eSPA) models, trained exclusively on satellite-era observational and reanalysis data to predict ENSO phase up to 24 months in advance. While eSPA ensembles yield state-of-the-art forecast skill, they are harder to interpret than individual eSPA models. We show how to compress the ensemble into a compact set of "distilled" models by aggregating the structure of only those ensemble members that make correct predictions. This process yields a single, diagnostically tractable model for each forecast lead time that preserves forecast performance while also enabling diagnostics that are impractical to implement on the full ensemble. An analysis of the regime persistence of the distilled model "superclusters", as well as cross-lead clustering consistency, shows that the discretised system accurately captures the spatiotemporal dynamics of ENSO. By considering the effective dimension of the feature importance vectors, the complexity of the input space required for correct ENSO phase prediction is shown to peak when forecasts must cross the boreal spring predictability barrier. Spatial importance maps derived from the feature importance vectors are introduced to identify where predictive information resides in each field and are shown to include known physical precursors at certain lead times. Case studies of key events are also presented, showing how fields reconstructed from distilled model centroids trace the evolution from extratropical and inter-basin precursors to the mature ENSO state. Overall, the distillation framework enables a rigorous investigation of long-range ENSO predictability that complements real-time data-driven operational forecasts.

physics.comp-ph↗

arXiv2025

An entropy-optimal path to humble AI

Progress of AI has led to very successful, but by no means humble models and tools, especially regarding (i) the huge and further exploding costs and resources they demand, and (ii) the over-confidence of these tools with the answers they provide. Here we introduce a novel mathematical framework for a non-equilibrium entropy-optimizing reformulation of Boltzmann machines based on the exact law of total probability and the exact convex polytope representations. We show that it results in the highly-performant, but much cheaper, gradient-descent-free learning framework with mathematically-justified existence and uniqueness criteria, and cheaply-computable confidence/reliability measures for both the model inputs and the outputs. Comparisons to state-of-the-art AI tools in terms of performance, cost and the model descriptor lengths on a broad set of synthetic and real-world problems with varying complexity reveal that the proposed method results in more performant and slim models, with the descriptor lengths being very close to the intrinsic complexity scaling bounds for the underlying problems. Applying this framework to historical climate data results in models with systematically higher prediction skills for the onsets of important La Niña and El Niño climate phenomena, requiring just few years of climate data for training - a small fraction of what is necessary for contemporary climate prediction tools.

cs.LG↗

arXiv2025

Entropic learning enables skilful forecasts of ENSO phase at up to two years lead time

This paper extends previous work (Groom et al., \emph{Artif. Intell. Earth Syst.}, 2024) in applying the entropy-optimal Sparse Probabilistic Approximation (eSPA) algorithm to predict ENSO phase, defined by thresholding the Niño3.4 index. Only satellite-era observational datasets are used for training and validation, while retrospective forecasts from 2012 to 2022 are used to assess out-of-sample skill at lead times up to 24 months. Rather than train a single eSPA model per lead, we introduce an ensemble approach in which multiple eSPA models are aggregated via a novel meta-learning strategy. The features used include the leading principal components from a delay-embedded EOF analysis of global sea surface temperature, vertical temperature gradient (a thermocline proxy), and tropical Pacific wind stresses. Crucially, the data is processed to prevent any form of information leakage from the future, ensuring realistic real-time forecasting conditions. Despite the limited number of training instances, eSPA avoids overfitting and produces probabilistic forecasts with skill comparable to the International Research Institute for Climate and Society (IRI) ENSO prediction plume. Beyond the IRI's lead times, eSPA maintains skill out to 22 months for the ranked probability skill score and 24 months for accuracy and area under the ROC curve, all at a fraction of the computational cost of a fully-coupled dynamical model. Furthermore, eSPA successfully forecasts the 2015/16 and 2018/19 El Niño events at 24 months lead, the 2016/17, 2017/18 and 2020/21 La Niña events at 24 months lead and the 2021/22 and 2022/23 La Niña events at 12 and 8 months lead.

physics.comp-ph↗

arXiv2024

Towards Generalized Entropic Sparsification for Convolutional Neural Networks

Convolutional neural networks (CNNs) are reported to be overparametrized. The search for optimal (minimal) and sufficient architecture is an NP-hard problem as the hyperparameter space for possible network configurations is vast. Here, we introduce a layer-by-layer data-driven pruning method based on the mathematical idea aiming at a computationally-scalable entropic relaxation of the pruning problem. The sparse subnetwork is found from the pre-trained (full) CNN using the network entropy minimization as a sparsity constraint. This allows deploying a numerically scalable algorithm with a sublinear scaling cost. The method is validated on several benchmarks (architectures): (i) MNIST (LeNet) with sparsity 55%-84% and loss in accuracy 0.1%-0.5%, and (ii) CIFAR-10 (VGG-16, ResNet18) with sparsity 73-89% and loss in accuracy 0.1%-0.5%.

cs.CV↗

arXiv2023

On existence, uniqueness and scalability of adversarial robustness measures for AI classifiers

Simply-verifiable mathematical conditions for existence, uniqueness and explicit analytical computation of minimal adversarial paths (MAP) and minimal adversarial distances (MAD) for (locally) uniquely-invertible classifiers, for generalized linear models (GLM), and for entropic AI (EAI) are formulated and proven. Practical computation of MAP and MAD, their comparison and interpretations for various classes of AI tools (for neuronal networks, boosted random forests, GLM and EAI) are demonstrated on the common synthetic benchmarks: on a double Swiss roll spiral and its extensions, as well as on the two biomedical data problems (for the health insurance claim predictions, and for the heart attack lethality classification). On biomedical applications it is demonstrated how MAP provides unique minimal patient-specific risk-mitigating interventions in the predefined subsets of accessible control variables.

stat.ML↗

arXiv2023

Gauge-optimal approximate learning for small data classification problems

Small data learning problems are characterized by a significant discrepancy between the limited amount of response variable observations and the large feature space dimension. In this setting, the common learning tools struggle to identify the features important for the classification task from those that bear no relevant information, and cannot derive an appropriate learning rule which allows to discriminate between different classes. As a potential solution to this problem, here we exploit the idea of reducing and rotating the feature space in a lower-dimensional gauge and propose the Gauge-Optimal Approximate Learning (GOAL) algorithm, which provides an analytically tractable joint solution to the dimension reduction, feature segmentation and classification problems for small data learning problems. We prove that the optimal solution of the GOAL algorithm consists in piecewise-linear functions in the Euclidean space, and that it can be approximated through a monotonically convergent algorithm which presents -- under the assumption of a discrete segmentation of the feature space -- a closed-form solution for each optimization substep and an overall linear iteration cost scaling. The GOAL algorithm has been compared to other state-of-the-art machine learning (ML) tools on both synthetic data and challenging real-world applications from climate science and bioinformatics (i.e., prediction of the El Nino Southern Oscillation and inference of epigenetically-induced gene-activity networks from limited experimental data). The experimental results show that the proposed algorithm outperforms the reported best competitors for these problems both in learning performance and computational cost.

cs.LG↗

arXiv2023

Linearly-scalable learning of smooth low-dimensional patterns with permutation-aided entropic dimension reduction

In many data science applications, the objective is to extract appropriately-ordered smooth low-dimensional data patterns from high-dimensional data sets. This is challenging since common sorting algorithms are primarily aiming at finding monotonic orderings in low-dimensional data, whereas typical dimension reduction and feature extraction algorithms are not primarily designed for extracting smooth low-dimensional data patterns. We show that when selecting the Euclidean smoothness as a pattern quality criterium, both of these problems (finding the optimal 'crisp' data permutation and extracting the sparse set of permuted low-dimensional smooth patterns) can be efficiently solved numerically as one unsupervised entropy-regularized iterative optimization problem. We formulate and prove the conditions for monotonicity and convergence of this linearly-scalable (in dimension) numerical procedure, with the iteration cost scaling of $\mathcal{O}(DT^2)$, where $T$ is the size of the data statistics and $D$ is a feature space dimension. The efficacy of the proposed method is demonstrated through the examination of synthetic examples as well as a real-world application involving the identification of smooth bankruptcy risk minimizing transition patterns from high-dimensional economical data. The results showcase that the statistical properties of the overall time complexity of the method exhibit linear scaling in the dimensionality $D$ within the specified confidence intervals.

cs.LG↗

arXiv2021

Robust learning of data anomalies with analytically-solvable entropic outlier sparsification

Entropic Outlier Sparsification (EOS) is proposed as a robust computational strategy for the detection of data anomalies in a broad class of learning methods, including the unsupervised problems (like detection of non-Gaussian outliers in mostly-Gaussian data) and in the supervised learning with mislabeled data. EOS dwells on the derived analytic closed-form solution of the (weighted) expected error minimization problem subject to the Shannon entropy regularization. In contrast to common regularization strategies requiring computational costs that scale polynomial with the data dimension, identified closed-form solution is proven to impose additional iteration costs that depend linearly on statistics size and are independent of data dimension. Obtained analytic results also explain why the mixtures of spherically-symmetric Gaussians - used heuristically in many popular data analysis algorithms - represent an optimal choice for the non-parametric probability distributions when working with squared Euclidean distances, combining expected error minimality, maximal entropy/unbiasedness, and a linear cost scaling. The performance of EOS is compared to a range of commonly-used tools on synthetic problems and on partially-mislabeled supervised classification problems from biomedicine.

stat.ME↗

arXiv2021

On Computationally-Scalable Spatio-Temporal Regression Clustering of Precipitation Threshold Excesses

Focusing on regression based analysis of extremes in a presence of systematically missing covariates, this work presents a data-driven spatio-temporal regression based clustering of threshold excesses. It is shown that in a presence of systematically missing covariates the behavior of threshold excesses becomes nonstationary and nonhomogenous. The presented approach describes this complex behavior by a set of local stationary Generalized Pareto Distribution (GPD) models, where the parameters are expressed as regression models, and a latent spatio-temporal switching process. The spatio-temporal switching process is resolved by the nonparametric Finite Element Methodology for time series analysis with Bounded Variation of the model parameters (FEM-BV). The presented FEM-BV-GPD approach goes beyond strong a priori assumptions made in standard latent class models like Mixture Models and Hidden Markov Models. In addition, it provides a pragmatic description of the underlying dependency structure. The performance of the framework is demonstrated on historical precipitation data for Switzerland and compared with the results obtained by the standard methods on the same data.

stat.ME↗

arXiv2020

Physically-inspired computational tools for sharp detection of material inhomogeneities in magnetic imaging

Detection of material inhomogeneities is an important task in magnetic imaging and plays a significant role in understanding physical processes. For example, in spintronics, the sample heterogeneity determines the onset of current-driven magnetization motion. While often a significant effort is made in enhancing the resolution of an experimental technique to obtain a deeper insight into the physical properties, here we want to emphasize that an advantageous data analysis has the potential to provide a lot more insight into given data set, in particular when being close to the resolution limit where the noise becomes at least of the same order as the signal. In this work, we introduce two tools - the average latent dimension and average latent entropy - which allow for the detection of very subtle material inhomogeneity patterns in the data. For example, for the Ising model, we show that these tools are able to resolve exchange differences down to $1\%$. For a micromagnetic model, we demonstrate that the latent entropy can be used to detect changes in the easy axis anisotropy from magnetization data. We show that the latent entropy remains robust when imposing noise on the data, changing less than $0.3\%$ after adding Gaussian noise of the same amplitude as the signal. Furthermore, we demonstrate that these data-driven tools can be used to visualize inhomogeneities based on MOKE data of magnetic whirls and thereby can help to explicitly resolve impurities and pinning centers. To evaluate the performance of the average latent dimension and entropy, we show that they outperform common instruments ranging from standard statistics measures to state-of-the-art data analysis techniques such as Gaussian mixture models not only in recognition quality but also in the required computational cost.

physics.comp-ph↗

arXiv2020

On a scalable entropic breaching of the overfitting barrier in machine learning

Overfitting and treatment of "small data" are among the most challenging problems in the machine learning (ML), when a relatively small data statistics size $T$ is not enough to provide a robust ML fit for a relatively large data feature dimension $D$. Deploying a massively-parallel ML analysis of generic classification problems for different $D$ and $T$, existence of statistically-significant linear overfitting barriers for common ML methods is demonstrated. For example, these results reveal that for a robust classification of bioinformatics-motivated generic problems with the Long Short-Term Memory deep learning classifier (LSTM) one needs in a best case a statistics $T$ that is at least 13.8 times larger then the feature dimension $D$. It is shown that this overfitting barrier can be breached at a $10^{-12}$ fraction of the computational cost by means of the entropy-optimal Scalable Probabilistic Approximations algorithm (eSPA), performing a joint solution of the entropy-optimal Bayesian network inference and feature space segmentation problems. Application of eSPA to experimental single cell RNA sequencing data exhibits a 30-fold classification performance boost when compared to standard bioinformatics tools - and a 7-fold boost when compared to the deep learning LSTM classifier.

cs.LG↗