arXiv subjects

Alexander Kovalenko

Publications and source records attributed to Alexander Kovalenko.

6 recordsLinked to original sources

arXiv2026

CASCADE: Cross-scale Advective Super-resolution with Climate Assimilation and Downscaling Evolution

Super-resolution of geophysical fields presents unique challenges beyond natural image enhancement: fine-scale structures must respect physical dynamics, conserve mass and energy, and evolve coherently in time. These constraints are especially critical for extreme events, where rare, localized, high-intensity features drive impacts and where temporally inconsistent "hallucinated" detail can misrepresent hazards. We introduce CASCADE (Cross-scale Advective Super-resolution with Climate Assimilation and Downscaling Evolution), a framework that reframes spatiotemporal super-resolution as an explicit transport process across scales. Rather than hallucinating high-frequency content per pixel, CASCADE reconstructs fine structure by iteratively advecting coarse information along learned, flow-conditioned velocity fields through semi-Lagrangian warping. The architecture decomposes motion into resolved (large-scale) and subgrid (unresolved) components, mirroring the closure problem in numerical weather prediction, and enforces low-resolution consistency through an assimilation-style innovation step. Evaluated on SEVIR radar data for 4x super-resolution of severe convective storms, CASCADE outperforms strong baselines across both continuous metrics (PSNR, SSIM, MAE) and threshold-based skill scores (CSI, HSS, POD) while providing interpretable diagnostics through visualizable velocity and correction fields. By encoding advection as the fundamental operator rather than learning it implicitly, CASCADE produces temporally coherent, physically consistent, and mass-conserving reconstructions well suited to advection-dominated extremes in atmospheric and oceanic applications.

physics.ao-ph↗

arXiv2025

Preserving Vector Space Properties in Dimensionality Reduction: A Relationship Preserving Loss Framework

Dimensionality reduction can distort vector space properties such as orthogonality and linear independence, which are critical for tasks including cross-modal retrieval, clustering, and classification. We propose a Relationship Preserving Loss (RPL), a loss function that preserves these properties by minimizing discrepancies between relationship matrices (e.g., Gram or cosine) of high-dimensional data and their low-dimensional embeddings. RPL trains neural networks for non-linear projections and is supported by error bounds derived from matrix perturbation theory. Initial experiments suggest that RPL reduces embedding dimensions while largely retaining performance on downstream tasks, likely due to its preservation of key vector space properties. While we describe here the use of RPL in dimensionality reduction, this loss can also be applied more broadly, for example to cross-domain alignment and transfer learning, knowledge distillation, fairness and invariance, dehubbing, graph and manifold learning, and federated learning, where distributed embeddings must remain geometrically consistent.

cs.LG↗

arXiv2022

Linear Self-Attention Approximation via Trainable Feedforward Kernel

In pursuit of faster computation, Efficient Transformers demonstrate an impressive variety of approaches -- models attaining sub-quadratic attention complexity can utilize a notion of sparsity or a low-rank approximation of inputs to reduce the number of attended keys; other ways to reduce complexity include locality-sensitive hashing, key pooling, additional memory to store information in compacted or hybridization with other architectures, such as CNN. Often based on a strong mathematical basis, kernelized approaches allow for the approximation of attention with linear complexity while retaining high accuracy. Therefore, in the present paper, we aim to expand the idea of trainable kernel methods to approximate the self-attention mechanism of the Transformer architecture.

cs.LG↗

arXiv2022

Graph Neural Networks with Trainable Adjacency Matrices for Fault Diagnosis on Multivariate Sensor Data

Timely detected anomalies in the chemical technological processes, as well as the earliest detection of the cause of the fault, significantly reduce the production cost in the industrial factories. Data on the state of the technological process and the operation of production equipment are received by a large number of different sensors. To better predict the behavior of the process and equipment, it is necessary not only to consider the behavior of the signals in each sensor separately, but also to take into account their correlation and hidden relationships with each other. Graph-based data representation helps with this. The graph nodes can be represented as data from the different sensors, and the edges can display the influence of these data on each other. In this work, the possibility of applying graph neural networks to the problem of fault diagnosis in a chemical process is studied. It was proposed to construct a graph during the training of graph neural network. This allows to train models on data where the dependencies between the sensors are not known in advance. In this work, several methods for obtaining adjacency matrices were considered, as well as their quality was studied. It has also been proposed to use multiple adjacency matrices in one model. We showed state-of-the-art performance on the fault diagnosis task with the Tennessee Eastman Process dataset. The proposed graph neural networks outperformed the results of recurrent neural networks.

cs.AI↗

arXiv2021

SimpleTRON: Simple Transformer with O(N) Complexity

In this paper, we propose that the dot product pairwise matching attention layer, which is widely used in Transformer-based models, is redundant for the model performance. Attention, in its original formulation, has to be seen rather as a human-level tool to explore and/or visualize relevancy scores in sequential data. However, the way how it is constructed leads to significant computational complexity. Instead, we present SimpleTRON: Simple Transformer with O(N) Complexity, a simple and fast alternative without any approximation that, unlike other approximation models, does not have any architecture-related overhead and therefore can be seen as a purely linear Transformer-like model. This architecture, to the best of our knowledge, outperforms existing sub-quadratic attention approximation models on several tasks from the Long-Range Arena benchmark. Moreover, we show, that SimpleTRON can benefit from weight transfer from pretrained large language models, as its parameters can be fully transferable.

cs.CL↗

arXiv2021

Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks

Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks, where now excessively large models are used. However, such models face several problems during the learning process, mainly due to the redundancy of the individual neurons, which results in sub-optimal accuracy or the need for additional training steps. Here, we explore the diversity of the neurons within the hidden layer during the learning process, and analyze how the diversity of the neurons affects predictions of the model. As following, we introduce several techniques to dynamically reinforce diversity between neurons during the training. These decorrelation techniques improve learning at early stages and occasionally help to overcome local minima faster. Additionally, we describe novel weight initialization method to obtain decorrelated, yet stochastic weight initialization for a fast and efficient neural network training. Decorrelated weight initialization in our case shows about 40% relative increase in test accuracy during the first 5 epochs.

cs.LG↗