arXiv subjects

Andrei Ivanov

Publications and source records attributed to Andrei Ivanov.

At least 19 recordsLinked to original sources

arXiv2025

PerfDojo: Automated ML Library Generation for Heterogeneous Architectures

The increasing complexity of machine learning models and the proliferation of diverse hardware architectures (CPUs, GPUs, accelerators) make achieving optimal performance a significant challenge. Heterogeneity in instruction sets, specialized kernel requirements for different data types and model features (e.g., sparsity, quantization), and architecture-specific optimizations complicate performance tuning. Manual optimization is resource-intensive, while existing automatic approaches often rely on complex hardware-specific heuristics and uninterpretable intermediate representations, hindering performance portability. We introduce PerfLLM, a novel automatic optimization methodology leveraging Large Language Models (LLMs) and Reinforcement Learning (RL). Central to this is PerfDojo, an environment framing optimization as an RL game using a human-readable, mathematically-inspired code representation that guarantees semantic validity through transformations. This allows effective optimization without prior hardware knowledge, facilitating both human analysis and RL agent training. We demonstrate PerfLLM's ability to achieve significant performance gains across diverse CPU (x86, Arm, RISC-V) and GPU architectures.

cs.PF↗

arXiv2025

Nanoscale lattice heterostructure in high Tc superconductors

Low temperature superconductivity was known since 1957 to be described by BCS theory for an effective single band metals controlled by the density of states at the Fermi level, very far from band edges, the electron phonon coupling, and the energy of the boson in the pairing interaction w0, but BCS has failed to predict high temperature superconductivity in different materials above about 23 K. High temperature superconductivity above 35 K since 1986 has been a matter of materials science where manipulating the lattice complexity of high temperature superconducting ceramic oxides (HTSC) has driven material scientists to grow new HTSC quantum materials up to 138K in HgBa2Ca2Cu3O8 (Hg1223) at ambient pressure and near room temperature in pressurized hydrides. This perspective covers the major results of materials scientist in these last 39 years investigating the role of lattice inhomogeneity detected in these new quantum complex materials. We highlight the nanoscale heterogeneity in these complex materials and elucidate their special role played in the physics for HTSC. Especially, it is pointed out that the geometry of lattice and charge complex heterogeneity at nanoscale is essential and intrinsic in the mechanism of rising quantum coherence at high temperature

cond-mat.supr-con↗

arXiv2023

VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores

The increasing success and scaling of Deep Learning models demands higher computational efficiency and power. Sparsification can lead to both smaller models as well as higher compute efficiency, and accelerated hardware is becoming available. However, exploiting it efficiently requires kernel implementations, pruning algorithms, and storage formats, to utilize hardware support of specialized sparse vector units. An example of those are the NVIDIA's Sparse Tensor Cores (SPTCs), which promise a 2x speedup. However, SPTCs only support the 2:4 format, limiting achievable sparsity ratios to 50%. We present the V:N:M format, which enables the execution of arbitrary N:M ratios on SPTCs. To efficiently exploit the resulting format, we propose Spatha, a high-performance sparse-library for DL routines. We show that Spatha achieves up to 37x speedup over cuBLAS. We also demonstrate a second-order pruning technique that enables sparsification to high sparsity ratios with V:N:M and little to no loss in accuracy in modern transformers.

cs.DC↗

arXiv2023

Cached Operator Reordering: A Unified View for Fast GNN Training

Graph Neural Networks (GNNs) are a powerful tool for handling structured graph data and addressing tasks such as node classification, graph classification, and clustering. However, the sparse nature of GNN computation poses new challenges for performance optimization compared to traditional deep neural networks. We address these challenges by providing a unified view of GNN computation, I/O, and memory. By analyzing the computational graphs of the Graph Convolutional Network (GCN) and Graph Attention (GAT) layers -- two widely used GNN layers -- we propose alternative computation strategies. We present adaptive operator reordering with caching, which achieves a speedup of up to 2.43x for GCN compared to the current state-of-the-art. Furthermore, an exploration of different caching schemes for GAT yields a speedup of up to 1.94x. The proposed optimizations save memory, are easily implemented across various hardware platforms, and have the potential to alleviate performance bottlenecks in training large-scale GNN models.

cs.LG↗

arXiv2023

TMPNN: High-Order Polynomial Regression Based on Taylor Map Factorization

Polynomial regression is widely used and can help to express nonlinear patterns. However, considering very high polynomial orders may lead to overfitting and poor extrapolation ability for unseen data. The paper presents a method for constructing a high-order polynomial regression based on the Taylor map factorization. This method naturally implements multi-target regression and can capture internal relationships between targets. Additionally, we introduce an approach for model interpretation in the form of systems of differential equations. By benchmarking on UCI open access datasets, Feynman symbolic regression datasets, and Friedman-1 datasets, we demonstrate that the proposed method performs comparable to the state-of-the-art regression methods and outperforms them on specific tasks.

cs.LG↗

arXiv2023

STen: Productive and Efficient Sparsity in PyTorch

As deep learning models grow, sparsity is becoming an increasingly critical component of deep neural networks, enabling improved performance and reduced storage. However, existing frameworks offer poor support for sparsity. Specialized sparsity engines focus exclusively on sparse inference, while general frameworks primarily focus on sparse tensors in classical formats and neglect the broader sparsification pipeline necessary for using sparse models, especially during training. Further, existing frameworks are not easily extensible: adding a new sparse tensor format or operator is challenging and time-consuming. To address this, we propose STen, a sparsity programming model and interface for PyTorch, which incorporates sparsity layouts, operators, and sparsifiers, in an efficient, customizable, and extensible framework that supports virtually all sparsification methods. We demonstrate this by developing a high-performance grouped n:m sparsity layout for CPU inference at moderate sparsity. STen brings high performance and ease of use to the ML community, making sparsity easily accessible.

cs.LG↗

arXiv2023

Eigenvalues and Eigenstates of Quantum Rabi Model

The scientific interest in the analytical solution of the quantum Rabi model is due to the widespread use of this simple model in quantum optics, quantum computing, cavity QED, and nanoelectromechanical systems. This interest is related to the need for the theoretical description of the interaction of a two-level system with a quantum oscillator in the case when the rotating wave approximation fails. In this Letter, we present an approach to the exact diagonalization of the quantum Rabi Hamiltonian. This approach is based on the properties of the Pauli operators and allows us to readily solve the stationary Schrodinger equation for a two-level system. First, we demonstrate the applicability of the approach to the Jaynes-Cummings Hamiltonian to get the well-known solution. Then, we obtain the eigenvalues and eigenstates for the quantum Rabi Hamiltonian using the proposed approach. It is shown that the obtained eigenstates can be represented in the basis of the eigenstates of the Jaynes-Cummings Hamiltonian.

quant-ph↗

arXiv2022

SAGE: Software-based Attestation for GPU Execution

With the application of machine learning to security-critical and sensitive domains, there is a growing need for integrity and privacy in computation using accelerators, such as GPUs. Unfortunately, the support for trusted execution on GPUs is currently very limited - trusted execution on accelerators is particularly challenging since the attestation mechanism should not reduce performance. Although hardware support for trusted execution on GPUs is emerging, we study purely software-based approaches for trusted GPU execution. A software-only approach offers distinct advantages: (1) complement hardware-based approaches, enhancing security especially when vulnerabilities in the hardware implementation degrade security, (2) operate on GPUs without hardware support for trusted execution, and (3) achieve security without reliance on secrets embedded in the hardware, which can be extracted as history has shown. In this work, we present SAGE, a software-based attestation mechanism for GPU execution. SAGE enables secure code execution on NVIDIA GPUs of the Ampere architecture (A100), providing properties of code integrity and secrecy, computation integrity, as well as data integrity and secrecy - all in the presence of malicious code running on the GPU and CPU. Our evaluation demonstrates that SAGE is already practical today for executing code in a trustworthy way on GPUs without specific hardware support.

cs.CR↗

arXiv2022

A Data-Centric Optimization Framework for Machine Learning

Rapid progress in deep learning is leading to a diverse set of quickly changing models, with a dramatically growing demand for compute. However, as frameworks specialize performance optimization to patterns in popular networks, they implicitly constrain novel and diverse models that drive progress in research. We empower deep learning researchers by defining a flexible and user-customizable pipeline for optimizing training of arbitrary deep neural networks, based on data movement minimization. The pipeline begins with standard networks in PyTorch or ONNX and transforms computation through progressive lowering. We define four levels of general-purpose transformations, from local intra-operator optimizations to global data movement reduction. These operate on a data-centric graph intermediate representation that expresses computation and data movement at all levels of abstraction, including expanding basic operators such as convolutions to their underlying computations. Central to the design is the interactive and introspectable nature of the pipeline. Every part is extensible through a Python API, and can be tuned interactively using a GUI. We demonstrate competitive performance or speedups on ten different networks, with interactive optimizations discovering new opportunities in EfficientNet.

cs.LG↗

arXiv2021

Data Movement Is All You Need: A Case Study on Optimizing Transformers

Transformers are one of the most important machine learning workloads today. Training one is a very compute-intensive task, often taking days or weeks, and significant attention has been given to optimizing transformers. Despite this, existing implementations do not efficiently utilize GPUs. We find that data movement is the key bottleneck when training. Due to Amdahl's Law and massive improvements in compute performance, training has now become memory-bound. Further, existing frameworks use suboptimal data layouts. Using these insights, we present a recipe for globally optimizing data movement in transformers. We reduce data movement by up to 22.91% and overall achieve a 1.30x performance improvement over state-of-the-art frameworks when training a BERT encoder layer and 1.19x for the entire BERT. Our approach is applicable more broadly to optimizing deep neural networks, and offers insight into how to tackle emerging performance bottlenecks.

cs.LG↗

arXiv2020

Physics-Based Deep Neural Networks for Beam Dynamics in Charged Particle Accelerators

This paper presents a novel approach for constructing neural networks which model charged particle beam dynamics. In our approach, the Taylor maps arising in the representation of dynamics are mapped onto the weights of a polynomial neural network. The resulting network approximates the dynamical system with perfect accuracy prior to training and provides a possibility to tune the network weights on additional experimental data. We propose a symplectic regularization approach for such polynomial neural networks that always restricts the trained model to Hamiltonian systems and significantly improves the training procedure. The proposed networks can be used for beam dynamics simulations or for fine-tuning of beam optics models with experimental data. The structure of the network allows for the modeling of large accelerators with a large number of magnets. We demonstrate our approach on the examples of the existing PETRA III and the planned PETRA IV storage rings at DESY.

cs.NE↗

arXiv2020

Physics-based polynomial neural networks for one-shot learning of dynamical systems from one or a few samples

This paper discusses an approach for incorporating prior physical knowledge into the neural network to improve data efficiency and the generalization of predictive models. If the dynamics of a system approximately follows a given differential equation, the Taylor mapping method can be used to initialize the weights of a polynomial neural network. This allows the fine-tuning of the model from one training sample of real system dynamics. The paper describes practical results on real experiments with both a simple pendulum and one of the largest worldwide X-ray source. It is demonstrated in practice that the proposed approach allows recovering complex physics from noisy, limited, and partial observations and provides meaningful predictions for previously unseen inputs. The approach mainly targets the learning of physical systems when state-of-the-art models are difficult to apply given the lack of training data.

cs.NE↗

arXiv2020

Vibronic coupling induced by fast Rabi oscillations for kinetic energy control in free atom

Vibronic coupling effects usually manifest themselves in molecules and crystals rather than in an unbound atoms. We theoretically demonstrate the existence of vibronic states in a moving two-level atom exposed to a strong electromagnetic wave. In this case, the Rabi oscillations of the electron density give rise to periodic displacements of the atom center of mass with the Rabi frequency. The periodic displacements mix the Stark split electron levels and lead to the establishment of a channel for energy transfer in the photon-electron-center-of-mass system. Such a channel paves the way for fast control of the kinetic energy of a free atom with the use of the parameters of an electromagnetic wave. Thus, the system kinetic energy can be decreased or increased if the detuning between the laser pulse frequency and the optical transition is positive or negative, respectively. For actual values of the detuning, the pulse duration should be an order of magnitude longer than the lifetime of the excited atomic level in order to complete the energy transfer process.

quant-ph↗

arXiv2019

Polynomial Neural Networks and Taylor maps for Dynamical Systems Simulation and Learning

The connection of Taylor maps and polynomial neural networks (PNN) to solve ordinary differential equations (ODEs) numerically is considered. Having the system of ODEs, it is possible to calculate weights of PNN that simulates the dynamics of these equations. It is shown that proposed PNN architecture can provide better accuracy with less computational time in comparison with traditional numerical solvers. Moreover, neural network derived from the ODEs can be used for simulation of system dynamics with different initial conditions, but without training procedure. On the other hand, if the equations are unknown, the weights of the PNN can be fitted in a data-driven way. In the paper we describe the connection of PNN with differential equations in a theoretical way along with the examples for both dynamics simulation and learning with data.

cs.NE↗

arXiv2019

Matrix Lie Maps and Neural Networks for Solving Differential Equations

The coincidence between polynomial neural networks and matrix Lie maps is discussed in the article. The matrix form of Lie transform is an approximation of the general solution of the nonlinear system of ordinary differential equations. It can be used for solving systems of differential equations more efficiently than traditional step-by-step numerical methods. Implementation of the Lie map as a polynomial neural network provides a tool for both simulation and data-driven identification of dynamical systems. If the differential equation is provided, training a neural network is unnecessary. The weights of the network can be directly calculated from the equation. On the other hand, for data-driven system learning, the weights can be fitted without any assumptions in view of differential equations. The proposed technique is discussed in the examples of both ordinary and partial differential equations. The building of a polynomial neural network that simulates the Van der Pol oscillator is discussed. For this example, we consider learning the dynamics from a single solution of the system. We also demonstrate the building of the neural network that describes the solution of Burgers' equation that is a fundamental partial differential equation.

cs.NE↗

arXiv2018

Lie Transform--based Neural Networks for Dynamics Simulation and Learning

In the article, we discuss the architecture of the polynomial neural network that corresponds to the matrix representation of Lie transform. The matrix form of Lie transform is an approximation of the general solution of the nonlinear system of ordinary differential equations. The proposed architecture can be trained with small data sets, extrapolate predictions outside the training data, and provide a possibility for interpretation. We provide a theoretical explanation of the proposed architecture, as well as demonstrate it in several applications. We present the results of modeling and identification for both simple and well-known dynamical systems, and more complicated examples from price dynamics, chemistry, and accelerator physics. From a practical point of view, we describe the training of a Lie transform--based neural network with a small data set containing only 10 data points. We also demonstrate an interpretation of the fitted neural network by converting it to a system of differential equations.

cs.NE↗

arXiv2017

Photoinduced vibronic coupling in two-level dissipative systems

Interaction of an electron system with a strong electromagnetic wave leads to rearrangement both the electron and vibrational energy spectra of a dissipative system. For instance, the optically coupled electron levels become split in the conditions of the ac Stark effect that gives rise to appearance of the nonadiabatic coupling between the electron and vibrational motions. The nonadiabatic coupling exerts a substantial impact on the electron and phonon dynamics and must be taken into account to determine the system wave functions. In this paper, the vibronic coupling induced by the ac Stark effect is considered. It is shown that the interaction between the electron states dressed by an electromagnetic field and the forced vibrations of reservoir oscillators under the action of rapid changing of the electron density with the Rabi frequency is responsible for establishment of the photoinduced vibronic coupling. However, if the resonance conditions for the optical phonon frequency and the transition frequency of electrons in the dressed state basis are satisfied, the vibronic coupling is due to the electron-phonon interaction. Additionally, photoinduced vibronic coupling results in appearance of the doubly dressed states which are formed by both the electron-photon and electron-vibrational interactions.

cond-mat.other↗

arXiv2017

Energy Transfer Controlled by Dynamical Stark Shift in Two-level Dissipative Systems

A strong electromagnetic field interacting with an electron system generates both the Rabi oscillations and the Stark splitting of the electron density. Changing of the electron density gives rise to nonadiabatic effects due to existence of the electron-vibrational interaction in a dissipative system. In this Letter, the mechanism of energy transfer between the electron system and the phonon reservoir is presented. This mechanism is based on establishment of the coupling between the electron states dressed by the electromagnetic field and the forced vibrations of reservoir oscillators under the action of rapid changing of the electron density with the Rabi frequency. The photoinduced vibronic coupling results in appearance of the states that are doubly dressed by interaction, first time due to the electron-photon interaction, and second time due to the electron-vibrational interaction. Moreover, this coupling opens the way to control energy which can be transferred to (heating) or removed from (cooling) the phonon reservoir depending on the parameters of the electromagnetic pulse.

cond-mat.mes-hall↗