Searcharxiv⌕ Search

arXiv subjects

Eran Treister

Publications and source records attributed to Eran Treister.

At least 19 recordsLinked to original sources

Explicit Regularization for Flow Matching

Learning the underlying probability distribution from samples is a core problem in statistics and applied mathematics. Generative models based on diffusion and flow matching learn a velocity field whose flow transports a known distribution to the data distribution. They produce novel, realistic samples, although the empirical learning problem is ill posed near the data distribution. Its minimizer transports the samples onto the training data, and small perturbations of the data change the velocity field strongly near the end of the flow. The observed generalization is often attributed to implicit regularization by the network parameterization, early stopping of the training, and the time integration during sampling. Explicit regularization has advanced the theory and computation of many inverse problems and has recently been proposed to prevent memorization in generative models. With a regularizer on the magnitude of the velocity, however, the field approaches the memorizing velocity unless the regularization parameter grows to infinity fast enough, and it then stays bounded by a constant that is small only for large parameters. We show that penalizing the derivatives of order $k$ of the velocity renders the $D$-dimensional empirical problem well posed uniformly in time if $2k > D$ and the training points are in general position. Our experiments in one and two dimensions discretize the velocity with splines and use direct solvers for the regularized least-squares problems to exclude optimization error. Refining the discretization in space and time changes the error of the Hessian-regularized flows only marginally, whereas without the regularizer the error tends to that of memorization. These flows generalize, with errors comparable to or lower than those of well-tuned kernel density estimates.

cs.LG↗

Fast and Memory-Efficient Wavelet Convolutions via I/O-Aware Reformulation

Wavelet convolution (WTConv) has emerged as an increasingly popular drop-in replacement for standard convolutions, expanding a network's receptive field exponentially with the number of decomposition levels while keeping the parameter count linear. However, its reference implementation is severely memory-bound due to excessive data movement through high-bandwidth memory (HBM). We develop an I/O model of WTConv to characterize this bottleneck and use it to guide three algebraic reformulations: (1) recomputing the inexpensive Haar analysis butterfly on chip, (2) collapsing the multi-level synthesis cascade into a single closed-form pass indexed by output-coordinate bits, and (3) folding learned per-channel scales into the convolution weights. Together, these reformulations enable an I/O-aware fused implementation that substantially reduces HBM traffic. We evaluate the WTConvNeXt configuration across decomposition levels and a broad range of tensor shapes. Despite performing comparable arithmetic, the reference WTConv is substantially slower than the depthwise convolution it replaces. Our reformulation reduces modeled HBM traffic by approximately $2.55\times$, yielding up to a $4.35\times$ training speedup over the reference while roughly halving peak memory usage. Thus, our reformulation preserves the benefits of WTConv while substantially reducing its execution time and memory footprint, removing the systems overhead that previously limited its practical efficiency.

cs.CV↗

Factorized Gradients for Scalable Highly-expressive Parametric Diffeomorphisms

Diffeomorphisms provide a flexible, topology-preserving mathematical tool for modeling complex spatial or temporal deformations and are used in various scientific fields. However, making these deformations simultaneously highly expressive and computationally efficient remains challenging. Continuous Piecewise-Affine Based (CPAB) transformations (Freifeld et al., ICCV 2015; IEEE TPAMI 2017) offer an attractive solution by parameterizing a family of diffeomorphisms via continuous velocity fields that are piecewise affine with respect to a chosen tessellation of the domain. Importantly, $d=\dim(θ)$, the dimension of the parameter vector $θ$, depends on the fineness of the tessellation rather than on the data resolution. Despite this compact parameterization, CPAB optimization has long been hindered by the tight coupling between trajectory integration and gradient computation with respect to $θ$. We introduce FG-CPAB, a factorized-gradient formulation for scalable CPAB transformations. By separating integration from parameter-space projection, our method reduces gradient computation complexity from $\mathcal{O}(dTN)$ to $\mathcal{O}(TN+dC)$, where $T$ is the number of integration steps, $N$ is the number of transformed points, and $C$ is the number of tessellation cells. This reformulation yields speedups of up to four orders of magnitude (e.g., $\approx 10^4\times$ in 2D), substantial memory savings (e.g., $\approx 500\times$ reduction in peak GPU memory usage in 2D), and dramatically higher expressiveness. This makes fine-tessellation CPAB practical in 2D and 3D for the first time, enabling highly expressive parametric diffeomorphisms.

cs.CV↗

Dispersion-corrected compact finite difference discretizations for the elastic Helmholtz equation in two and three dimensions

We develop a family of compact, dispersion-corrected marker-and-cell (MAC)-based finite-difference discretizations for the elastic Helmholtz equation. While dispersion correction is well established for scalar Helmholtz problems, fewer approaches exist for the elastic problems, and most use non-compact stencils. To this end, we introduce a generalized MAC (GMAC) framework that enables compact discretization of the elastic Helmholtz equation. We prove that even though GMAC is non-symmetric, this discretization produces no numerical dissipation and no shear-mode separation. For certain weights, GMAC has lower polarization error than standard MAC. We also introduce an elastic dispersion correction (EDC) process, extending real-shifted-wavenumber dispersion correction methods from acoustic to elastic Helmholtz problems. We prove and demonstrate numerically that GMAC is second-order accurate. Furthermore, we show that EDC substantially reduces dispersion for standard MAC and can provide additional correction in some instances of GMAC. Dispersion analysis shows that the resulting method outperforms existing compact finite-difference discretizations in dispersion accuracy and is competitive with some non-compact schemes. We give numerical evidence for improved phase alignment in mildly heterogeneous media.

math.NA↗

Vanka-smoothed shifted Laplacian multigrid preconditioners for the Helmholtz equations

We present an improved multigrid preconditioner for the acoustic Helmholtz equation with enhanced scalability. Standard multigrid fails to converge for the Helmholtz equation, and the well-known complex shifted Laplacian method overcomes it by adding a complex shift and using the shifted system as a preconditioner. However, the added complex shift grows with the frequency and interferes with the preconditioner's scalability. In this work, we present an additive Vanka smoother that requires a much lower shift than point-wise smoothers, and thereby enhances the scalability. By carefully designing different ingredients of the multigrid cycle, the presented method enables deep V-cycles with a small and bounded shift, even when many levels are used. We validate our method theoretically by local Fourier analysis, and hold numerical experiments for homogeneous and heterogeneous media. We show that our method outperforms plain shifted Laplacian in terms of runtimes and performs well on challenging geophysical media in 2D and 3D.

math.NA↗

RAPNet: Accelerating Algebraic Multigrid with Learned Sparse Corrections

The scalable solution of large sparse linear systems is a bottleneck in scientific computing and graph analysis. While algebraic multigrid (AMG) offers optimal linear scaling, its performance is severely constrained by the trade-off between the sparsity and convergence quality of coarse-grid operators. Classical AMG heuristics struggle to balance these objectives, often sacrificing stability or performance for sparsity. We propose RAPNet, a graph neural network (GNN) framework that resolves this trade-off by learning to generate sparse, robust coarse operators directly from the sparse algebraic system. Key to our approach is a level-wise training strategy that enables learning from small subgraphs and generalization to million-node domains, bypassing the bottlenecks of prior neural AMG attempts. RAPNet executes exclusively during the solver setup phase, ensuring that the solve phase retains its favorable computational properties. We show that our method outperforms classical non-Galerkin baselines on diverse PDE discretizations and graph Laplacians, making it particularly effective for multi-query tasks such as eigenproblems, time-dependent simulations, and inverse or design problems.

cs.LG↗

Scalable Multigrid Solver for the Helmholtz Equation: Real-Shifted Coarse Grid Correction

We present a convergent and scalable multigrid solver for high-frequency Helmholtz equations. Standard multigrid methods do not converge for high-frequency Helmholtz problems, and a common cure is adding a complex shift and using the shifted operator as a preconditioner. Nevertheless, the complex shift prevents scalability. In this work we present a new method that achieves scalable convergence of a 3-level cycle without a complex shift. Our key idea is real-shifting the coarsest grid Galerkin operator, to correct the numerical dispersion between the grids. We show that this real-shifted coarse grid correction leads to a scalable 3-level method, for problems with 12 grid points per wavelength on the fine grid, and a convergent cycle with very few iterations for 11 grid points per wavelength, using standard point-smoothers. For problems with 10 grid points per wavelength, our method combined with a modest complex shift outperforms the standard complex shifted Laplacian method by an order of magnitude. We demonstrate wavenumber independent convergence for heterogeneous geophysical media in 2D and 3D.

math.NA↗

A SIMPLE-Based Preconditioned Solver for the Direct-Forcing Immersed Boundary Method

We present a robust and scalable solver for direct-forcing immersed boundary simulations, based on a preconditioned SIMPLE algorithm. The method applies block elimination to the pressure-force coupled system, and utilizes the discrete Laplacian operator as an efficient preconditioner for the resulting Schur complement. We rigorously demonstrate the spectral equivalence between the Schur complement and the discrete Laplacian, ensuring convergence behavior that is independent of grid resolution and physical parameters. This enables accurate, stable, and efficient two-way coupled fluid-structure interaction (FSI) simulations with moving boundaries and significant added-mass effects. These simulations are all executable on standard computing platforms. Extensive validation and verification - including simulations of oscillating, sedimenting, and buoyant spheres, as well as configurations involving multiple immersed bodies - confirm the solver's accuracy and efficiency across a broad range of FSI scenarios. The proposed approach introduces a novel and accessible framework for immersed boundary simulations requiring strong pressure-force coupling.

physics.comp-ph↗

A block-acoustic preconditioner for the elastic Helmholtz equation

We present a novel block-preconditioner for the elastic Helmholtz equation, based on a reduction to acoustic Helmholtz equations. Both versions of the Helmholtz equations are challenging numerically. The elastic Helmholtz equation is larger, as a system of PDEs, and harder to solve due to its more complicated physics. It was recently suggested that the elastic Helmholtz equation can be reformulated as a generalized saddle-point system, opening the door to the current development. Utilizing the approximate commutativity of the underlying differential operators, we suggest a block-triangular preconditioner whose diagonal blocks are acoustic Helmholtz operators. Thus, we enable the solution of the elastic version using virtually any existing solver for the acoustic version as a black-box. We prove a sufficient condition for the convergence of our method, that sheds light on the long questioned role of the commutator in the convergence of approximate commutator preconditioners. We show scalability of our preconditioner with respect to the Poisson ratio and with respect to the grid size. We compare our approach, combined with multigrid solve of each block, to a recent monolithic multigrid method for the elastic Helmholtz equation. The block-acoustic multigrid achieves a lower computational cost for various heterogeneous media, and a significantly lower memory consumption, compared to the monolithic approach. It results in a fast solution method for wave propagation problems in challenging heterogeneous media in 2D and 3D.

math.NA↗

Reversing Large Language Models for Efficient Training and Fine-Tuning

Large Language Models (LLMs) are known for their expensive and time-consuming training. Thus, oftentimes, LLMs are fine-tuned to address a specific task, given the pretrained weights of a pre-trained LLM considered a foundation model. In this work, we introduce memory-efficient, reversible architectures for LLMs, inspired by symmetric and symplectic differential equations, and investigate their theoretical properties. Different from standard, baseline architectures that store all intermediate activations, the proposed models use time-reversible dynamics to retrieve hidden states during backpropagation, relieving the need to store activations. This property allows for a drastic reduction in memory consumption, allowing for the processing of larger batch sizes for the same available memory, thereby offering improved throughput. In addition, we propose an efficient method for converting existing, non-reversible LLMs into reversible architectures through fine-tuning, rendering our approach practical for exploiting existing pre-trained models. Our results show comparable or improved performance on several datasets and benchmarks, on several LLMs, building a scalable and efficient path towards reducing the memory and computational costs associated with both training from scratch and fine-tuning of LLMs.

cs.CL↗

Towards Efficient Training of Graph Neural Networks: A Multiscale Approach

Graph Neural Networks (GNNs) have become powerful tools for learning from graph-structured data, finding applications across diverse domains. However, as graph sizes and connectivity increase, standard GNN training methods face significant computational and memory challenges, limiting their scalability and efficiency. In this paper, we present a novel framework for efficient multiscale training of GNNs. Our approach leverages hierarchical graph representations and subgraphs, enabling the integration of information across multiple scales and resolutions. By utilizing coarser graph abstractions and subgraphs, each with fewer nodes and edges, we significantly reduce computational overhead during training. Building on this framework, we propose a suite of scalable training strategies, including coarse-to-fine learning, subgraph-to-full-graph transfer, and multiscale gradient computation. We also provide some theoretical analysis of our methods and demonstrate their effectiveness across various datasets and learning tasks. Our results show that multiscale training can substantially accelerate GNN training for large scale problems while maintaining, or even improving, predictive performance.

cs.LG↗

Disparate Conditional Prediction in Multiclass Classifiers

We propose methods for auditing multiclass classifiers for fairness under multiclass equalized odds,by estimating the deviation from equalized odds when the classifier is not completely fair. We generalize to multiclass classifiers the measure of Disparate Conditional Prediction (DCP), originally suggested by Sabato & Yom-Tov (2020) for binary classifiers. DCP is defined as the fraction of the population for which the classifier predicts with conditional prediction probabilities that differ from the closest common baseline. We provide new local-optimization methods for estimating the multiclass DCPunder two different regimes,one in which the conditional confusion matrices for each protected sub-population are known, and one in which these cannot be estimated, for instance, because the classifier is inaccessible or because good-quality individual-level data is not available. These methods can be used to detect classifiers that likely treat a significant fraction of the population unfairly. Experiments demonstrate the accuracy of the methods. Code is provided at https://github.com/sivansabato/DCPmulticlass.

cs.LG↗

Zero-Shot Detection of LLM-Generated Code via Approximated Task Conditioning

Detecting Large Language Model (LLM)-generated code is a growing challenge with implications for security, intellectual property, and academic integrity. We investigate the role of conditional probability distributions in improving zero-shot LLM-generated code detection, when considering both the code and the corresponding task prompt that generated it. Our key insight is that when evaluating the probability distribution of code tokens using an LLM, there is little difference between LLM-generated and human-written code. However, conditioning on the task reveals notable differences. This contrasts with natural language text, where differences exist even in the unconditional distributions. Leveraging this, we propose a novel zero-shot detection approach that approximates the original task used to generate a given code snippet and then evaluates token-level entropy under the approximated task conditioning (ATC). We further provide a mathematical intuition, contextualizing our method relative to previous approaches. ATC requires neither access to the generator LLM nor the original task prompts, making it practical for real-world applications. To the best of our knowledge, it achieves state-of-the-art results across benchmarks and generalizes across programming languages, including Python, CPP, and Java. Our findings highlight the importance of task-level conditioning for LLM-generated code detection. The supplementary materials and code are available at https://github.com/maorash/ATC, including the dataset gathering implementation, to foster further research in this area.

cs.CL↗

Understanding and Improving Laplacian Positional Encodings For Temporal GNNs

Temporal graph learning has applications in recommendation systems, traffic forecasting, and social network analysis. Although multiple architectures have been introduced, progress in positional encoding for temporal graphs remains limited. Extending static Laplacian eigenvector approaches to temporal graphs through the supra-Laplacian has shown promise, but also poses key challenges: high eigendecomposition costs, limited theoretical understanding, and ambiguity about when and how to apply these encodings. In this paper, we address these issues by (1) offering a theoretical framework that connects supra-Laplacian encodings to per-time-slice encodings, highlighting the benefits of leveraging additional temporal connectivity, (2) introducing novel methods to reduce the computational overhead, achieving up to 56x faster runtimes while scaling to graphs with 50,000 active nodes, and (3) conducting an extensive experimental study to identify which models, tasks, and datasets benefit most from these encodings. Our findings reveal that while positional encodings can significantly boost performance in certain scenarios, their effectiveness varies across different models.

cs.LG↗

Improving the Effective Receptive Field of Message-Passing Neural Networks

Message-Passing Neural Networks (MPNNs) have become a cornerstone for processing and analyzing graph-structured data. However, their effectiveness is often hindered by phenomena such as over-squashing, where long-range dependencies or interactions are inadequately captured and expressed in the MPNN output. This limitation mirrors the challenges of the Effective Receptive Field (ERF) in Convolutional Neural Networks (CNNs), where the theoretical receptive field is underutilized in practice. In this work, we show and theoretically explain the limited ERF problem in MPNNs. Furthermore, inspired by recent advances in ERF augmentation for CNNs, we propose an Interleaved Multiscale Message-Passing Neural Networks (IM-MPNN) architecture to address these problems in MPNNs. Our method incorporates a hierarchical coarsening of the graph, enabling message-passing across multiscale representations and facilitating long-range interactions without excessive depth or parameterization. Through extensive evaluations on benchmarks such as the Long-Range Graph Benchmark (LRGB), we demonstrate substantial improvements over baseline MPNNs in capturing long-range dependencies while maintaining computational efficiency.

cs.LG↗

Towards Croppable Implicit Neural Representations

Implicit Neural Representations (INRs) have peaked interest in recent years due to their ability to encode natural signals using neural networks. While INRs allow for useful applications such as interpolating new coordinates and signal compression, their black-box nature makes it difficult to modify them post-training. In this paper we explore the idea of editable INRs, and specifically focus on the widely used cropping operation. To this end, we present Local-Global SIRENs -- a novel INR architecture that supports cropping by design. Local-Global SIRENs are based on combining local and global feature extraction for signal encoding. What makes their design unique is the ability to effortlessly remove specific portions of an encoded signal, with a proportional weight decrease. This is achieved by eliminating the corresponding weights from the network, without the need for retraining. We further show how this architecture can be used to support the straightforward extension of previously encoded signals. Beyond signal editing, we examine how the Local-Global approach can accelerate training, enhance encoding of various signals, improve downstream performance, and be applied to modern INRs such as INCODE, highlighting its potential and flexibility. Code is available at https://github.com/maorash/Local-Global-INRs.

cs.CV↗

Wavelet Convolutions for Large Receptive Fields

In recent years, there have been attempts to increase the kernel size of Convolutional Neural Nets (CNNs) to mimic the global receptive field of Vision Transformers' (ViTs) self-attention blocks. That approach, however, quickly hit an upper bound and saturated way before achieving a global receptive field. In this work, we demonstrate that by leveraging the Wavelet Transform (WT), it is, in fact, possible to obtain very large receptive fields without suffering from over-parameterization, e.g., for a $k \times k$ receptive field, the number of trainable parameters in the proposed method grows only logarithmically with $k$. The proposed layer, named WTConv, can be used as a drop-in replacement in existing architectures, results in an effective multi-frequency response, and scales gracefully with the size of the receptive field. We demonstrate the effectiveness of the WTConv layer within ConvNeXt and MobileNetV2 architectures for image classification, as well as backbones for downstream tasks, and show it yields additional properties such as robustness to image corruption and an increased response to shapes over textures. Our code is available at https://github.com/BGU-CS-VIL/WTConv.

cs.CV↗

Global-Local Graph Neural Networks for Node-Classification

The task of graph node classification is often approached by utilizing a local Graph Neural Network (GNN), that learns only local information from the node input features and their adjacency. In this paper, we propose to improve the performance of node classification GNNs by utilizing both global and local information, specifically by learning label- and node- features. We therefore call our method Global-Local-GNN (GLGNN). To learn proper label features, for each label, we maximize the similarity between its features and nodes features that belong to the label, while maximizing the distance between nodes that do not belong to the considered label. We then use the learnt label features to predict the node classification map. We demonstrate our GLGNN using three different GNN backbones, and show that our approach improves baseline performance, revealing the importance of global information utilization for node classification.

cs.LG↗