arXiv subjects

Mojtaba Soltanalian

Publications and source records attributed to Mojtaba Soltanalian.

At least 19 recordsLinked to original sources

arXiv2026

Training Under Challenge: Executable Certificates and Challenge-Closed Optimality for Neural Networks

A flat training curve does not reveal whether a neural network has reached a global optimum, is locally trapped, is representation-limited, or is mismatched to its trainer. We introduce Training Under Challenge, an executable-certificate framework in which predeclared, architecture-valid procedures construct complete alternatives in the same certified class and reevaluate the same objective. Any lower-valued candidate is a replayable witness that lower-bounds the checkpoint's empirical global-optimality gap. Passing a finite suite is only suite-relative; global-gap conclusions require a separately justified coverage mechanism. We define a resource-indexed challenge-power modulus that characterizes the largest gap compatible with passage. For squared loss, current block-decrease operators make coverage checkable and yield uniform and realized-residual bounds. We prove the converse frontier: without coverage, a first-order ReLU trainer can reach infinitely many exact conditional head optima while converging to a non-global point. On a channel-gated ResNet-18 distillation problem with known optimum, eight internal challenges cover all 240 audited output directions, and realized-residual bounds lie within factors of 1.74--3.02 of the true gap. Paired predictive certificates separate decoder under-use from representation insufficiency, while quantized-denoising studies demonstrate diagnosis, repair, and current-state recertification.

cs.LG↗

arXiv2026

The Rank-Collapse Principle for Quadratic Optimization

Quadratic optimization becomes hard as soon as either the matrix in the quadratic form has an unfavorable curvature or the feasible set is discrete, combinatorial, or otherwise nonconvex. A complementary phenomenon is also well known in the signal-processing and optimization communities: when the matrix in the quadratic form has small rank, some hard-looking quadratic programs admit exact polynomial-time algorithms for fixed rank. We study the common positive-semidefinite geometry behind this phenomenon. If $Q=BB^\top$ is positive semidefinite, the objective depends on $x$ only through the rank-space shadow $y=B^\top x$. Every optimal shadow $y^*$ uniquely maximizes the linear functional defined by its own direction and satisfies a quantitative quadratic margin. Thus nonlinear optimality collapses to a low-dimensional, self-generated linear exposure direction. We call this the rank-collapse principle. The principle alone does not imply a finite candidate set: efficient exact optimization additionally depends on the projected or active geometry of the feasible family. We organize this distinction through projected-shadow scattering and active-structure collapse, relate it explicitly to established zonotope, convex-combinatorial, edge-skeleton, projected-normal-fan, and fixed-rank sparse-PCA methods, and derive tie-safe consequences for binary and finite-phase vectors, cardinality constraints, matroid bases, and sparse PCA. We also give directional-stability and approximately low-rank certificates, together with reproducible experiments. The paper's contribution is a unified, careful framework and a set of quantitative consequences, rather than a claim to originate the known fixed-rank tractability results that motivate it.

math.OC↗

arXiv2026

When Can Depth Replace Precision? A Resource Theory of Quantized Neural Computation

When can additional low-bit residual computation replace missing numerical precision for a fixed input-output map? We model a quantized residual system over a fixed horizon as a pure schedule selecting fields from a declared low-bit operation library, and use relaxed controls to characterize its infinite-depth limit. The distance from the target to the closed relaxed reachable set is the exact structural floor: no increase in depth can remove it for that library. Pure schedules approach the relaxed class at rate $O(D^{-1})$ under bounded-variation time dependence and $O(D^{-\vartheta}+D^{-1})$ under Holder dependence of exponent $\vartheta$. Execution arithmetic can reverse this conclusion: full-state write-back introduces a $D\rho_z$ penalty and can freeze residual updates, whereas increment error feedback replaces this growth by a bounded carry term and obeys an exact common-lattice conservation law. A fixed-teacher converse makes this rate sharp: for coherent depth-$L$ first-order high-precision comparators, accuracy matching requires $D=\Theta(L)$. Learned codebooks add a metadata resource, while state-dependent routing introduces hybrid event conditions. Verified primal and dual bounds yield feasible, impossible, or unresolved decisions before training. Companion software implements the workflow, and Lean 4 machine-checks the exact discrete core. Depth replaces precision only relative to a declared library, horizon, execution semantics, and routing model.

cs.LG↗

arXiv2026

Random Parameter Noise Does Not Make Exact ReLU Verification Easy

We study exact verification of ReLU networks in an adversarial smoothed model. Every network weight and bias is independently perturbed by Gaussian noise, clipped to $[-2,2]$, and rounded to the exact dyadic grid determined by the input bit complexity. We show that, under the standard assumption $\mathrm{NP}\not\subseteq\mathrm{BPP}$, there is no sound and complete verifier whose expected running time is polynomial in network size, bit complexity, and inverse noise level for every base instance. The conclusion already holds at the fixed noise level $\sigma_\star=2^{-11}$ for one-hidden-layer networks over a unit box, with hidden fan-in at most three and base coefficients in $[-1,1]$. The proof combines an exact gap embedding with a quantitative robustness argument. For every E3SAT formula $\Phi$ with $m$ clauses, a four-ReLU-per-clause construction satisfies $\max_{x\in[0,1]^n} g_\Phi(x)=(m-\operatorname{unsat}(\Phi))/3$, and coordinatewise threshold rounding never decreases the objective. A weighted parameter-sensitivity inequality and Gaussian concentration then show that a verification gap linear in $m$ survives the aggregate perturbation of all coefficients with probability at least $1-e^{-m/8}$. The proof includes clipping, exact dyadic rounding, output-layer perturbations, polynomial-bit sampling of the rounded Gaussian law, and the conversion from expected smoothed running time to a BPP algorithm. Computational checks test the exact identity and illustrate the different scaling of extensive and constant gaps; they are diagnostics rather than evidence for the complexity theorem. The result concerns worst-case base networks in the stated absolute-noise model, but it shows that parameter nondegeneracy alone does not yield a universal smoothed-polynomial guarantee for exact verification.

cs.CC↗

arXiv2026

Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring

Understanding whether deep neural networks are effectively optimized remains challenging, as training occurs in highly nonconvex landscapes and standard metrics provide limited visibility into layer-wise learning quality. This challenge is particularly acute for transformer-based language models, where training is expensive, models are often reused in frozen form, and poorly optimized layers can silently degrade performance. We propose a layer-wise peeling framework for monitoring training dynamics, in which each transformer layer is locally optimized against intermediate representations of the trained model. By constructing lightweight, layer-specific reference solutions and projecting layers onto multiple intermediate outputs via different permutations, we obtain achievable baselines that enable fine-grained diagnosis of under-optimized layers. Experiments on decoder-only transformer models show that these layer-wise reference bounds can match or even surpass the trained model at various stages of training, exposing inefficiencies that remain hidden in aggregate loss curves. We further demonstrate that this analysis remains effective under binarization and quantized settings, where training dynamics are particularly fragile. Across all numerical results, the proposed bounds consistently separate apparent convergence from effective optimality, highlighting optimization opportunities that are invisible when relying on training loss alone.

cs.LG↗

arXiv2026

Linearization Explains Fine-Tuning in Large Language Models

Parameter-Efficient Fine-Tuning (PEFT) is a popular class of techniques that strive to adapt large models in a scalable and resource-efficient manner. Yet, the mechanisms underlying their training performance and generalization remain underexplored. In this paper, we provide several insights into such fine-tuning through the lens of linearization. Fine-tuned models are often implicitly encouraged to remain close to the pretrained model. By making this explicit, using an Euclidean distance inductive bias in parameter space, we show that fine-tuning dynamics become equivalent to learning with the positive-definite neural tangent kernel (NTK). We specifically analyze how close the fully linear and the linearized fine-tuning optimizations are, based on the strength of the regularization. This allows us to be pragmatic about how good a model linearization is when fine-tuning large language models (LLMs). When linearization is a good model, our findings reveal a strong correlation between the eigenvalue spectrum of the NTK and the performance of model adaptation. Motivated by this, we give spectral perturbation bounds on the NTK induced by the choice of layers selected for fine-tuning. We empirically validate our theory on Low Rank Adaptation (LoRA) on LLMs. These insights not only characterize fine-tuning but also have the potential to enhance PEFT techniques, paving the way to better informed and more nimble adaptation in LLMs.

cs.LG↗

arXiv2025

From Quasi-Isometric Embeddings to Finite-Volume Property: A Theoretical Framework for Quantized Matrix Completion

We delve into the impact of memoryless scalar quantization on matrix completion. Our primary motivation for this research is to evaluate the recovery performance of nuclear norm minimization in handling quantized matrix problems without the use of any regularization terms such as those stemming from maximum likelihood estimation. We broaden our theoretical discussion to encompass the coarse quantization scenario with a dithering scheme, where the only available information for low-rank matrix recovery is a few-bit low-resolution data. We furnish theoretical guarantees for both scenarios: when access to dithers is available during the reconstruction process, and when we have access solely to the statistical properties of the dithers. Additionally, we conduct a comprehensive analysis of the effects of sign flips and prequantization noise on the recovery performance, particularly when the impact of sign flips is quantified using the well-known Hamming distance in the upper bound of recovery error.

cs.IT↗

arXiv2025

Fast and Robust LRSD-based SAR/ISAR Imaging and Decomposition

The earlier works in the context of low-rank-sparse-decomposition (LRSD)-driven stationary synthetic aperture radar (SAR) imaging have shown significant improvement in the reconstruction-decomposition process. Neither of the proposed frameworks, however, can achieve satisfactory performance when facing a platform residual phase error (PRPE) arising from the instability of airborne platforms. More importantly, in spite of the significance of real-time processing requirements in remote sensing applications, these prior works have only focused on enhancing the quality of the formed image, not reducing the computational burden. To address these two concerns, this article presents a fast and unified joint SAR imaging framework where the dominant sparse objects and low-rank features of the image background are decomposed and enhanced through a robust LRSD. In particular, our unified algorithm circumvents the tedious task of computing the inverse of large matrices for image formation and takes advantage of the recent advances in constrained quadratic programming to handle the unimodular constraint imposed due to the PRPE. Furthermore, we extend our approach to ISAR autofocusing and imaging. Specifically, due to the intrinsic sparsity of ISAR images, the LRSD framework is essentially tasked with the recovery of a sparse image. Several experiments based on synthetic and real data are presented to validate the superiority of the proposed method in terms of imaging quality and computational cost compared to the state-of-the-art methods.

eess.IV↗

arXiv2025

Physics-Inspired Binary Neural Networks: Interpretable Compression with Theoretical Guarantees

Why rely on dense neural networks and then blindly sparsify them when prior knowledge about the problem structure is already available? Many inverse problems admit algorithm-unrolled networks that naturally encode physics and sparsity. In this work, we propose a Physics-Inspired Binary Neural Network (PIBiNN) that combines two key components: (i) data-driven one-bit quantization with a single global scale, and (ii) problem-driven sparsity predefined by physics and requiring no updates during training. This design yields compression rates below one bit per weight by exploiting structural zeros, while preserving essential operator geometry. Unlike ternary or pruning-based schemes, our approach avoids ad-hoc sparsification, reduces metadata overhead, and aligns directly with the underlying task. Experiments suggest that PIBiNN achieves advantages in both memory efficiency and generalization compared to competitive baselines such as ternary and channel-wise quantization.

cs.LG↗

arXiv2025

Sample Abundance for Signal Processing: A Brief Introduction

This paper reports, by way of introduction, on the advances made by our group and the broader signal processing community on the concept of sample abundance; a phenomenon that naturally arises in one-bit and few-bit signal processing frameworks. By leveraging large volumes of low-precision measurements, we show how traditionally costly constraints, such as matrix semi-definiteness and rank conditions, become redundant, yielding simple overdetermined linear feasibility problems. We illustrate key algorithms, theoretical guarantees via the Finite Volume Property, and the sample abundance singularity phenomenon, where computational complexity sharply drops.

cs.IT↗

arXiv2025

RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates

We propose RoCoFT, a parameter-efficient fine-tuning method for large-scale language models (LMs) based on updating only a few rows and columns of the weight matrices in transformers. Through extensive experiments with medium-size LMs like BERT and RoBERTa, and larger LMs like Bloom-7B, Llama2-7B, and Llama2-13B, we show that our method gives comparable or better accuracies than state-of-art PEFT methods while also being more memory and computation-efficient. We also study the reason behind the effectiveness of our method with tools from neural tangent kernel theory. We empirically demonstrate that our kernel, constructed using a restricted set of row and column parameters, are numerically close to the full-parameter kernel and gives comparable classification performance. Ablation studies are conducted to investigate the impact of different algorithmic choices, including the selection strategy for rows and columns as well as the optimal rank for effective implementation of our method.

cs.CL↗

arXiv2025

Predicting Through Generation: Why Generation Is Better for Prediction

This paper argues that generating output tokens is more effective than using pooled representations for prediction tasks because token-level generation retains more mutual information. Since LLMs are trained on massive text corpora using next-token prediction, generation aligns naturally with their learned behavior. Using the Data Processing Inequality (DPI), we provide both theoretical and empirical evidence supporting this claim. However, autoregressive models face two key challenges when used for prediction: (1) exposure bias, where the model sees ground truth tokens during training but relies on its own predictions during inference, leading to errors, and (2) format mismatch, where discrete tokens do not always align with the tasks required output structure. To address these challenges, we introduce PredGen(Predicting Through Generating), an end to end framework that (i) uses scheduled sampling to reduce exposure bias, and (ii) introduces a task adapter to convert the generated tokens into structured outputs. Additionally, we introduce Writer-Director Alignment Loss (WDAL), which ensures consistency between token generation and final task predictions, improving both text coherence and numerical accuracy. We evaluate PredGen on multiple classification and regression benchmarks. Our results show that PredGen consistently outperforms standard baselines, demonstrating its effectiveness in structured prediction tasks.

cs.CL↗

arXiv2025

Quantum Compressive Sensing Meets Quantum Noise: A Practical Exploration

Compressive sensing is a signal processing technique that enables the reconstruction of sparse signals from a limited number of measurements, leveraging the signal's inherent sparsity to facilitate efficient recovery. Recent works on the Quantum Compressive Sensing (QCS) architecture, a quantum data-driven approach to compressive sensing where the state of the tensor network is represented by a quantum state over a set of entangled qubits, have shown promise in advancing quantum data-driven methods for compressive sensing. However, the QCS framework has remained largely untested on quantum computing resources or in the presence of quantum noise. In this work, we present a practical implementation of QCS on Amazon Braket, utilizing the Quantum Imaginary Time Evolution (QITE) projection technique to assess the framework's capabilities under quantum noise. We outline the necessary modifications to the QCS framework for deployment on Amazon Braket, followed by results under four types of quantum noise. Finally, we discuss potential long-term directions aimed at unlocking the full potential of quantum compressive sensing for applications such as signal recovery and image processing.

quant-ph↗

arXiv2024

Data-Aware Training Quality Monitoring and Certification for Reliable Deep Learning

Deep learning models excel at capturing complex representations through sequential layers of linear and non-linear transformations, yet their inherent black-box nature and multi-modal training landscape raise critical concerns about reliability, robustness, and safety, particularly in high-stakes applications. To address these challenges, we introduce YES training bounds, a novel framework for real-time, data-aware certification and monitoring of neural network training. The YES bounds evaluate the efficiency of data utilization and optimization dynamics, providing an effective tool for assessing progress and detecting suboptimal behavior during training. Our experiments show that the YES bounds offer insights beyond conventional local optimization perspectives, such as identifying when training losses plateau in suboptimal regions. Validated on both synthetic and real data, including image denoising tasks, the bounds prove effective in certifying training quality and guiding adjustments to enhance model performance. By integrating these bounds into a color-coded cloud-based monitoring system, we offer a powerful tool for real-time evaluation, setting a new standard for training quality assurance in deep learning.

cs.LG↗

arXiv2024

Deep Learning-Enabled One-Bit DoA Estimation

Unrolled deep neural networks have attracted significant attention for their success in various practical applications. In this paper, we explore an application of deep unrolling in the direction of arrival (DoA) estimation problem when coarse quantization is applied to the measurements. We present a compressed sensing formulation for DoA estimation from one-bit data in which estimating target DoAs requires recovering a sparse signal from a limited number of severely quantized linear measurements. In particular, we exploit covariance recovery from one-bit dither samples. To recover the covariance of transmitted signal, the learned iterative shrinkage and thresholding algorithm (LISTA) is employed fed by one-bit data. We demonstrate that the upper bound of estimation performance is governed by the recovery error of the transmitted signal covariance matrix. Through numerical experiments, we demonstrate the proposed LISTA-based algorithm's capability in estimating target locations. The code employed in this study is available online.

eess.SP↗

arXiv2024

Antenna Failure Resilience: Deep Learning-Enabled Robust DOA Estimation with Single Snapshot Sparse Arrays

Recent advancements in Deep Learning (DL) for Direction of Arrival (DOA) estimation have highlighted its superiority over traditional methods, offering faster inference, enhanced super-resolution, and robust performance in low Signal-to-Noise Ratio (SNR) environments. Despite these advancements, existing research predominantly focuses on multi-snapshot scenarios, a limitation in the context of automotive radar systems which demand high angular resolution and often rely on limited snapshots, sometimes as scarce as a single snapshot. Furthermore, the increasing interest in sparse arrays for automotive radar, owing to their cost-effectiveness and reduced antenna element coupling, presents additional challenges including susceptibility to random sensor failures. This paper introduces a pioneering DL framework featuring a sparse signal augmentation layer, meticulously crafted to bolster single snapshot DOA estimation across diverse sparse array setups and amidst antenna failures. To our best knowledge, this is the first work to tackle this issue. Our approach improves the adaptability of deep learning techniques to overcome the unique difficulties posed by sparse arrays with single snapshot. We conduct thorough evaluations of our network's performance using simulated and real-world data, showcasing the efficacy and real-world viability of our proposed solution. The code and real-world dataset employed in this study are available at https://github.com/ruxinzh/Deep_RSA_DOA.

eess.SP↗

arXiv2024

Collaborative Automotive Radar Sensing via Mixed-Precision Distributed Array Completion

This paper investigates the effects of coarse quantization with mixed precision on measurements obtained from sparse linear arrays, synthesized by a collaborative automotive radar sensing strategy. The mixed quantization precision significantly reduces the data amount that needs to be shared from radar nodes to the fusion center for coherent processing. We utilize the low-rank properties inherent in the constructed Hankel matrix of the mixed-precision array, to recover azimuth angles from quantized measurements. Our proposed approach addresses the challenge of mixed-quantized Hankel matrix completion, allowing for accurate estimation of the azimuth angles of interest. To evaluate the recovery performance of the proposed scheme, we establish a quasi-isometric embedding with a high probability for mixed-precision quantization. The effectiveness of our proposed scheme is demonstrated through numerical results, highlighting successful reconstruction.

eess.SP↗

arXiv2024

Automotive Radar Sensing with Sparse Linear Arrays Using One-Bit Hankel Matrix Completion

The design of sparse linear arrays has proven instrumental in the implementation of cost-effective and efficient automotive radar systems for high-resolution imaging. This paper investigates the impact of coarse quantization on measurements obtained from such arrays. To recover azimuth angles from quantized measurements, we leverage the low-rank properties of the constructed Hankel matrix. In particular, by addressing the one-bit Hankel matrix completion problem through a developed singular value thresholding algorithm, our proposed approach accurately estimates the azimuth angles of interest. We provide comprehensive insights into recovery performance and the required number of one-bit samples. The effectiveness of our proposed scheme is underscored by numerical results, demonstrating successful reconstruction using only one-bit data.

eess.SP↗