arXiv subjects

Makan Fardad

Publications and source records attributed to Makan Fardad.

At least 19 recordsLinked to original sources

arXiv2026

Maximum Likelihood Particle Tracking in Turbulent Flows via Sparse Optimization

Lagrangian particle tracking is essential for characterizing turbulent flows, but inferring particle acceleration from inherently noisy position data remains a significant challenge. Fluid particles in turbulence experience extreme, intermittent accelerations, resulting in heavy-tailed probability density functions (PDFs) that deviate strongly from Gaussian predictions. Existing filtering techniques, such as Gaussian kernels and penalized B-splines, implicitly assume Gaussian-distributed jerk, thereby penalizing sparse, high-magnitude acceleration changes and artificially suppressing the intermittent tails. In this work, we develop a novel maximum likelihood estimation (MLE) framework that explicitly accounts for this non-Gaussian intermittency. By formulating a modified Gaussian process to model the random incremental forcing, we introduce a sparse optimization scheme utilizing a convex 1-norm relaxation. To overcome the numerical stiffness associated with high-order difference operators, the problem is efficiently solved using an iteratively reweighted least squares (IRLS) algorithm. The proposed filter is evaluated against direct numerical simulation (DNS) data of homogeneous, isotropic turbulence (Re approx. 310). Results demonstrate that the IRLS approach consistently outperforms state-of-the-art discrete MLE, continuous MLE, and B-spline methods, yielding systematic reductions in root-mean-squared error (RMSE) across position, velocity, and acceleration. Most importantly, the proposed framework succeeds in better recovering the heavy-tailed statistical structure of both acceleration and acceleration differences (jerk) across temporal scales, preserving the physical intermittency characteristic of high-Reynoldsnumber turbulent flows that baseline methods severely attenuate.

physics.data-an↗

arXiv2025

Adaptive Clutter Suppression via Convex Optimization

Passive and bistatic radar systems are often limited by strong clutter and direct-path interference that mask weak moving targets. Conventional cancellation methods such as the extensive cancellation algorithm require careful tuning and can distort the delay-Doppler response. This paper introduces a convex optimization framework that adaptively synthesizes per-cell delay-Doppler filters to suppress clutter while preserving the canonical cross-ambiguity function (CAF). The approach formulates a quadratic program that minimizes distortion of the CAF surface subject to linear clutter-suppression constraints, eliminating the need for a separate cancellation stage. Monte Carlo simulations using common communication waveforms demonstrate strong clutter suppression, accurate CFAR calibration, and major detection-rate gains over the classical CAF. The results highlight a scalable, CAF-faithful method for adaptive clutter mitigation in passive radar.

math.OC↗

arXiv2023

Optimization Based Data Enrichment Using Stochastic Dynamical System Models

We develop a general framework for state estimation in systems modeled with noise-polluted continuous time dynamics and discrete time noisy measurements. Our approach is based on maximum likelihood estimation and employs the calculus of variations to derive optimality conditions for continuous time functions. We make no prior assumptions on the form of the mapping from measurements to state-estimate or on the distributions of the noise terms, making the framework more general than Kalman filtering/smoothing where this mapping is assumed to be linear and the noises Gaussian. The optimal solution that arises is interpreted as a continuous time spline, the structure and temporal dependency of which is determined by the system dynamics and the distributions of the process and measurement noise. Similar to Kalman smoothing, the optimal spline yields increased data accuracy at instants when measurements are taken, in addition to providing continuous time estimates outside the measurement instances. We demonstrate the utility and generality of our approach via illustrative examples that render both linear and nonlinear data filters depending on the particular system.

math.OC↗

arXiv2023

Loss Attitude Aware Energy Management for Signal Detection

This work considers a Bayesian signal processing problem where increasing the power of the probing signal may cause risks or undesired consequences. We employ a market based approach to solve energy management problems for signal detection while balancing multiple objectives. In particular, the optimal amount of resource consumption is determined so as to maximize a profit-loss based expected utility function. Next, we study the human behavior of resource consumption while taking individuals' behavioral disparity into account. Unlike rational decision makers who consume the amount of resource to maximize the expected utility function, human decision makers act to maximize their subjective utilities. We employ prospect theory to model humans' loss aversion towards a risky event. The amount of resource consumption that maximizes the humans' subjective utility is derived to characterize the actual behavior of humans. It is shown that loss attitudes may lead the human to behave quite differently from a rational decision maker.

eess.SP↗

arXiv2021

Compact Multi-level Sparse Neural Networks with Input Independent Dynamic Rerouting

Deep neural networks (DNNs) have shown to provide superb performance in many real life applications, but their large computation cost and storage requirement have prevented them from being deployed to many edge and internet-of-things (IoT) devices. Sparse deep neural networks, whose majority weight parameters are zeros, can substantially reduce the computation complexity and memory consumption of the models. In real-use scenarios, devices may suffer from large fluctuations of the available computation and memory resources under different environment, and the quality of service (QoS) is difficult to maintain due to the long tail inferences with large latency. Facing the real-life challenges, we propose to train a sparse model that supports multiple sparse levels. That is, a hierarchical structure of weights are satisfied such that the locations and the values of the non-zero parameters of the more-sparse sub-model area subset of the less-sparse sub-model. In this way, one can dynamically select the appropriate sparsity level during inference, while the storage cost is capped by the least sparse sub-model. We have verified our methodologies on a variety of DNN models and tasks, including the ResNet-50, PointNet++, GNMT, and graph attention networks. We obtain sparse sub-models with an average of 13.38% weights and 14.97% FLOPs, while the accuracies are as good as their dense counterparts. More-sparse sub-models with 5.38% weights and 4.47% of FLOPs, which are subsets of the less-sparse ones, can be obtained with only 3.25% relative accuracy loss.

cs.NE↗

arXiv2021

Adversarial Attack Generation Empowered by Min-Max Optimization

The worst-case training principle that minimizes the maximal adversarial loss, also known as adversarial training (AT), has shown to be a state-of-the-art approach for enhancing adversarial robustness. Nevertheless, min-max optimization beyond the purpose of AT has not been rigorously explored in the adversarial context. In this paper, we show how a general framework of min-max optimization over multiple domains can be leveraged to advance the design of different types of adversarial attacks. In particular, given a set of risk sources, minimizing the worst-case attack loss can be reformulated as a min-max problem by introducing domain weights that are maximized over the probability simplex of the domain set. We showcase this unified framework in three attack generation problems -- attacking model ensembles, devising universal perturbation under multiple inputs, and crafting attacks resilient to data transformations. Extensive experiments demonstrate that our approach leads to substantial attack improvement over the existing heuristic strategies as well as robustness improvement over state-of-the-art defense methods trained to be robust against multiple perturbation types. Furthermore, we find that the self-adjusted domain weights learned from our min-max framework can provide a holistic tool to explain the difficulty level of attack across domains. Code is available at https://github.com/wangjksjtu/minmax-adv.

cs.LG↗

arXiv2020

A Unified DNN Weight Compression Framework Using Reweighted Optimization Methods

To address the large model size and intensive computation requirement of deep neural networks (DNNs), weight pruning techniques have been proposed and generally fall into two categories, i.e., static regularization-based pruning and dynamic regularization-based pruning. However, the former method currently suffers either complex workloads or accuracy degradation, while the latter one takes a long time to tune the parameters to achieve the desired pruning rate without accuracy loss. In this paper, we propose a unified DNN weight pruning framework with dynamically updated regularization terms bounded by the designated constraint, which can generate both non-structured sparsity and different kinds of structured sparsity. We also extend our method to an integrated framework for the combination of different DNN compression tasks.

cs.LG↗

arXiv2019

Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMM

Weight pruning and weight quantization are two important categories of DNN model compression. Prior work on these techniques are mainly based on heuristics. A recent work developed a systematic frame-work of DNN weight pruning using the advanced optimization technique ADMM (Alternating Direction Methods of Multipliers), achieving one of state-of-art in weight pruning results. In this work, we first extend such one-shot ADMM-based framework to guarantee solution feasibility and provide fast convergence rate, and generalize to weight quantization as well. We have further developed a multi-step, progressive DNN weight pruning and quantization framework, with dual benefits of (i) achieving further weight pruning/quantization thanks to the special property of ADMM regularization, and (ii) reducing the search space within each step. Extensive experimental results demonstrate the superior performance compared with prior work. Some highlights: (i) we achieve 246x,36x, and 8x weight pruning on LeNet-5, AlexNet, and ResNet-50 models, respectively, with (almost) zero accuracy loss; (ii) even a significant 61x weight pruning in AlexNet (ImageNet) results in only minor degradation in actual accuracy compared with prior work; (iii) we are among the first to derive notable weight pruning results for ResNet and MobileNet models; (iv) we derive the first lossless, fully binarized (for all layers) LeNet-5 for MNIST and VGG-16 for CIFAR-10; and (v) we derive the first fully binarized (for all layers) ResNet for ImageNet with reasonable accuracy loss.

cs.NE↗

arXiv2019

StructADMM: A Systematic, High-Efficiency Framework of Structured Weight Pruning for DNNs

Weight pruning methods of DNNs have been demonstrated to achieve a good model pruning rate without loss of accuracy, thereby alleviating the significant computation/storage requirements of large-scale DNNs. Structured weight pruning methods have been proposed to overcome the limitation of irregular network structure and demonstrated actual GPU acceleration. However, in prior work the pruning rate (degree of sparsity) and GPU acceleration are limited (to less than 50%) when accuracy needs to be maintained. In this work,we overcome these limitations by proposing a unified, systematic framework of structured weight pruning for DNNs. It is a framework that can be used to induce different types of structured sparsity, such as filter-wise, channel-wise, and shape-wise sparsity, as well non-structured sparsity. The proposed framework incorporates stochastic gradient descent with ADMM, and can be understood as a dynamic regularization method in which the regularization target is analytically updated in each iteration. Without loss of accuracy on the AlexNet model, we achieve 2.58X and 3.65X average measured speedup on two GPUs, clearly outperforming the prior work. The average speedups reach 3.15X and 8.52X when allowing a moderate ac-curacy loss of 2%. In this case the model compression for convolutional layers is 15.0X, corresponding to 11.93X measured CPU speedup. Our experiments on ResNet model and on other data sets like UCF101 and CIFAR-10 demonstrate the consistently higher performance of our framework.

cs.NE↗

arXiv2018

Progressive Weight Pruning of Deep Neural Networks using ADMM

Deep neural networks (DNNs) although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices. Extensive research work have been conducted on DNN model compression or pruning. However, most of the previous work took heuristic approaches. This work proposes a progressive weight pruning approach based on ADMM (Alternating Direction Method of Multipliers), a powerful technique to deal with non-convex optimization problems with potentially combinatorial constraints. Motivated by dynamic programming, the proposed method reaches extremely high pruning rate by using partial prunings with moderate pruning rates. Therefore, it resolves the accuracy degradation and long convergence time problems when pursuing extremely high pruning ratios. It achieves up to 34 times pruning rate for ImageNet dataset and 167 times pruning rate for MNIST dataset, significantly higher than those reached by the literature work. Under the same number of epochs, the proposed method also achieves faster convergence and higher compression rates. The codes and pruned DNN models are released in the link bit.ly/2zxdlss

cs.LG↗

arXiv2018

A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers

Weight pruning methods for deep neural networks (DNNs) have been investigated recently, but prior work in this area is mainly heuristic, iterative pruning, thereby lacking guarantees on the weight reduction ratio and convergence time. To mitigate these limitations, we present a systematic weight pruning framework of DNNs using the alternating direction method of multipliers (ADMM). We first formulate the weight pruning problem of DNNs as a nonconvex optimization problem with combinatorial constraints specifying the sparsity requirements, and then adopt the ADMM framework for systematic weight pruning. By using ADMM, the original nonconvex optimization problem is decomposed into two subproblems that are solved iteratively. One of these subproblems can be solved using stochastic gradient descent, the other can be solved analytically. Besides, our method achieves a fast convergence rate. The weight pruning results are very promising and consistently outperform the prior work. On the LeNet-5 model for the MNIST data set, we achieve 71.2 times weight reduction without accuracy loss. On the AlexNet model for the ImageNet data set, we achieve 21 times weight reduction without accuracy loss. When we focus on the convolutional layer pruning for computation reductions, we can reduce the total computation by five times compared with the prior work (achieving a total of 13.4 times weight reduction in convolutional layers). Our models and codes are released at https://github.com/KaiqiZhang/admm-pruning

cs.NE↗

arXiv2018

Systematic Weight Pruning of DNNs using Alternating Direction Method of Multipliers

We present a systematic weight pruning framework of deep neural networks (DNNs) using the alternating direction method of multipliers (ADMM). We first formulate the weight pruning problem of DNNs as a constrained nonconvex optimization problem, and then adopt the ADMM framework for systematic weight pruning. We show that ADMM is highly suitable for weight pruning due to the computational efficiency it offers. We achieve a much higher compression ratio compared with prior work while maintaining the same test accuracy, together with a faster convergence rate. Our models are released at https://github.com/KaiqiZhang/admm-pruning

cs.LG↗

arXiv2017

A Memristor-Based Optimization Framework for AI Applications

Memristors have recently received significant attention as ubiquitous device-level components for building a novel generation of computing systems. These devices have many promising features, such as non-volatility, low power consumption, high density, and excellent scalability. The ability to control and modify biasing voltages at the two terminals of memristors make them promising candidates to perform matrix-vector multiplications and solve systems of linear equations. In this article, we discuss how networks of memristors arranged in crossbar arrays can be used for efficiently solving optimization and machine learning problems. We introduce a new memristor-based optimization framework that combines the computational merit of memristor crossbars with the advantages of an operator splitting method, alternating direction method of multipliers (ADMM). Here, ADMM helps in splitting a complex optimization problem into subproblems that involve the solution of systems of linear equations. The capability of this framework is shown by applying it to linear programming, quadratic programming, and sparse optimization. In addition to ADMM, implementation of a customized power iteration (PI) method for eigenvalue/eigenvector computation using memristor crossbars is discussed. The memristor-based PI method can further be applied to principal component analysis (PCA). The use of memristor crossbars yields a significant speed-up in computation, and thus, we believe, has the potential to advance optimization and machine learning research in artificial intelligence (AI).

cs.ET↗

arXiv2016

Optimized Sensor Collaboration for Estimation of Temporally Correlated Parameters

In this paper, we aim to design the optimal sensor collaboration strategy for the estimation of time-varying parameters, where collaboration refers to the act of sharing measurements with neighboring sensors prior to transmission to a fusion center. We begin by addressing the sensor collaboration problem for the estimation of uncorrelated parameters. We show that the resulting collaboration problem can be transformed into a special nonconvex optimization problem, where a difference of convex functions carries all the nonconvexity. This specific problem structure enables the use of a convex-concave procedure to obtain a near-optimal solution. When the parameters of interest are temporally correlated, a penalized version of the convex-concave procedure becomes well suited for designing the optimal collaboration scheme. In order to improve computational efficiency, we further propose a fast algorithm that scales gracefully with problem size via the alternating direction method of multipliers. Numerical results are provided to demonstrate the effectiveness of our approach and the impact of parameter correlation and temporal dynamics of sensor networks on estimation performance.

cs.IT↗

arXiv2016

Sensor Selection for Estimation with Correlated Measurement Noise

In this paper, we consider the problem of sensor selection for parameter estimation with correlated measurement noise. We seek optimal sensor activations by formulating an optimization problem, in which the estimation error, given by the trace of the inverse of the Bayesian Fisher information matrix, is minimized subject to energy constraints. Fisher information has been widely used as an effective sensor selection criterion. However, existing information-based sensor selection methods are limited to the case of uncorrelated noise or weakly correlated noise due to the use of approximate metrics. By contrast, here we derive the closed form of the Fisher information matrix with respect to sensor selection variables that is valid for any arbitrary noise correlation regime, and develop both a convex relaxation approach and a greedy algorithm to find near-optimal solutions. We further extend our framework of sensor selection to solve the problem of sensor scheduling, where a greedy algorithm is proposed to determine non-myopic (multi-time step ahead) sensor schedules. Lastly, numerical results are provided to illustrate the effectiveness of our approach, and to reveal the effect of noise correlation on estimation performance.

stat.AP↗

arXiv2015

Sparsity-Aware Sensor Collaboration for Linear Coherent Estimation

In the context of distributed estimation, we consider the problem of sensor collaboration, which refers to the act of sharing measurements with neighboring sensors prior to transmission to a fusion center. While incorporating the cost of sensor collaboration, we aim to find optimal sparse collaboration schemes subject to a certain information or energy constraint. Two types of sensor collaboration problems are studied: minimum energy with an information constraint; and maximum information with an energy constraint. To solve the resulting sensor collaboration problems, we present tractable optimization formulations and propose efficient methods which render near-optimal solutions in numerical experiments. We also explore the situation in which there is a cost associated with the involvement of each sensor in the estimation scheme. In such situations, the participating sensors must be chosen judiciously. We introduce a unified framework to jointly design the optimal sensor selection and collaboration schemes. For a given estimation performance, we show empirically that there exists a trade-off between sensor selection and sensor collaboration.

stat.ME↗

arXiv2014

Optimal Periodic Sensor Scheduling in Networks of Dynamical Systems

We consider the problem of finding optimal time-periodic sensor schedules for estimating the state of discrete-time dynamical systems. We assume that {multiple} sensors have been deployed and that the sensors are subject to resource constraints, which limits the number of times each can be activated over one period of the periodic schedule. We seek an algorithm that strikes a balance between estimation accuracy and total sensor activations over one period. We make a correspondence between active sensors and the nonzero columns of estimator gain. We formulate an optimization problem in which we minimize the trace of the error covariance with respect to the estimator gain while simultaneously penalizing the number of nonzero columns of the estimator gain. This optimization problem is combinatorial in nature, and we employ the alternating direction method of multipliers (ADMM) to find its locally optimal solutions. Numerical results and comparisons with other sensor scheduling algorithms in the literature are provided to illustrate the effectiveness of our proposed method.

stat.AP↗

arXiv2013

Design of optimal sparse interconnection graphs for synchronization of oscillator networks

We study the optimal design of a conductance network as a means for synchronizing a given set of oscillators. Synchronization is achieved when all oscillator voltages reach consensus, and performance is quantified by the mean-square deviation from the consensus value. We formulate optimization problems that address the trade-off between synchronization performance and the number and strength of oscillator couplings. We promote the sparsity of the coupling network by penalizing the number of interconnection links. For identical oscillators, we establish convexity of the optimization problem and demonstrate that the design problem can be formulated as a semidefinite program. Finally, for special classes of oscillator networks we derive explicit analytical expressions for the optimal conductance values.

math.OC↗