arXiv subjects

Mohammad Mozaffari

Publications and source records attributed to Mohammad Mozaffari.

At least 19 recordsLinked to original sources

arXiv2026

Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression

Prohibitive computational and environmental costs impede the scalable deployment of Large Language Models (LLMs). Traditional compression techniques (sparsity, quantization, low-rank approximations) are typically applied in isolation, and each hits an accuracy-efficiency wall. This thesis proposes the "Compression Trinity," a unified framework that applies the three pillars jointly: sparsity to reduce computation, quantization to minimize memory bandwidth, and low-rank approximations to recover accuracy. To accelerate pretraining, we apply the Trinity to the optimizer and model architecture. MKOR approximates curvature via block-diagonal sparsity and low-rank inversion, maintaining numerical stability for quantized states; it reduces curvature update complexity from $O(d^3)$ to $O(d^2)$ and accelerates convergence by up to 1.85x over KFAC. SLoPe accelerates training by up to 1.25x via a double-pruned backward pass for N:M sparsity, using low-rank "lazy" adapters in the final 1% of training to recover accuracy. For post-training compression, OPTIMA stabilizes static masks in a zero-training regime by formulating weight reconstruction as globally optimal column-wise quadratic programs, improving zero-shot accuracy by up to 3.97%. Given a fine-tuning budget, PATCH breaks the ceiling of static masks by learning a dynamic hybrid sparsity ratio between 0% and 50%, yielding up to 1.38x speedups. Finally, SLiM realizes the full Compression Trinity in one shot, using mathematically derived low-rank adapters to recover information lost to quantization and sparsity, improving accuracy by up to 5.66% over state-of-the-art methods and outperforming uncompressed dense models at equal parameter budgets by 0.6%. Together, these results show that jointly applying the Compression Trinity is essential for efficient, scalable, high-performance LLMs.

cs.AI↗

arXiv2026

LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models

Unstructured sparsity is now natively accelerated by recent GPU kernels and dataflow hardware, shifting the bottleneck from inference execution to the pruning algorithm. State-of-the-art methods for unstructured LLM pruning are layer-wise surrogates derived from the Optimal Brain Surgeon principle, and they sacrifice end-to-end accuracy, especially under aggressive sparsity. End-to-end alternatives such as MaskLLM and PATCH show that learnable masks can close this gap, but their categorical-over-patterns parameterization scales with the number of valid masks per row and does not port to the unstructured setting. We introduce LEAP, which replaces this intractable parameterization with a per-weight Bernoulli-via-Gumbel-sigmoid relaxation that makes end-to-end unstructured mask learning tractable. Across five LLM families from 0.5B to 8B parameters at 50% and 60% sparsity, LEAP improves six-task average zero-shot accuracy by +2.59 points on average over ADMM, the best layer-wise baseline in our sweep.

cs.LG↗

arXiv2026

PATCH: Learnable Tile-level Hybrid Sparsity for LLMs

Large language models (LLMs) deliver impressive performance but incur prohibitive memory and compute costs at deployment. Model pruning is an effective way to reduce these overheads, yet existing approaches face challenges: unstructured sparsity, where nonzeros can appear anywhere, preserves accuracy but yields irregular access patterns that prevent GPU acceleration, while semi-structured 2:4 sparsity is hardware-friendly but enforces a rigid 50% pattern that degrades model quality. To bridge this gap, we introduce PATCH, a hybrid sparsity framework that enables a continuous sparsity ratio between 0% and 50%. PATCH partitions weight matrices into tiles, assigning each tile to be either dense or 2:4 sparse via a learnable mask selection mechanism. This design provides fine-grained control over accuracy-acceleration tradeoffs and supports non-uniform sparsity across layers, leading to superior overall quality. Across models from 0.5B to 13B parameters, PATCH consistently narrows the gap to dense accuracy while delivering practical speedups. For instance, on LLaMA-2 7B with an A6000 GPU, PATCH achieves 1.18x-1.38x end-to-end speedup over dense baselines while improving accuracy by 0.37%-2.96% compared to the state-of-the-art 2:4 pruning method, MaskLLM.

cs.LG↗

arXiv2025

OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction

Post-training model pruning is a promising solution, yet it faces a trade-off: simple heuristics that zero weights are fast but degrade accuracy, while principled joint optimization methods recover accuracy but are computationally infeasible at modern scale. One-shot methods such as SparseGPT offer a practical trade-off in optimality by applying efficient, approximate heuristic weight updates. To close this gap, we introduce OPTIMA, a practical one-shot post-training pruning method that balances accuracy and scalability. OPTIMA casts layer-wise weight reconstruction after mask selection as independent, row-wise Quadratic Programs (QPs) that share a common layer Hessian. Solving these QPs yields the per-row globally optimal update with respect to the reconstruction objective given the estimated Hessian. The shared-Hessian structure makes the problem highly amenable to batching on accelerators. We implement an accelerator-friendly QP solver that accumulates one Hessian per layer and solves many small QPs in parallel, enabling one-shot post-training pruning at scale on a single accelerator without fine-tuning. OPTIMA integrates with existing mask selectors and consistently improves zero-shot performance across multiple LLM families and sparsity regimes, yielding up to 3.97% absolute accuracy improvement. On an NVIDIA H100, OPTIMA prunes a 8B-parameter transformer end-to-end in 40 hours with 60GB peak memory. Together, these results set a new state-of-the-art accuracy-efficiency trade-offs for one-shot post-training pruning.

cs.LG↗

arXiv2025

Resource Allocation for XR with Edge Offloading: A Reinforcement Learning Approach

Future immersive XR applications will require energy-efficient, high data rate, and low-latency wireless communications in uplink and downlink. One of the key considerations for supporting such XR applications is intelligent and adaptive resource allocation with edge offloading. To address these demands, this paper proposes a reinforcement learning-based resource allocation framework that dynamically allocates uplink and downlink slots while making offloading decisions based on the XR headset's capabilities and network conditions. The paper presents a numerical analysis of the tradeoff between frame loss rate (FLR) and energy efficiency, identifying decision regions for partial offloading to optimize performance. Results show that for the used set of system parameters, partial offloading can extend the coverage area by 55% and reduce energy consumption by up to 34%, compared to always or never offloading. The results demonstrate that the headset's local computing capability plays a crucial role in offloading decisions. Higher computing abilities enable more efficient local processing, reduce the need for offloading, and enhance energy savings.

cs.IT↗

arXiv2025

SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression

Conventional model compression techniques for LLMs address high memory consumption and slow inference challenges but typically require computationally expensive retraining to preserve accuracy. In contrast, one-shot compression methods eliminate retraining cost, but struggle to achieve accuracy comparable to dense models. This paper presents SLIM, a new one-shot compression framework that holistically integrates hardware-friendly quantization, sparsity, and low-rank approximation into a unified process. First, we formulate the quantization process using a probabilistic approach (SLIM-Quant) that enables us to apply uniform quantization. Then, we use an existing one-shot pruning method to apply semi-structured sparsity on top of the quantized weights. Finally, to compensate for the introduced aggregated quantization and sparsity error, we use a novel saliency function with unique invertible and additive features that enables us to mathematically compute the value of low-rank adapters. SLIM improves model accuracy by up to 5.66% (LLaMA-2-7B) for 2:4 sparsity with 4-bit weight quantization, outperforming prior methods. Models compressed with SLIM achieve up to 4.3x and 3.8x on Nvidia RTX3060 and A100 GPUs, respectively. Additionally, they achieve up to 0.23x end-to-end memory reduction in comparison to their dense counterparts. We also propose an optional PEFT recipe that further improves accuracy by up to 1.66% (LLaMA-2-13B) compared to SLIM without fine-tuning.

cs.LG↗

arXiv2025

SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs

We propose SLoPe, a Double-Pruned Sparse Plus Lazy Low-rank Adapter Pretraining method for LLMs that improves the accuracy of sparse LLMs while accelerating their pretraining and inference and reducing their memory footprint. Sparse pretraining of LLMs reduces the accuracy of the model, to overcome this, prior work uses dense models during fine-tuning. SLoPe improves the accuracy of sparsely pretrained models by adding low-rank adapters in the final 1% iterations of pretraining without adding significant overheads to the model pretraining and inference. In addition, SLoPe uses a double-pruned backward pass formulation that prunes the transposed weight matrix using N:M sparsity structures to enable an accelerated sparse backward pass. SLoPe accelerates the training and inference of models with billions of parameters up to $1.25\times$ and $1.54\times$ respectively (OPT-33B and OPT-66B) while reducing their memory usage by up to $0.63\times$ and $0.61\times$ for training and inference respectively.

cs.LG↗

arXiv2024

Scaling 6G Subscribers with Fewer BS Antennas using Multi-carrier NOMA in Fixed Wireless Access

This paper introduces a novel power allocation and subcarrier optimization algorithm tailored for fixed wireless access (FWA) networks operating under low-rank channel conditions, where the number of subscriber antennas far exceeds those at the base station (BS). As FWA networks grow to support more users, traditional approaches like orthogonal multiple access (OMA) and non-orthogonal multiple access (NOMA) struggle to maintain high data rates and energy efficiency due to the limited degrees of freedom in low-rank scenarios. Our proposed solution addresses this by combining optimal power-subcarrier allocation with an adaptive time-sharing algorithm that dynamically adjusts decoding orders to optimize performance across multiple users. The algorithm leverages a generalized decision feedback equalizer (GDFE) approach to effectively manage inter-symbol interference and crosstalk, leading to superior data rates and energy savings. Simulation results demonstrate that our approach significantly outperforms existing OMA and NOMA baselines, particularly in low-rank conditions, with substantial gains in both data rate and energy efficiency. The findings highlight the potential of this method to meet the growing demand for scalable, high-performance FWA networks.

eess.SP↗

arXiv2024

Towards Green Communication: Soft Decoding Scheme for OOK Signals in Zero-Energy Devices

The booming of Internet-of-Things (IoT) is expected to provide more intelligent and reliable communication services for higher network coverage, massive connectivity, and low-cost solutions for 6G services. However, frequent charging and battery replacement of these massive IoT devices brings a series of challenges. Zero energy devices, which rely on energy-harvesting technologies and can operate without battery replacement or charging, play a pivotal role in facilitating the massive use of IoT devices. In order to enable reliable communications of such low-power devices, Manchester-coded on-off keying (OOK) modulation and non-coherent detections are attractive techniques due to their energy efficiency, robustness in noisy environments, and simplicity in receiver design. Moreover, to extend their communication range, employing channel coding along with enhanced detection schemes is crucial. In this paper, a novel soft-decision decoder is designed for OOK-based low-power receivers to enhance their detection performance. In addition, exact closed-form expressions and two simplified approximations are derived for the log-likelihood ratio (LLR), an essential metric for soft decoding. Numerical results demonstrate the significant coverage gain achieved through soft decoding for convolutional code.

cs.IT↗

arXiv2024

Pose-aware 3D Beamwidth Adaptation for Mobile Extended Reality

This paper presents a sensor-aided pose-aware beamwidth adaptation design for a conceptual extended reality (XR) Head-Mounted Display (HMD) equipped with a 2D planar array. The beam is tracked and adapted on the user side by leveraging HMD orientation estimates. The beamwidth adaptation scheme is effected by selective deactivation of elements in the 2D antenna array, employing the angular estimation covariance matrix to overlap the beam with the estimation confidence interval. The proposed method utilizes the estimation correlations to adapt the beamwidth along the confidence interval of these estimates. Compared to a beamwidth adaptation without leveraging estimation correlations, the proposed method demonstrates the gain of leveraging estimation correlations by improving the coverage area for a given outage probability threshold by approximately 16%, or equivalently increasing the power efficiency up to 18%.

cs.IT↗

arXiv2024

MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates

This work proposes a Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 updates, called MKOR, that improves the training time and convergence properties of deep neural networks (DNNs). Second-order techniques, while enjoying higher convergence rates vs first-order counterparts, have cubic complexity with respect to either the model size and/or the training batch size. Hence they exhibit poor scalability and performance in transformer models, e.g. large language models (LLMs), because the batch sizes in these models scale by the attention mechanism sequence length, leading to large model size and batch sizes. MKOR's complexity is quadratic with respect to the model size, alleviating the computation bottlenecks in second-order methods. Because of their high computation complexity, state-of-the-art implementations of second-order methods can only afford to update the second order information infrequently, and thus do not fully exploit the promise of better convergence from these updates. By reducing the communication complexity of the second-order updates as well as achieving a linear communication complexity, MKOR increases the frequency of second order updates. We also propose a hybrid version of MKOR (called MKOR-H) that mid-training falls backs to a first order optimizer if the second order updates no longer accelerate convergence. Our experiments show that MKOR outperforms state -of-the-art first order methods, e.g. the LAMB optimizer, and best implementations of second-order methods, i.e. KAISA/KFAC, up to 2.57x and 1.85x respectively on BERT-Large-Uncased on 64 GPUs.

cs.LG↗

arXiv2024

3GPP Release 18 Wake-up Receiver: Feature Overview and Evaluations

Enhancing the energy efficiency of devices stands as one of the key requirements in the fifth-generation (5G) cellular network and its evolutions toward the next generation wireless technology. Specifically, for battery-limited Internet-of-Things (IoT) devices where downlink monitoring significantly contributes to energy consumption, efficient solutions are required for power saving while addressing performance tradeoffs. In this regard, the use of a low-power wake-up receiver (WUR) and wake-up signal (WUS) is an attractive solution for reducing the energy consumption of devices without compromising the downlink latency. This paper provides an overview of the standardization study on the design of low-power WUR and WUS within Release 18 of the third-generation partnership project (3GPP). We describe design principles, receiver architectures, waveform characteristics, and device procedures upon detection of WUS. In addition, we provide representative results to show the performance of the WUR in terms of power saving, coverage, and network overhead along with highlighting design tradeoffs.

cs.IT↗

arXiv2023

Green, Quantized Federated Learning over Wireless Networks: An Energy-Efficient Design

In this paper, a green-quantized FL framework, which represents data with a finite precision level in both local training and uplink transmission, is proposed. Here, the finite precision level is captured through the use of quantized neural networks (QNNs) that quantize weights and activations in fixed-precision format. In the considered FL model, each device trains its QNN and transmits a quantized training result to the base station. Energy models for the local training and the transmission with quantization are rigorously derived. To minimize the energy consumption and the number of communication rounds simultaneously, a multi-objective optimization problem is formulated with respect to the number of local iterations, the number of selected devices, and the precision levels for both local training and transmission while ensuring convergence under a target accuracy constraint. To solve this problem, the convergence rate of the proposed FL system is analytically derived with respect to the system control variables. Then, the Pareto boundary of the problem is characterized to provide efficient solutions using the normal boundary inspection method. Design insights on balancing the tradeoff between the two objectives while achieving a target accuracy are drawn from using the Nash bargaining solution and analyzing the derived convergence rate. Simulation results show that the proposed FL framework can reduce energy consumption until convergence by up to 70\% compared to a baseline FL algorithm that represents data with full precision without damaging the convergence rate.

cs.LG↗

arXiv2023

On the Tradeoff between Energy, Precision, and Accuracy in Federated Quantized Neural Networks

Deploying federated learning (FL) over wireless networks with resource-constrained devices requires balancing between accuracy, energy efficiency, and precision. Prior art on FL often requires devices to train deep neural networks (DNNs) using a 32-bit precision level for data representation to improve accuracy. However, such algorithms are impractical for resource-constrained devices since DNNs could require execution of millions of operations. Thus, training DNNs with a high precision level incurs a high energy cost for FL. In this paper, a quantized FL framework, that represents data with a finite level of precision in both local training and uplink transmission, is proposed. Here, the finite level of precision is captured through the use of quantized neural networks (QNNs) that quantize weights and activations in fixed-precision format. In the considered FL model, each device trains its QNN and transmits a quantized training result to the base station. Energy models for the local training and the transmission with the quantization are rigorously derived. An energy minimization problem is formulated with respect to the level of precision while ensuring convergence. To solve the problem, we first analytically derive the FL convergence rate and use a line search method. Simulation results show that our FL framework can reduce energy consumption by up to 53% compared to a standard FL model. The results also shed light on the tradeoff between precision, energy, and accuracy in FL over wireless networks.

cs.LG↗

arXiv2022

Toward Smaller and Lower-Cost 5G Devices with Longer Battery Life: An Overview of 3GPP Release 17 RedCap

The fifth generation (5G) wireless technology is primarily developed to support three classes of use cases, namely, enhanced mobile broadband (eMBB), ultra-reliable and low-latency communication (URLLC), and massive machine-type communication (mMTC), with significantly different requirements in terms of data rate, latency, connection density and power consumption. Meanwhile, there are several key use cases, such as industrial wireless sensor networks, video surveillance, and wearables, whose requirements fall in-between those of eMBB, URLLC, and mMTC. In this regard, 5G can be further optimized to efficiently support such mid-range use cases. Therefore, in Release 17, the 3rd generation partnership project (3GPP) developed the essential features to support a new device type enabling reduced capability (RedCap) NR devices aiming at lower cost/complexity, smaller physical size, and longer battery life compared to regular 5G NR devices. In this paper, we provide a comprehensive overview of 3GPP Release 17 RedCap while describing newly introduced features, cost reduction and power saving gains, and performance and coexistence impacts. Moreover, we present key design guidelines, fundamental tradeoffs, and future outlook for RedCap evolution.

cs.IT↗

arXiv2021

Towards 6G with Connected Sky: UAVs and Beyond

The large-scale and ever-growing use of unmanned aerial vehicles (UAVs) in a wide range of applications is foreseen to be a major part of beyond 5G and 6G wireless networks in the next decade. The effective support of such massive deployment of UAVs requires offering reliable, secure, and cost-effective wireless connectivity. In this regard, cellular networks play essential roles in serving UAVs acting as flying user equipments. While the cellular networks provide promising connectivity solutions for UAVs, enabling robust UAV operations faces several challenges. In this paper, an overview of key barriers and design considerations of widespread commercial use of flying UAVs are presented along with their potential solutions. In addition, we discuss how cellular networks can support UAVs by relying on their advanced features, network intelligence, key enabling technologies for beyond 5G and 6G, and exploiting new tools from machine learning. Finally, we shed light on offering wireless services to high altitudes and the integration of non-terrestrial networks with terrestrial networks towards limitless connectivity in 6G.

cs.IT↗

arXiv2021

Coverage Evaluation for 5G Reduced Capability New Radio (NR-RedCap)

The fifth generation (5G) wireless technology is primarily designed to address a wide range of use cases categorized into the enhanced mobile broadband (eMBB), ultra-reliable and low latency communication (URLLC), and massive machine-type communication (mMTC). Nevertheless, there are a few other use cases which are in-between these main use cases such as industrial wireless sensor networks, video surveillance, or wearables. In order to efficiently serve such use cases, in Release 17, the 3rd generation partnership project (3GPP) introduced the reduced capability NR devices (NR-RedCap) with lower cost and complexity, smaller form factor and longer battery life compared to regular NR devices. However, one key potential consequence of device cost and complexity reduction is the coverage loss. In this paper, we provide a comprehensive evaluation of NR RedCap coverage for different physical channels and initial access messages to identify the channels/messages that are potentially coverage limiting for RedCap UEs. We perform the coverage evaluations for RedCap UEs operating in three different scenarios, namely Rural, Urban and Indoor with carrier frequencies 700 MHz, 2.6 GHz and 28 GHz, respectively. Our results confirm that for all the considered scenarios, the amounts of required coverage recovery for RedCap channels are either less than 1 dB or can be compensated by considering smaller data rate targets for RedCap use cases.

cs.IT↗

arXiv2020

5G Air-to-Ground Network Design and Optimization: A Deep Learning Approach

Direct air-to-ground (A2G) communications leveraging the fifth-generation (5G) new radio (NR) can provide high-speed broadband in-flight connectivity to aircraft in the sky. A2G network deployment entails optimizing various design parameters such as inter-site distances, number of sectors per site, and the up-tilt angles of sector antennas. The system-level design guidelines in the existing work on A2G network are rather limited. In this paper, a novel deep learning-based framework is proposed for efficient design and optimization of a 5G A2G network. The devised architecture comprises two deep neural networks (DNNs): the first DNN is used for approximating the 5G A2G network behavior in terms of user throughput, and the second DNN is developed as a function optimizer to find the throughput-optimal deployment parameters including antenna up-tilt angles and inter-site distances. Simulation results are provided to validate the proposed model and reveal system-level design insights.

cs.IT↗