arXiv subjects

Lei Guan

Publications and source records attributed to Lei Guan.

13 recordsLinked to original sources

arXiv2026

X-SAM: Boosting Sharpness-Aware Minimization with Dominant-Eigenvector Gradient Correction

Sharpness-Aware Minimization (SAM) aims to improve generalization by minimizing a worst-case perturbed loss over a small neighborhood of model parameters. However, during training, its optimization behavior does not always align with theoretical expectations, since both sharp and flat regions may yield a small perturbed loss. In such cases, the gradient may still point toward sharp regions, failing to achieve the intended effect of SAM. To address this issue, we investigate SAM from a spectral and geometric perspective: specifically, we utilize the angle between the gradient and the leading eigenvector of the Hessian as a measure of sharpness. Our analysis illustrates that when this angle is less than or equal to ninety degrees, the effect of SAM's sharpness regularization can be weakened. Furthermore, we propose an explicit eigenvector-aligned SAM (X-SAM), which corrects the gradient via orthogonal decomposition along the top eigenvector, enabling more direct and efficient regularization of the Hessian's maximum eigenvalue. We prove X-SAM's convergence and superior generalization, with extensive experimental evaluations confirming both theoretical and practical advantages.

cs.LG↗

arXiv2026

Multi-User Covert Communications via Intelligent Spectrum Control

This paper investigates the performance of multi-user covert communications over a fixed bandwidth in a multi-cell scenario with both eavesdroppers and malicious jammers. We propose an intelligent spectrum control (ISC) scheme that combines high-accuracy spectrum sensing with AI-assisted real-time decision-making to generate time-frequency dynamic occupation patterns for multiple legitimate users. The scheme can proactively avoid external interference and intra-system co-channel collisions, thereby improving covertness and reliability. Within this framework, we derive closed-form expressions for the detection error probability (DEP) of the eavesdropper and the reliable transmission probability (RTP) of legitimate users under multi-user joint detection. We then analytically optimize the transmission power that can maximize the covert rate (CR), as well as the maximum number of users that can access the system covertly and concurrently under given covertness and reliability constraints. Simulation results confirm the tight match between the analytical and Monte Carlo curves, and show that the proposed scheme can achieve a higher DEP, a larger RTP, and a greater multi-user capacity than the benchmark scheme.

cs.IT↗

arXiv2026

Multi-Agent-Driven Cognitive Secure Communications in Satellite-Terrestrial Networks

Satellite-terrestrial networks (STNs) have emerged as a promising architecture for providing seamless wireless coverage and connectivity for multiple users. However, potential malicious eavesdroppers pose a serious threat to the private information via STNs due to their non-cooperative behavior and ability to launch intelligent attacks. To address this challenge, we propose a cognitive secure communication framework driven by multiple agents that coordinates spectrum scheduling and protection through real-time sensing, thereby disrupting the judgment of eavesdroppers while preserving reliable data transmission. On this basis, we formulate an optimization problem to maximize the secrecy probability of legitimate users, subject to a reliable transmission probability threshold. To tackle this problem, we propose a two-layer coordinated defense system. First, we develop a foundation layer based on multi-agent coordination schedule to determine the satellite operation matrix and the frequency slot occupation matrices, aiming to mitigate spectrum congestion and enhance transmission reliability. Then, we exploit generative adversarial networks to produce adversarial matrices, and employ learning-aided power control to set real and adversarial signal powers for protection layer, which actively degrades the inference capability of eavesdroppers. Simulation results demonstrate that the proposed method outperforms benchmark methods in terms of enhancing security performance and reducing power overhead for STNs in the cognitive secure communication scenario.

cs.CR↗

arXiv2025

PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction

Asynchronous pipeline model parallelism with a "1F1B" (one forward, one backward) schedule generates little bubble overhead and always provides quite a high throughput. However, the "1F1B" schedule inevitably leads to weight inconsistency and weight staleness issues due to the cross-training of different mini-batches across GPUs. To simultaneously address these two problems, in this paper, we propose an optimizer-dependent weight prediction strategy (a.k.a PipeOptim) for asynchronous pipeline training. The key insight of our proposal is that we employ a weight prediction strategy in the forward pass to ensure that each mini-batch uses consistent and staleness-free weights to compute the forward pass. To be concrete, we first construct the weight prediction scheme based on the update rule of the used optimizer when training the deep neural network models. Then throughout the "1F1B" pipelined training, each mini-batch is mandated to execute weight prediction ahead of the forward pass, subsequently employing the predicted weights to perform the forward pass. As a result, PipeOptim 1) inherits the advantage of the "1F1B" schedule and generates pretty high throughput, and 2) can ensure effective parameter learning regardless of the type of the used optimizer. To verify the effectiveness of our proposal, we conducted extensive experimental evaluations using eight different deep-learning models spanning three machine-learning tasks including image classification, sentiment analysis, and machine translation. The experiment results demonstrate that PipeOptim outperforms the popular pipelined approaches including GPipe, PipeDream, PipeDream-2BW, and SpecTrain. The code of PipeOptim can be accessible at https://github.com/guanleics/PipeOptim.

cs.LG↗

arXiv2024

XGrad: Boosting Gradient-Based Optimizers With Weight Prediction

In this paper, we propose a general deep learning training framework XGrad which introduces weight prediction into the popular gradient-based optimizers to boost their convergence and generalization when training the deep neural network (DNN) models. In particular, ahead of each mini-batch training, the future weights are predicted according to the update rule of the used optimizer and are then applied to both the forward pass and backward propagation. In this way, during the whole training period, the optimizer always utilizes the gradients w.r.t. the future weights to update the DNN parameters, making the gradient-based optimizer achieve better convergence and generalization compared to the original optimizer without weight prediction. XGrad is rather straightforward to implement yet pretty effective in boosting the convergence of gradient-based optimizers and the accuracy of DNN models. Empirical results concerning five popular optimizers including SGD with momentum, Adam, AdamW, AdaBelief, and AdaM3 demonstrate the effectiveness of our proposal. The experimental results validate that XGrad can attain higher model accuracy than the baseline optimizers when training the DNN models. The code of XGrad will be available at: https://github.com/guanleics/XGrad.

cs.LG↗

arXiv2023

AdaPlus: Integrating Nesterov Momentum and Precise Stepsize Adjustment on AdamW Basis

This paper proposes an efficient optimizer called AdaPlus which integrates Nesterov momentum and precise stepsize adjustment on AdamW basis. AdaPlus combines the advantages of AdamW, Nadam, and AdaBelief and, in particular, does not introduce any extra hyper-parameters. We perform extensive experimental evaluations on three machine learning tasks to validate the effectiveness of AdaPlus. The experiment results validate that AdaPlus (i) among all the evaluated adaptive methods, performs most comparable with (even slightly better than) SGD with momentum on image classification tasks and (ii) outperforms other state-of-the-art optimizers on language modeling tasks and illustrates pretty high stability when training GANs. The experiment code of AdaPlus will be accessible at: https://github.com/guanleics/AdaPlus.

cs.LG↗

arXiv2023

Weight Prediction Boosts the Convergence of AdamW

In this paper, we introduce weight prediction into the AdamW optimizer to boost its convergence when training the deep neural network (DNN) models. In particular, ahead of each mini-batch training, we predict the future weights according to the update rule of AdamW and then apply the predicted future weights to do both forward pass and backward propagation. In this way, the AdamW optimizer always utilizes the gradients w.r.t. the future weights instead of current weights to update the DNN parameters, making the AdamW optimizer achieve better convergence. Our proposal is simple and straightforward to implement but effective in boosting the convergence of DNN training. We performed extensive experimental evaluations on image classification and language modeling tasks to verify the effectiveness of our proposal. The experimental results validate that our proposal can boost the convergence of AdamW and achieve better accuracy than AdamW when training the DNN models.

cs.LG↗

arXiv2022

A Cooperative Deception Strategy for Covert Communication in Presence of a Multi-antenna Adversary

Covert transmission is investigated for a cooperative deception strategy, where a cooperative jammer (Jammer) tries to attract a multi-antenna adversary (Willie) and degrade the adversary's reception ability for the signal from a transmitter (Alice). For this strategy, we formulate an optimization problem to maximize the covert rate when three different types of channel state information (CSI) are available. The total power is optimally allocated between Alice and Jammer subject to Kullback-Leibler (KL) divergence constraint. Different from the existing literature, in our proposed strategy, we also determine the optimal transmission power at the jammer when Alice is silent, while existing works always assume that the jammer's power is fixed. Specifically, we apply the S-procedure to convert infinite constraints into linear-matrix-inequalities (LMI) constraints. When statistical CSI at Willie is available, we convert double integration to single integration using asymptotic approximation and substitution method. In addition, the transmission strategy without jammer deception is studied as a benchmark. Finally, our simulation results show that for the proposed strategy, the covert rate is increased with the number of antennas at Willie. Moreover, compared to the benchmark, our proposed strategy is more robust in face of imperfect CSI.

eess.SP↗

arXiv2021

Covert Transmission Assisted by Intelligent Reflecting Surface

Covert transmission is studied for an intelligent reflecting surface (IRS) aided communication system, where Alice aims to transmit messages to Bob without being detected by the warden Willie. Specifically, an IRS is used to increase the data rate at Bob under a covert constraint. For the considered model, when Alice is equipped with a single antenna, the transmission power at Alice and phase shifts at the IRS are jointly optimized to maximize the covert transmission rate with either instantaneous or partial channel state information (CSI) of Willie's link. In addition, when multiple antennas are deployed at Alice, we formulate a joint transmit beamforming and IRS phase shift optimization problem to maximize the covert transmission rate. One optimal algorithm and two low-complexity suboptimal algorithms are proposed to solve the problem. Furthermore, for the case of imperfect CSI of Willie's link, the optimization problem is reformulated by using the triangle and the Cauchy-Schwarz inequalities. The reformulated optimization problems are solved using an alterative algorithm, semidefinite relaxation (SDR) and Gaussian randomization techniques. Finally, simulations are performed to verify our analysis. The simulation results show that an IRS can degrade the covert transmission rate when Willie is closer to the IRS than Bob.

cs.IT↗

arXiv2020

XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training

We propose XPipe, an efficient asynchronous pipeline model parallelism approach for multi-GPU DNN training. XPipe is designed to use multiple GPUs to concurrently and continuously train different parts of a DNN model. To improve GPU utilization and achieve high throughput, it splits a mini-batch into a set of micro-batches. It allows the overlapping of the pipelines of multiple micro-batches, including those belonging to different mini-batches. Most importantly, the novel weight prediction strategy adopted by XPipe enables it to effectively address the weight inconsistency and staleness issues incurred by the asynchronous pipeline parallelism. As a result, XPipe incorporates the advantages of both synchronous and asynchronous pipeline model parallelism approaches. Concretely, it can achieve very comparable (even slightly better) model accuracy as its synchronous counterpart while obtaining higher throughput than it. Experimental results show that XPipe outperforms other state-of-the-art synchronous and asynchronous model parallelism approaches.

cs.LG↗

arXiv2018

Non-ergodic Convergence Analysis of Heavy-Ball Algorithms

In this paper, we revisit the convergence of the Heavy-ball method, and present improved convergence complexity results in the convex setting. We provide the first non-ergodic O(1/k) rate result of the Heavy-ball algorithm with constant step size for coercive objective functions. For objective functions satisfying a relaxed strongly convex condition, the linear convergence is established under weaker assumptions on the step size and inertial parameter than made in the existing literature. We extend our results to multi-block version of the algorithm with both the cyclic and stochastic update rules. In addition, our results can also be extended to decentralized optimization, where the ergodic analysis is not applicable.

math.OC↗

arXiv2018

An Efficient ADMM-Based Algorithm to Nonconvex Penalized Support Vector Machines

Support vector machines (SVMs) with sparsity-inducing nonconvex penalties have received considerable attentions for the characteristics of automatic classification and variable selection. However, it is quite challenging to solve the nonconvex penalized SVMs due to their nondifferentiability, nonsmoothness and nonconvexity. In this paper, we propose an efficient ADMM-based algorithm to the nonconvex penalized SVMs. The proposed algorithm covers a large class of commonly used nonconvex regularization terms including the smooth clipped absolute deviation (SCAD) penalty, minimax concave penalty (MCP), log-sum penalty (LSP) and capped-$\ell_1$ penalty. The computational complexity analysis shows that the proposed algorithm enjoys low computational cost. Moreover, the convergence of the proposed algorithm is guaranteed. Extensive experimental evaluations on five benchmark datasets demonstrate the superior performance of the proposed algorithm to other three state-of-the-art approaches.

stat.ML↗

arXiv2015

Hybrid RF and Digital Beamformer for Cellular Networks: Algorithms, Microwave Architectures and Measurements

Modern wireless communication networks, particularly cellular networks utilize multiple antennas to improve the capacity and signal coverage. In these systems, typically an active transceiver is connected to each antenna. However, this one-to-one mapping between transceivers and antennas will dramatically increase the cost and complexity of a large phased antenna array system. In this paper, firstly we propose a \emph{partially adaptive} beamformer architecture where a reduced number of transceivers with a digital beamformer (DBF) is connected to an increased number of antennas through an RF beamforming network (RFBN). Then, based on the proposed architecture, we present a methodology to derive the minimum number of transceivers that are required for marco-cell and small-cell base stations, respectively. Subsequently, in order to achieve optimal beampatterns with given cellular standard requirements and RF operational constraints, we propose efficient algorithms to jointly design DBF and RFBN. Starting from the proposed algorithms, we specify generic microwave RFBNs for optimal marco-cell and small-cell networks. In order to verify the proposed approaches, we compare the performance of RFBN using simulations and anechoic chamber measurements. Experimental measurement results confirm the robustness and performance of the proposed hybrid DBF-RFBN concept eventually ensuring that theoretical multi-antenna capacity and coverage are achieved at a little incremental cost.

cs.IT↗