arXiv subjects

Xiangming Meng

Publications and source records attributed to Xiangming Meng.

34 records · Page 2Linked to original sources

arXiv2021

Ising Model Selection Using $\ell_{1}$-Regularized Linear Regression: A Statistical Mechanics Analysis

We theoretically analyze the typical learning performance of $\ell_{1}$-regularized linear regression ($\ell_1$-LinR) for Ising model selection using the replica method from statistical mechanics. For typical random regular graphs in the paramagnetic phase, an accurate estimate of the typical sample complexity of $\ell_1$-LinR is obtained. Remarkably, despite the model misspecification, $\ell_1$-LinR is model selection consistent with the same order of sample complexity as $\ell_{1}$-regularized logistic regression ($\ell_1$-LogR), i.e., $M=\mathcal{O}\left(\log N\right)$, where $N$ is the number of variables of the Ising model. Moreover, we provide an efficient method to accurately predict the non-asymptotic behavior of $\ell_1$-LinR for moderate $M, N$, such as precision and recall. Simulations show a fairly good agreement between theoretical predictions and experimental results, even for graphs with many loops, which supports our findings. Although this paper mainly focuses on $\ell_1$-LinR, our method is readily applicable for precisely characterizing the typical learning performances of a wide class of $\ell_{1}$-regularized $M$-estimators including $\ell_1$-LogR and interaction screening.

cs.LG↗

arXiv2021

Unitary Approximate Message Passing for Sparse Bayesian Learning

Sparse Bayesian learning (SBL) can be implemented with low complexity based on the approximate message passing (AMP) algorithm. However, it does not work well for a generic measurement matrix, which may cause AMP to diverge. Damped AMP has been used for SBL to alleviate the problem at the cost of reducing convergence speed. In this work, we propose a new SBL algorithm based on structured variational inference, leveraging AMP with a unitary transformation (UAMP). Both single measurement vector and multiple measurement vector problems are investigated. It is shown that, compared to state-of-the-art AMP-based SBL algorithms, the proposed UAMP-SBL is more robust and efficient, leading to remarkably better performance.

eess.SP↗

arXiv2020

Structure Learning in Inverse Ising Problems Using $\ell_2$-Regularized Linear Estimator

The inference performance of the pseudolikelihood method is discussed in the framework of the inverse Ising problem when the $\ell_2$-regularized (ridge) linear regression is adopted. This setup is introduced for theoretically investigating the situation where the data generation model is different from the inference one, namely the model mismatch situation. In the teacher-student scenario under the assumption that the teacher couplings are sparse, the analysis is conducted using the replica and cavity methods, with a special focus on whether the presence/absence of teacher couplings is correctly inferred or not. The result indicates that despite the model mismatch, one can perfectly identify the network structure using naive linear regression without regularization when the number of spins $N$ is smaller than the dataset size $M$, in the thermodynamic limit $N\to \infty$. Further, to access the underdetermined region $M < N$, we examine the effect of the $\ell_2$ regularization, and find that biases appear in all the coupling estimates, preventing the perfect identification of the network structure. We, however, find that the biases are shown to decay exponentially fast as the distance from the center spin chosen in the pseudolikelihood method grows. Based on this finding, we propose a two-stage estimator: In the first stage, the ridge regression is used and the estimates are pruned by a relatively small threshold; in the second stage the naive linear regression is conducted only on the remaining couplings, and the resultant estimates are again pruned by another relatively large threshold. This estimator with the appropriate regularization coefficient and thresholds is shown to achieve the perfect identification of the network structure even in $0<M/N<1$. Results of extensive numerical experiments support these findings.

cond-mat.dis-nn↗

arXiv2020

Training Binary Neural Networks using the Bayesian Learning Rule

Neural networks with binary weights are computation-efficient and hardware-friendly, but their training is challenging because it involves a discrete optimization problem. Surprisingly, ignoring the discrete nature of the problem and using gradient-based methods, such as the Straight-Through Estimator, still works well in practice. This raises the question: are there principled approaches which justify such methods? In this paper, we propose such an approach using the Bayesian learning rule. The rule, when applied to estimate a Bernoulli distribution over the binary weights, results in an algorithm which justifies some of the algorithmic choices made by the previous approaches. The algorithm not only obtains state-of-the-art performance, but also enables uncertainty estimation for continual learning to avoid catastrophic forgetting. Our work provides a principled approach for training binary neural networks which justifies and extends existing approaches.

cs.LG↗

arXiv2020

Training Restricted Boltzmann Machines with Binary Synapses using the Bayesian Learning Rule

Restricted Boltzmann machines (RBMs) with low-precision synapses are much appealing with high energy efficiency. However, training RBMs with binary synapses is challenging due to the discrete nature of synapses. Recently Huang proposed one efficient method to train RBMs with binary synapses by using a combination of gradient ascent and the message passing algorithm under the variational inference framework. However, additional heuristic clipping operation is needed. In this technical note, inspired from Huang's work , we propose one alternative optimization method using the Bayesian learning rule, which is one natural gradient variational inference method. As opposed to Huang's method, we update the natural parameters of the variational symmetric Bernoulli distribution rather than the expectation parameters. Since the natural parameters take values in the entire real domain, no additional clipping is needed. Interestingly, the algorithm in \cite{huang2019data} could be viewed as one first-order approximation of the proposed algorithm, which justifies its efficacy with heuristic clipping.

cs.LG↗

arXiv2019

Grid-less Variational Bayesian Inference of Line Spectral from Quantized Samples

Efficient estimation of line spectral from quantized samples is of significant importance in information theory and signal processing, e.g., channel estimation in energy efficient massive MIMO systems and direction of arrival estimation. The goal of this paper is to recover the line spectral as well as its corresponding parameters including the model order, frequencies and amplitudes from heavily quantized samples. To this end, we propose an efficient grid-less Bayesian algorithm named VALSE-EP, which is a combination of the variational line spectral estimation (VALSE) and expectation propagation (EP). The basic idea of VALSE-EP is to iteratively approximate the challenging quantized model of line spectral estimation as a sequence of simple pseudo unquantized models so that the VALSE can be applied. Note that the noise in the pseudo linear model is heteroscedastic, i.e., different components having different variances, and a variant of the VALSE is re-derived to obtain the final VALSE-EP. Moreover, to obtain a benchmark performance of the proposed algorithm, the Cramér Rao bound (CRB) is derived. Finally, numerical experiments on both synthetic and real data are performed, demonstrating the near CRB performance of the proposed VALSE-EP for line spectral estimation from quantized samples.

cs.IT↗

arXiv2018

Bilinear Adaptive Generalized Vector Approximate Message Passing

This paper considers the generalized bilinear recovery problem which aims to jointly recover the vector $\mathbf b$ and the matrix $\mathbf X$ from componentwise nonlinear measurements ${\mathbf Y}\sim p({\mathbf Y}|{\mathbf Z})=\prod\limits_{i,j}p(Y_{ij}|Z_{ij})$, where ${\mathbf Z}={\mathbf A}({\mathbf b}){\mathbf X}$, ${\mathbf A}(\cdot)$ is a known affine linear function of $\mathbf b$, and $p(Y_{ij}|Z_{ij})$ is a scalar conditional distribution which models the general output transform. A wide range of real-world applications, e.g., quantized compressed sensing with matrix uncertainty, blind self-calibration and dictionary learning from nonlinear measurements, one-bit matrix completion, etc., can be cast as the generalized bilinear recovery problem. To address this problem, we propose a novel algorithm called the Bilinear Adaptive Generalized Vector Approximate Message Passing (BAd-GVAMP), which extends the recently proposed Bilinear Adaptive Vector AMP (BAd-VAMP) algorithm to incorporate arbitrary distributions on the output transform. Numerical results on various applications demonstrate the effectiveness of the proposed BAd-GVAMP algorithm.

cs.IT↗

arXiv2018

A Universal Receiver for Uplink NOMA Systems

Given its capability in efficient radio resource sharing, non-orthogonal multiple access (NOMA) has been identified as a promising technology in 5G to improve the system capacity, user connectivity, and scheduling latency. A dozen of uplink NOMA schemes have been proposed recently and this paper considers the design of a universal receiver suitable for all potential designs of NOMA schemes. Firstly, a general turbo-like iterative receiver structure is introduced, under which, a universal expectation propagation algorithm (EPA) detector with hybrid parallel interference cancellation (PIC) is proposed (EPA in short). Link-level simulations show that the proposed EPA receiver can achieve superior block error rate (BLER) performance with implementation friendly complexity and fast convergence, and is always better than the traditional codeword level MMSE-PIC receiver for various kinds of NOMA schemes.

cs.IT↗

arXiv2018

Turbo-like Iterative Multi-user Receiver Design for 5G Non-orthogonal Multiple Access

Non-orthogonal multiple access (NoMA) as an efficient way of radio resource sharing has been identified as a promising technology in 5G to help improving system capacity, user connectivity, and service latency in 5G communications. This paper provides a brief overview of the progress of NoMA transceiver study in 3GPP, with special focus on the design of turbo-like iterative multi-user (MU) receivers. There are various types of MU receivers depending on the combinations of MU detectors and interference cancellation (IC) schemes. Link-level simulations show that expectation propagation algorithm (EPA) with hybrid parallel interference cancellation (PIC) is a promising MU receiver, which can achieve fast convergence and similar performance as message passing algorithm (MPA) with much lower complexity.

cs.IT↗

arXiv2018

Compressive Massive Random Access for Massive Machine-Type Communications (mMTC)

In future wireless networks, one fundamental challenge for massive machine-type communications (mMTC) lies in the reliable support of massive connectivity with low latency. Against this background, this paper proposes a compressive sensing (CS)-based massive random access scheme for mMTC by leveraging the inherent sporadic traffic, where both the active devices and their channels can be jointly estimated with low overhead. Specifically, we consider devices in the uplink massive random access adopt pseudo random pilots, which are designed under the framework of CS theory. Meanwhile, the massive random access at the base stations (BS) can be formulated as the sparse signal recovery problem by leveraging the sparse nature of active devices. Moreover, by exploiting the structured sparsity among different receiver antennas and subcarriers, we develop a distributed multiple measurement vector approximate message passing (DMMV-AMP) algorithm for further improved performance. Additionally, the state evolution (SE) of the proposed DMMV-AMP algorithm is derived to predict the performance. Simulation results demonstrate the superiority of the proposed scheme, which exhibits a good tightness with the theoretical SE.

cs.IT↗

arXiv2018

Vector Approximate Message Passing Algorithm for Structured Perturbed Sensing Matrix

In this paper, we consider a general form of noisy compressive sensing (CS) where the sensing matrix is not precisely known. Such cases exist when there are imperfections or unknown calibration parameters during the measurement process. Particularly, the sensing matrix may have some structure, which makes the perturbation follow a fixed pattern. While previous work has focused on extending the approximate message passing (AMP) and LASSO algorithm to deal with the independent and identically distributed (i.i.d.) perturbation, we propose the robust variant vector approximate message passing (VAMP) algorithm with the perturbation being structured, based on the recent VAMP algorithm. The performance of the robust version of VAMP is demonstrated numerically.

eess.SP↗

arXiv2018

Binary Sparse Bayesian Learning Algorithm for One-bit Compressed Sensing

In this letter, a binary sparse Bayesian learning (BSBL) algorithm is proposed to slove the one-bit compressed sensing (CS) problem in both single measurement vector (SMV) and multiple measurement vectors (MMVs). By utilising the Bussgang-like decomposition, the one-bit CS problem can be approximated as a standard linear model. Consequently, the standard SBL algorithm can be naturally incorporated. Numerical results demonstrate the effectiveness of the BSBL algorithm.

cs.IT↗

arXiv2017

A Unified Bayesian Inference Framework for Generalized Linear Models

In this letter, we present a unified Bayesian inference framework for generalized linear models (GLM) which iteratively reduces the GLM problem to a sequence of standard linear model (SLM) problems. This framework provides new perspectives on some established GLM algorithms derived from SLM ones and also suggests novel extensions for some other SLM algorithms. Specific instances elucidated under such framework are the GLM versions of approximate message passing (AMP), vector AMP (VAMP), and sparse Bayesian learning (SBL). It is proved that the resultant GLM version of AMP is equivalent to the well-known generalized approximate message passing (GAMP). Numerical results for 1-bit quantized compressed sensing (CS) demonstrate the effectiveness of this unified framework.

cs.IT↗

arXiv2017

Low Complexity Receiver for Uplink SCMA System via Expectation Propagation

Sparse code multiple access (SCMA) scheme is considered to be one promising non-orthogonal multiple access technology for the future fifth generation (5G) communications. Due to the sparse nature, message passing algorithm (MPA) has been used as the receiver to achieve close to maximum likelihood (ML) detection performance with much lower complexity. However, the complexity order of MPA is still exponential with the size of codebook and the degree of signal superposition on a given resource element. In this paper, we propose a novel low complexity iterative receiver based on expectation propagation algorithm (EPA), which reduces the complexity order from exponential to linear. Simulation results demonstrate that the proposed EPA receiver achieves nearly the same block error rate (BLER) performance as the conventional message passing algorithm (MPA) receiver with orders less complexity.

cs.IT↗

arXiv2016

Concise Derivation of Complex Bayesian Approximate Message Passing via Expectation Propagation

In this paper, we address the problem of recovering complex-valued signals from a set of complex-valued linear measurements. Approximate message passing (AMP) is one state-of-the-art algorithm to recover real-valued sparse signals. However, the extension of AMP to complex-valued case is nontrivial and no detailed and rigorous derivation has been explicitly presented. To fill this gap, we extend AMP to complex Bayesian approximate message passing (CB-AMP) using expectation propagation (EP). This novel perspective leads to a concise derivation of CB-AMP without sophisticated transformations between the complex domain and the real domain. In addition, we have derived state evolution equations to predict the reconstruction performance of CB-AMP. Simulation results are presented to demonstrate the efficiency of CB-AMP and state evolution.

cs.IT↗

arXiv2016

Approximate Message Passing with Nearest Neighbor Sparsity Pattern Learning

We consider the problem of recovering clustered sparse signals with no prior knowledge of the sparsity pattern. Beyond simple sparsity, signals of interest often exhibits an underlying sparsity pattern which, if leveraged, can improve the reconstruction performance. However, the sparsity pattern is usually unknown a priori. Inspired by the idea of k-nearest neighbor (k-NN) algorithm, we propose an efficient algorithm termed approximate message passing with nearest neighbor sparsity pattern learning (AMP-NNSPL), which learns the sparsity pattern adaptively. AMP-NNSPL specifies a flexible spike and slab prior on the unknown signal and, after each AMP iteration, sets the sparse ratios as the average of the nearest neighbor estimates via expectation maximization (EM). Experimental results on both synthetic and real data demonstrate the superiority of our proposed algorithm both in terms of reconstruction performance and computational complexity.

cs.IT↗