arXiv subjects

Brian M. Kurkoski

Publications and source records attributed to Brian M. Kurkoski.

At least 19 recordsLinked to original sources

arXiv2026

Memory AMP: Overflow Avoidance, Complexity Reduction, and Comparative Analysis

Approximate message passing (AMP)-type algorithms are widely used for signal recovery in high-dimensional noisy linear systems. Recently, a framework called memory AMP (MAMP) was introduced, offering a new approach to incorporating memory terms within AMP algorithms. Building on this, a low-complexity gradient descent MAMP (GD-MAMP) was proposed for right-unitarily invariant matrices. In this paper, we first address an overflow problem in GD-MAMP caused by intermediate variables exceeding the floating-point range, which typically occurs when the condition number is large. Second, we propose two low-complexity variants of GD-MAMP: one replaces full-length memory with partial memory, while the other reduces the number of matrix-vector products per iteration by $1/3$ (from three to two). Neither degrades the convergence speed notably. Third, we develop a general gradient-based formulation for designing MAMP algorithms. This formulation recovers warm-started conjugate gradient VAMP (WS-CG-VAMP) as a special case. Furthermore, we show that the computation of the orthogonalization parameters in this formulation can suffer from catastrophic cancellation, which explains the finite-precision instability of WS-CG-VAMP. Finally, we derive an equivalent reformulation, termed WS-CG-VAMP(r), which reduces the number of matrix-vector products by up to $50\%$. Measured by matrix-vector products, GD-MAMP converges faster for small condition numbers, whereas WS-CG-VAMP(r) converges faster for large ones under high-precision arithmetic but may diverge in IEEE double precision due to catastrophic cancellation.

cs.IT↗

arXiv2026

Binary Autoencoder for Mechanistic Interpretability of Large Language Models

Existing works are dedicated to untangling atomized numerical components (features) from the hidden states of Large Language Models (LLMs). However, they typically rely on autoencoders constrained by some training-time regularization on single training instances, without an explicit guarantee of global sparsity among instances, causing a large amount of dense (simultaneously inactive) features, harming the feature sparsity and atomization. In this paper, we propose a novel autoencoder variant that enforces minimal entropy on minibatches of hidden activations, thereby promoting feature independence and sparsity across instances. For efficient entropy calculation, we discretize the hidden activations to 1-bit via a step function and apply gradient estimation to enable backpropagation, so that we term it as Binary Autoencoder (BAE) and empirically demonstrate two major applications: (1) Feature set entropy calculation. Entropy can be reliably estimated on binary hidden activations, which can be leveraged to characterize the inference dynamics of LLMs. (2) Feature untangling. Compared to typical methods, due to improved training strategy, BAE avoids dense features while producing the largest number of interpretable ones among baselines.

cs.LG↗

arXiv2025

Construction A Lattice Design Based on the Truncated Union Bound

This paper considers $n= 128$ dimensional construction A lattice design, using binary codes with known minimum Hamming distance and codeword multiplicity, the number of minimum weight codeword. A truncated theta series of the lattice is explicitly given to obtain the truncated union bound to estimate the word error rate under maximum likelihood decoding. The best component code is selected by minimizing the required volume-to-noise ratio (VNR) for a target word error rate $P_e$. The estimate becomes accurate for $P_e \leq 10^{-4}$, and design examples are given with the best extended BCH codes and polar codes for $P_e= 10^{-4}$ to $10^{-8}$. A lower error rate is achieved compared to that by the classic balanced distance rule and the equal error probability rule. The $(128, 106, 8)$ EBCH code gives the best-known $n=128$ construction A lattice at $P_e= 10^{-5}$.

cs.IT↗

arXiv2025

Finite Dimensional Lattice Codes with Self Error-Detection and Retry Decoding

Lattice codes with optimal decoding coefficient are capacity-achieving when dimension $N \rightarrow \infty$. In communications systems, finite dimensional lattice codes are considered, where the optimal decoding coefficients may still fail decoding even when $R< C$. This paper presents a new retry decoding scheme for finite dimensional lattice-based transmissions. When decoding errors are detected, the receiver is allowed to adjust the value of decoding coefficients and retry decoding, instead of requesting a re-transmission immediately which causes high latency. This scheme is considered for both point-to-point single user transmission and compute-forward (CF) relaying with power unconstrained relays, by which a lower word error rate (WER) is achieved than conventional one-shot decoding with optimal coefficients. A lattice/lattice code construction, called CRC-embedded lattice/lattice code, is presented to provide physical layer error detection to enable retry decoding. For CF relaying, a shaping lattice design is given so that the decoder is able to detect errors from CF linear combinations without requiring individual users' messages. The numerical results show gains of up to 1.31 dB and 1.08 dB at error probability $10^{-5}$ for a 2-user CF relay using 128- and 256-dimensional lattice codes with optimized CRC length and 2 decoding trials in total.

cs.IT↗

arXiv2025

Lower Bound on the Error Rate of Genie-Aided Lattice Decoding

A genie-aided decoder for finite dimensional lattice codes is considered. The decoder may exhaustively search through all possible scaling factors $α\in \mathbb{R}$. We show that this decoder can achieve lower word error rate (WER) than the one-shot decoder using $α_{MMSE}$ as a scaling factor. A lower bound on the WER for the decoder is found by considering the covering sphere of the lattice Voronoi region. The proposed decoder and the bound are valid for both power-constrained lattice codes and lattices. If the genie is applied at the decoder, E8 lattice code has 0.5 dB gain and BW16 lattice code has 0.4 dB gain at WER of $10^{-4}$ compared with the one-shot decoder using $α_{MMSE}$. A method for estimating the WER of the decoder is provided by considering the effective sphere of the lattice Voronoi region, which shows an accurate estimate for E8 and BW16 lattice codes. In the case of per-dimension power $P \rightarrow \infty$, an asymptotic expression of the bound is given in a closed form. A practical implementation of a simplified decoder is given by considering CRC-embedded $n=128$ polar code lattice.

cs.IT↗

arXiv2024

Overflow-Avoiding Memory AMP

Approximate Message Passing (AMP) type algorithms are widely used for signal recovery in high-dimensional noisy linear systems. Recently, a principle called Memory AMP (MAMP) was proposed. Leveraging this principle, the gradient descent MAMP (GD-MAMP) algorithm was designed, inheriting the strengths of AMP and OAMP/VAMP. In this paper, we first provide an overflow-avoiding GD-MAMP (OA-GD-MAMP) to address the overflow problem that arises from some intermediate variables exceeding the range of floating point numbers. Second, we develop a complexity-reduced GD-MAMP (CR-GD-MAMP) to reduce the number of matrix-vector products per iteration by 1/3 (from 3 to 2) with little to no impact on the convergence speed.

cs.IT↗

arXiv2024

On the Existence of Cyclic Lattice Codes

A coding lattice $Λ_c$ and a shaping lattice $Λ_s$ forms a nested lattice code $\mathcal{C}$ if $Λ_s \subseteq Λ_c$. Under some conditions, $\mathcal{C}$ is a finite cyclic group formed by rectangular encoding. This paper presents the conditions for the existence of such $\mathcal{C}$ and provides some designs. These designs correspond to solutions to linear Diophantine equations so that a cyclic lattice code $\mathcal C$ of arbitrary codebook size $M$ can possess group isomorphism, which is an essential property for a nested lattice code to be applied in physical layer network relaying techniques such as compute and forward.

cs.IT↗

arXiv2023

Algebra of L-banded Matrices

Convergence is a crucial issue in iterative algorithms. Damping is commonly employed to ensure the convergence of iterative algorithms. The conventional ways of damping are scalar-wise, and either heuristic or empirical. Recently, an analytically optimized vector damping was proposed for memory message-passing (iterative) algorithms. As a result, it yields a special class of covariance matrices called L-banded matrices. In this paper, we show these matrices have broad algebraic properties arising from their L-banded structure. In particular, compact analytic expressions for the LDL decomposition, the Cholesky decomposition, the determinant after a column substitution, minors, and cofactors are derived. Furthermore, necessary and sufficient conditions for an L-banded matrix to be definite, a recurrence to obtain the characteristic polynomial, and some other properties are given. In addition, we give new derivations of the determinant and the inverse. (It's crucial to emphasize that some works have independently studied matrices with this special structure, named as L-matrices. Specifically, L-banded matrices are regarded as L-matrices with real and finite entries.)

eess.SP↗

arXiv2023

Sufficient-Statistic Memory AMP

Approximate message passing (AMP) type algorithms have been widely used in the signal reconstruction of certain large random linear systems. A key feature of the AMP-type algorithms is that their dynamics can be correctly described by state evolution. While state evolution is a useful analytic tool, its convergence is not guaranteed. To solve the convergence problem of the state evolution of AMP-type algorithms in principle, this paper proposes a sufficient-statistic memory AMP (SS-MAMP) algorithm framework under the conditions of right-unitarily invariant sensing matrices, Lipschitz-continuous local processors and the sufficient-statistic constraint (i.e., the current message of each local processor is a sufficient statistic of the signal vector given the current and all preceding messages). We show that the covariance matrices of SS-MAMP are L-banded and convergent, which is an optimal framework (from the local MMSE/LMMSE perspective) for AMP-type algorithms given the Lipschitz-continuous local processors. Given an arbitrary MAMP, we can construct an SS-MAMP by damping, which not only ensures the convergence of the state evolution, but also preserves the orthogonality, i.e., its dynamics can be correctly described by state evolution. As a byproduct, we prove that the Bayes-optimal orthogonal/vector AMP (BO-OAMP/VAMP) is an SS-MAMP. As an example, we construct a sufficient-statistic Bayes-optimal MAMP (SS-BO-MAMP) whose state evolution converges to the minimum (i.e., Bayes-optimal) mean square error (MSE) predicted by replica methods when it has a unique fixed point. In addition, the MSE of SS-BO-MAMP is not worse than the original BO-MAMP. Finally, simulations are provided to support the theoretical results.

cs.IT↗

arXiv2022

Memory AMP

Approximate message passing (AMP) is a low-cost iterative parameter-estimation technique for certain high-dimensional linear systems with non-Gaussian distributions. AMP only applies to independent identically distributed (IID) transform matrices, but may become unreliable (e.g., perform poorly or even diverge) for other matrix ensembles, especially for ill-conditioned ones. To solve this issue, orthogonal/vector AMP (OAMP/VAMP) was proposed for general right-unitarily-invariant matrices. However, the Bayes-optimal OAMP/VAMP (BO-OAMP/VAMP) requires a high-complexity linear minimum mean square error (MMSE) estimator. This prevents OAMP/VAMP from being used in large-scale systems. To address the drawbacks of AMP and BO-OAMP/VAMP, this paper offers a memory AMP (MAMP) framework based on the orthogonality principle, which ensures that estimation errors in MAMP are asymptotically IID Gaussian. To realize the required orthogonality for MAMP, we provide an orthogonalization procedure for the local memory estimators. In addition, we propose a Bayes-optimal MAMP (BO-MAMP), in which a long-memory matched filter is used for interference suppression. The complexity of BO-MAMP is comparable to AMP. To asymptotically characterize the performance of BO-MAMP, a state evolution is derived. The relaxation parameters and damping vector in BO-MAMP are optimized based on state evolution. Most crucially, the state evolution of the optimized BO-MAMP converges to the same fixed point as that of the high-complexity BO-OAMP/VAMP for all right-unitarily-invariant matrices, and achieves the Bayes optimal MSE predicted by the replica method if its state evolution has a unique fixed point. Finally, simulations are provided to verify the theoretical results' validity and accuracy.

cs.IT↗

arXiv2022

Sufficient Statistic Memory Approximate Message Passing

cs.IT↗

arXiv2021

Construction D' Lattices for Power-Constrained Communications

Designs and methods for nested lattice codes using Construction D' lattices for coding and convolutional code lattices for shaping are described. Two encoding methods and a decoding algorithm for Construction D' coding lattices that can be used with shaping lattices for power-constrained channels are given. We construct nested lattice codes with good coding properties, a high shaping gain, and low-complexity encoding and decoding. Convolutional code generator polynomials for Construction A lattices with the greatest shaping gain are given, as a result of an extensive search. It is shown that rate 1/3 convolutional codes provide a more favorable performance-complexity trade-off than rate 1/2 convolutional codes. Tail-biting convolutional codes have higher shaping gain than that of zero-tailed convolutional codes. A design for quasi-cyclic low-density parity-check (QC-LDPC) codes to form Construction D' lattices which have efficient encoding and indexing is presented. The resulting QC-LDPC Construction D' lattices are evaluated using four shaping lattices: the $E_8$ lattice, the $BW_{16}$ lattice, the Leech lattice and our best-found convolutional code lattice, showing a shaping gain of approximately 0.65 dB, 0.86 dB, 1.03 dB and 1.25 dB at dimension 2304.

cs.IT↗

arXiv2021

Memory Approximate Message Passing

Approximate message passing (AMP) is a low-cost iterative parameter-estimation technique for certain high-dimensional linear systems with non-Gaussian distributions. However, AMP only applies to independent identically distributed (IID) transform matrices, but may become unreliable for other matrix ensembles, especially for ill-conditioned ones. To handle this difficulty, orthogonal/vector AMP (OAMP/VAMP) was proposed for general right-unitarily-invariant matrices. However, the Bayes-optimal OAMP/VAMP requires high-complexity linear minimum mean square error estimator. To solve the disadvantages of AMP and OAMP/VAMP, this paper proposes a memory AMP (MAMP), in which a long-memory matched filter is proposed for interference suppression. The complexity of MAMP is comparable to AMP. The asymptotic Gaussianity of estimation errors in MAMP is guaranteed by the orthogonality principle. A state evolution is derived to asymptotically characterize the performance of MAMP. Based on the state evolution, the relaxation parameters and damping vector in MAMP are optimized. For all right-unitarily-invariant matrices, the optimized MAMP converges to OAMP/VAMP, and thus is Bayes-optimal if it has a unique fixed point. Finally, simulations are provided to verify the validity and accuracy of the theoretical results.

cs.IT↗

arXiv2021

Irregularly Clipped Sparse Regression Codes

Recently, it was found that clipping can significantly improve the section error rate (SER) performance of sparse regression (SR) codes if an optimal clipping threshold is chosen. In this paper, we propose irregularly clipped SR codes, where multiple clipping thresholds are applied to symbols according to a distribution, to further improve the SER performance of SR codes. Orthogonal approximate message passing (OAMP) algorithm is used for decoding. Using state evolution, the distribution of irregular clipping thresholds is optimized to minimize the SER of OAMP decoding. As a result, optimized irregularly clipped SR codes achieve a better tradeoff between clipping distortion and noise distortion than regularly clipped SR codes. Numerical results demonstrate that irregularly clipped SR codes achieve 0.4 dB gain in signal-to-noise-ratio (SNR) over regularly clipped SR codes at code length$\,\approx2.5\!\times\! 10^4$ and SER$\,\approx10^{-5}$. We further show that irregularly clipped SR codes are robust over a wide range of code rates.

cs.IT↗

arXiv2021

Encoding and Decoding Construction D' Lattices for Power-Constrained Communications

This paper focuses on the encoding and decoding of Construction D' coding lattices that can be used with shaping lattices for power-constrained channels. Two encoding methods and a decoding algorithm for Construction D' lattices are given. A design of quasi-cyclic low-density parity-check (QC-LDPC) codes to form Construction D' lattices is presented. This allows construction of nested lattice codes which are good for coding, good for shaping, and have low complexity encoding and decoding. Numerical results of using $E_8$, $BW_{16}$ and Leech lattices for shaping a Construction D' lattice indicate that the shaping gains 0.65 dB, 0.86 dB and 1.03 dB are preserved, respectively.

cs.IT↗

arXiv2021

Design of Polar Code Lattices of Small Dimension

Polar code lattices are formed from binary polar codes using Construction D. In this paper, we propose a design technique for finite-dimension polar code lattices. The dimension $n$ and target probability of decoding error are parameters for this design. To select the rates of the Construction D component codes, rather than using the capacity as in past work, we use the explicit finite-length properties of the polar code. Under successive cancellation decoding, density evolution allows choosing code rates that satisfy the equal error probability rule. At an error-rate of $10^{-4}$, a dimension $n=128$ polar code lattice achieves a VNR of 2.5 dB, within 0.2 dB of the best-known BCH code lattice, but with significantly lower decoding complexity.

cs.IT↗

arXiv2020

Opportunistic Cooperation Strategies for Multiple Access Relay Channels with Compute-and-Forward

This paper studies the application of compute-and-forward to multiple access relay channels (MARC). Despite its promising advantage of improving network throughput, it is not straightforward to apply compute-and-forward to the MARC. This paper proposes two efficient cooperation strategies for the MARC with compute-and-forward. Both proposed strategies are opportunistic in the sense that the cooperation between the relay and the destination are performed only when it is needed for the sake of high transmission efficiency. In the first strategy, the relay helps the destination by sending its local optimal integer coefficient vector without taking into account that the resulting integer coefficient matrix at the destination is full rank. In contrast, in the second strategy, the relay forwards its ``best'' coefficient vector that always ensures a full rank coefficient matrix at the destination. Both of the proposed strategies achieve twice the network throughput of the existing strategies at high signal-to-noise power ratio (SNR). They also have lower outage probability, independent of relay placement. Furthermore, the first strategy nearly achieves diversity gain of order two, and the second one achieves exactly the diversity gain of order two, which cannot be achieved by the existing strategies.

cs.IT↗

arXiv2019

Steepest Gradient-Based Orthogonal Precoder For Integer-Forcing MIMO

In this paper, we develop an orthogonal precoding scheme for integer-forcing (IF) linear receivers using the steepest gradient algorithm. Although this scheme can be viewed as a special case of the unitary precoded integer-forcing (UPIF), it has two major advantages. First, the orthogonal precoding outperforms its unitary counterpart in terms of achievable rate, outage probability, and error rate. We verify this advantage via theoretical and numerical analyses. Second, it exhibits lower complexity as the dimension of orthogonal matrices is half that of unitary matrices in the real-valued domain. For finding ``good'' orthogonal precoder matrices, we propose an efficient algorithm based on the steepest gradient algorithm that exploits the geometrical properties of orthogonal matrices as a Lie group. The proposed algorithm has low complexity and can be easily applied to an arbitrary MIMO configuration. We also confirm numerically that the proposed orthogonal precoding outperforms UPIF type II in some scenarios and the X-precoder in high-order QAM schemes, e.g., $64$- and $256$-QAM.

cs.IT↗