arXiv subjects

Michael Ng

Publications and source records attributed to Michael Ng.

15 recordsLinked to original sources

arXiv2026

Hardware-accelerated Aggregation: Unification and Specialization

The high efficiency of domain-specific hardware has sparked substantial interest in adopting accelerators in data analytics systems. Among many choices, GPUs and FPGAs thrived as two popular solutions due to their prevalent deployments in cloud data centers. This paper investigates hardware acceleration solutions for aggregation, a critical data analytics operation. Specifically, we implement aggregation with a unified hardware acceleration framework, which trades efficiency for ease of programming and portability, and then further develop hardware-specific optimizations. We evaluate these solutions on three recent computing hardware platforms: a CPU, a GPU, and an FPGA, with metrics that cover both the performance and energy consumption of on-device and end-to-end processing.

cs.DC↗

arXiv2026

QuatIca: Advanced Numerical Linear Algebra and Optimization for Quaternionic Matrices in Python

Quaternion-valued representations provide a convenient way to model coupled multi-channel signals (e.g., RGB imagery, polarization data, vector fields, and multi-detector time series). Yet practical and numerically reliable software support remains far less mature than those based on the real/complex setting. Here, we present QuatIca, an open-source Python library for quaternion numerical linear algebra and optimization, designed for both research prototyping and reproducible experimentation. QuatIca provides core quaternion matrix operations and norms; dense decompositions and reductions (QR, LU, Q-SVD, eigendecomposition, Hessenberg/tridiagonal reduction, Cholesky decomposition, and Schur helpers); iterative solvers including quaternion GMRES (with preconditioning) and Newton-Schulz pseudoinverse schemes; and domain-focused routines for signal and image processing such as quaternion Tikhonov restoration. The library also includes OptiQ, which solves quaternion Hermitian semidefinite programs using log-det barrier Newton methods with $\mu$-continuation. We highlight design choices that preserve quaternion structure, and we provide end-to-end demonstrations including quaternion image deblurring, Lorenz-attractor filtering, and quaternion image completion. QuatIca is distributed via PyPI and accompanied by open-source development on GitHub and continuously deployed documentation with runnable tutorials.

math.NA↗

arXiv2025

A Provably-Correct and Robust Convex Model for Smooth Separable NMF

Nonnegative matrix factorization (NMF) is a linear dimensionality reduction technique for nonnegative data, with applications such as hyperspectral unmixing and topic modeling. NMF is a difficult problem in general (NP-hard), and its solutions are typically not unique. To address these two issues, additional constraints or assumptions are often used. In particular, separability assumes that the basis vectors in the NMF are equal to some columns of the input matrix. In that case, the problem is referred to as separable NMF (SNMF) and can be solved in polynomial-time with robustness guarantees, while identifying a unique solution. However, in real-world scenarios, due to noise or variability, multiple data points may lie near the basis vectors, which SNMF does not leverage. In this work, we rely on the smooth separability assumption, which assumes that each basis vector is close to multiple data points. We explore the properties of the corresponding problem, referred to as smooth SNMF (SSNMF), and examine how it relates to SNMF and orthogonal NMF. We then propose a convex model for SSNMF and show that it provably recovers the sought-after factors, even in the presence of noise. We finally adapt an existing fast gradient method to solve this convex model for SSNMF, and show that it compares favorably with state-of-the-art methods on both synthetic and hyperspectral datasets.

math.NA↗

arXiv2024

DAPE V2: Process Attention Score as Feature Map for Length Extrapolation

The attention mechanism is a fundamental component of the Transformer model, contributing to interactions among distinct tokens, in contrast to earlier feed-forward neural networks. In general, the attention scores are determined simply by the key-query products. However, this work's occasional trial (combining DAPE and NoPE) of including additional MLPs on attention scores without position encoding indicates that the classical key-query multiplication may limit the performance of Transformers. In this work, we conceptualize attention as a feature map and apply the convolution operator (for neighboring attention scores across different heads) to mimic the processing methods in computer vision. Specifically, the main contribution of this paper is identifying and interpreting the Transformer length extrapolation problem as a result of the limited expressiveness of the naive query and key dot product, and we successfully translate the length extrapolation issue into a well-understood feature map processing problem. The novel insight, which can be adapted to various attention-related models, reveals that the current Transformer architecture has the potential for further evolution. Extensive experiments demonstrate that treating attention as a feature map and applying convolution as a processing method significantly enhances Transformer performance.

cs.CL↗

arXiv2024

Hyperspectral and multispectral image fusion with arbitrary resolution through self-supervised representations

The fusion of a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) has emerged as an effective technique for achieving HSI super-resolution (SR). Previous studies have mainly concentrated on estimating the posterior distribution of the latent high-resolution hyperspectral image (HR-HSI), leveraging an appropriate image prior and likelihood computed from the discrepancy between the latent HSI and observed images. Low rankness stands out for preserving latent HSI characteristics through matrix factorization among the various priors. However, the primary limitation in previous studies lies in the generalization of a fusion model with fixed resolution scales, which necessitates retraining whenever output resolutions are changed. To overcome this limitation, we propose a novel continuous low-rank factorization (CLoRF) by integrating two neural representations into the matrix factorization, capturing spatial and spectral information, respectively. This approach enables us to harness both the low rankness from the matrix factorization and the continuity from neural representation in a self-supervised manner.Theoretically, we prove the low-rank property and Lipschitz continuity in the proposed continuous low-rank factorization. Experimentally, our method significantly surpasses existing techniques and achieves user-desired resolutions without the need for neural network retraining. Code is available at https://github.com/wangting1907/CLoRF-Fusion.

cs.CV↗

arXiv2024

DAPE: Data-Adaptive Positional Encoding for Length Extrapolation

Positional encoding plays a crucial role in transformers, significantly impacting model performance and length generalization. Prior research has introduced absolute positional encoding (APE) and relative positional encoding (RPE) to distinguish token positions in given sequences. However, both APE and RPE remain fixed after model training regardless of input data, limiting their adaptability and flexibility. Hence, we expect that the desired positional encoding should be data-adaptive and can be dynamically adjusted with the given attention. In this paper, we propose a Data-Adaptive Positional Encoding (DAPE) method, which dynamically and semantically adjusts based on input context and learned fixed priors. Experimental validation on real-world datasets (Arxiv, Books3, and CHE) demonstrates that DAPE enhances model performances in terms of trained length and length generalization, where the improvements are statistically significant. The model visualization suggests that our model can keep both local and anti-local information. Finally, we successfully train the model on sequence length 128 and achieve better performance at evaluation sequence length 8192, compared with other static positional encoding methods, revealing the benefit of the adaptive positional encoding method.

cs.CL↗

arXiv2023

Stability and Generalization of Hypergraph Collaborative Networks

Graph neural networks have been shown to be very effective in utilizing pairwise relationships across samples. Recently, there have been several successful proposals to generalize graph neural networks to hypergraph neural networks to exploit more complex relationships. In particular, the hypergraph collaborative networks yield superior results compared to other hypergraph neural networks for various semi-supervised learning tasks. The collaborative network can provide high quality vertex embeddings and hyperedge embeddings together by formulating them as a joint optimization problem and by using their consistency in reconstructing the given hypergraph. In this paper, we aim to establish the algorithmic stability of the core layer of the collaborative network and provide generalization guarantees. The analysis sheds light on the design of hypergraph filters in collaborative networks, for instance, how the data and hypergraph filters should be scaled to achieve uniform stability of the learning process. Some experimental results on real-world datasets are presented to illustrate the theory.

cs.LG↗

arXiv2021

A Generalized Framework for Edge-preserving and Structure-preserving Image Smoothing

Image smoothing is a fundamental procedure in applications of both computer vision and graphics. The required smoothing properties can be different or even contradictive among different tasks. Nevertheless, the inherent smoothing nature of one smoothing operator is usually fixed and thus cannot meet the various requirements of different applications. In this paper, we first introduce the truncated Huber penalty function which shows strong flexibility under different parameter settings. A generalized framework is then proposed with the introduced truncated Huber penalty function. When combined with its strong flexibility, our framework is able to achieve diverse smoothing natures where contradictive smoothing behaviors can even be achieved. It can also yield the smoothing behavior that can seldom be achieved by previous methods, and superior performance is thus achieved in challenging cases. These together enable our framework capable of a range of applications and able to outperform the state-of-the-art approaches in several tasks, such as image detail enhancement, clip-art compression artifacts removal, guided depth map restoration, image texture removal, etc. In addition, an efficient numerical solution is provided and its convergence is theoretically guaranteed even the optimization framework is non-convex and non-smooth. A simple yet effective approach is further proposed to reduce the computational cost of our method while maintaining its performance. The effectiveness and superior performance of our approach are validated through comprehensive experiments in a range of applications. Our code is available at https://github.com/wliusjtu/Generalized-Smoothing-Framework.

cs.CV↗

arXiv2020

Deep Plug-and-play Prior for Low-rank Tensor Completion

Multi-dimensional images, such as color images and multi-spectral images, are highly correlated and contain abundant spatial and spectral information. However, real-world multi-dimensional images are usually corrupted by missing entries. By integrating deterministic low-rankness prior to the data-driven deep prior, we suggest a novel regularized tensor completion model for multi-dimensional image completion. In the objective function, we adopt the newly emerged tensor nuclear norm (TNN) to characterize the global low-rankness prior of the multi-dimensional images. We also formulate an implicit regularizer by plugging into a denoising neural network (termed as deep denoiser), which is convinced to express the deep image prior learned from a large number of natural images. The resulting model can be solved by the alternating directional method of multipliers algorithm under the plug-and-play (PnP) framework. Experimental results on color images, videos, and multi-spectral images demonstrate that the proposed method can recover both the global structure and fine details very well and achieve superior performance over competing methods in terms of quality metrics and visual effects.

cs.CV↗

arXiv2019

Tucker Decomposition Network: Expressive Power and Comparison

Deep neural networks have achieved a great success in solving many machine learning and computer vision problems. The main contribution of this paper is to develop a deep network based on Tucker tensor decomposition, and analyze its expressive power. It is shown that the expressiveness of Tucker network is more powerful than that of shallow network. In general, it is required to use an exponential number of nodes in a shallow network in order to represent a Tucker network. Experimental results are also given to compare the performance of the proposed Tucker network with hierarchical tensor network and shallow network, and demonstrate the usefulness of Tucker network in image classification problems.

cs.LG↗

arXiv2019

3D Point Cloud Denoising using Graph Laplacian Regularization of a Low Dimensional Manifold Model

3D point cloud - a new signal representation of volumetric objects - is a discrete collection of triples marking exterior object surface locations in 3D space. Conventional imperfect acquisition processes of 3D point cloud - e.g., stereo-matching from multiple viewpoint images or depth data acquired directly from active light sensors - imply non-negligible noise in the data. In this paper, we adopt a previously proposed low-dimensional manifold model for the surface patches in the point cloud and seek self-similar patches to denoise them simultaneously using the patch manifold prior. Due to discrete observations of the patches on the manifold, we approximate the manifold dimension computation defined in the continuous domain with a patch-based graph Laplacian regularizer and propose a new discrete patch distance measure to quantify the similarity between two same-sized surface patches for graph construction that is robust to noise. We show that our graph Laplacian regularizer has a natural graph spectral interpretation, and has desirable numerical stability properties via eigenanalysis. Extensive simulation results show that our proposed denoising scheme can outperform state-of-the-art methods in objective metrics and can better preserve visually salient structural features like edges.

cs.CV↗

arXiv2019

A Fast Algorithm for Cosine Transform Based Tensor Singular Value Decomposition

Recently, there has been a lot of research into tensor singular value decomposition (t-SVD) by using discrete Fourier transform (DFT) matrix. The main aims of this paper are to propose and study tensor singular value decomposition based on the discrete cosine transform (DCT) matrix. The advantages of using DCT are that (i) the complex arithmetic is not involved in the cosine transform based tensor singular value decomposition, so the computational cost required can be saved; (ii) the intrinsic reflexive boundary condition along the tubes in the third dimension of tensors is employed, so its performance would be better than that by using the periodic boundary condition in DFT. We demonstrate that the tensor product between two tensors by using DCT can be equivalent to the multiplication between a block Toeplitz-plus-Hankel matrix and a block vector. Numerical examples of low-rank tensor completion are further given to illustrate that the efficiency by using DCT is two times faster than that by using DFT and also the errors of video and multispectral image completion by using DCT are smaller than those by using DFT.

cs.CV↗

arXiv2018

Graph Spectral Image Processing

Recent advent of graph signal processing (GSP) has spurred intensive studies of signals that live naturally on irregular data kernels described by graphs (e.g., social networks, wireless sensor networks). Though a digital image contains pixels that reside on a regularly sampled 2D grid, if one can design an appropriate underlying graph connecting pixels with weights that reflect the image structure, then one can interpret the image (or image patch) as a signal on a graph, and apply GSP tools for processing and analysis of the signal in graph spectral domain. In this article, we overview recent graph spectral techniques in GSP specifically for image / video processing. The topics covered include image compression, image restoration, image filtering and image segmentation.

eess.IV↗

arXiv2015

Robust data assimilation using $L_1$ and Huber norms

Data assimilation is the process to fuse information from priors, observations of nature, and numerical models, in order to obtain best estimates of the parameters or state of a physical system of interest. Presence of large errors in some observational data, e.g., data collected from a faulty instrument, negatively affect the quality of the overall assimilation results. This work develops a systematic framework for robust data assimilation. The new algorithms continue to produce good analyses in the presence of observation outliers. The approach is based on replacing the traditional $Ł_2$ norm formulation of data assimilation problems with formulations based on $Ł_1$ and Huber norms. Numerical experiments using the Lorenz-96 and the shallow water on the sphere models illustrate how the new algorithms outperform traditional data assimilation approaches in the presence of data outliers.

math.NA↗

arXiv2010

Minimax Current Density Coil Design

'Coil design' is an inverse problem in which arrangements of wire are designed to generate a prescribed magnetic field when energized with electric current. The design of gradient and shim coils for magnetic resonance imaging (MRI) are important examples of coil design. The magnetic fields that these coils generate are usually required to be both strong and accurate. Other electromagnetic properties of the coils, such as inductance, may be considered in the design process, which becomes an optimization problem. The maximum current density is additionally optimized in this work and the resultant coils are investigated for performance and practicality. Coils with minimax current density were found to exhibit maximally spread wires and may help disperse localized regions of Joule heating. They also produce the highest possible magnetic field strength per unit current for any given surface and wire size. Three different flavours of boundary element method that employ different basis functions (triangular elements with uniform current, cylindrical elements with sinusoidal current and conic section elements with sinusoidal-uniform current) were used with this approach to illustrate its generality.

physics.class-ph↗