arXiv subjects

Rongrong Wang

Publications and source records attributed to Rongrong Wang.

At least 55 records · Page 3Linked to original sources

arXiv2024

Analysis of Deep Image Prior and Exploiting Self-Guidance for Image Reconstruction

The ability of deep image prior (DIP) to recover high-quality images from incomplete or corrupted measurements has made it popular in inverse problems in image restoration and medical imaging including magnetic resonance imaging (MRI). However, conventional DIP suffers from severe overfitting and spectral bias effects. In this work, we first provide an analysis of how DIP recovers information from undersampled imaging measurements by analyzing the training dynamics of the underlying networks in the kernel regime for different architectures. This study sheds light on important underlying properties for DIP-based recovery. Current research suggests that incorporating a reference image as network input can enhance DIP's performance in image reconstruction compared to using random inputs. However, obtaining suitable reference images requires supervision, and raises practical difficulties. In an attempt to overcome this obstacle, we further introduce a self-driven reconstruction process that concurrently optimizes both the network weights and the input while eliminating the need for training data. Our method incorporates a novel denoiser regularization term which enables robust and stable joint estimation of both the network input and reconstructed image. We demonstrate that our self-guided method surpasses both the original DIP and modern supervised methods in terms of MR image reconstruction performance and outperforms previous DIP-based schemes for image inpainting.

cs.CV↗

arXiv2023

PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descent

Fine-tuning pretrained language models (PLMs) for downstream tasks is a large-scale optimization problem, in which the choice of the training algorithm critically determines how well the trained model can generalize to unseen test data, especially in the context of few-shot learning. To achieve good generalization performance and avoid overfitting, techniques such as data augmentation and pruning are often applied. However, adding these regularizations necessitates heavy tuning of the hyperparameters of optimization algorithms, such as the popular Adam optimizer. In this paper, we propose a two-stage fine-tuning method, PAC-tuning, to address this optimization challenge. First, based on PAC-Bayes training, PAC-tuning directly minimizes the PAC-Bayes generalization bound to learn proper parameter distribution. Second, PAC-tuning modifies the gradient by injecting noise with the variance learned in the first stage into the model parameters during training, resulting in a variant of perturbed gradient descent (PGD). In the past, the few-shot scenario posed difficulties for PAC-Bayes training because the PAC-Bayes bound, when applied to large models with limited training data, might not be stringent. Our experimental results across 5 GLUE benchmark tasks demonstrate that PAC-tuning successfully handles the challenges of fine-tuning tasks and outperforms strong baseline methods by a visible margin, further confirming the potential to apply PAC training for any other settings where the Adam optimizer is currently used for training.

cs.LG↗

arXiv2023

Neural Network Approximation of Continuous Functions in High Dimensions with Applications to Inverse Problems

The remarkable successes of neural networks in a huge variety of inverse problems have fueled their adoption in disciplines ranging from medical imaging to seismic analysis over the past decade. However, the high dimensionality of such inverse problems has simultaneously left current theory, which predicts that networks should scale exponentially in the dimension of the problem, unable to explain why the seemingly small networks used in these settings work as well as they do in practice. To reduce this gap between theory and practice, we provide a general method for bounding the complexity required for a neural network to approximate a Hölder (or uniformly) continuous function defined on a high-dimensional set with a low-complexity structure. The approach is based on the observation that the existence of a Johnson-Lindenstrauss embedding $A\in\mathbb{R}^{d\times D}$ of a given high-dimensional set $S\subset\mathbb{R}^D$ into a low dimensional cube $[-M,M]^d$ implies that for any Hölder (or uniformly) continuous function $f:S\to\mathbb{R}^p$, there exists a Hölder (or uniformly) continuous function $g:[-M,M]^d\to\mathbb{R}^p$ such that $g(Ax)=f(x)$ for all $x\in S$. Hence, if one has a neural network which approximates $g:[-M,M]^d\to\mathbb{R}^p$, then a layer can be added that implements the JL embedding $A$ to obtain a neural network that approximates $f:S\to\mathbb{R}^p$. By pairing JL embedding results along with results on approximation of Hölder (or uniformly) continuous functions by neural networks, one then obtains results which bound the complexity required for a neural network to approximate Hölder (or uniformly) continuous functions on high dimensional sets. The end result is a general theoretical framework which can then be used to better explain the observed empirical successes of smaller networks in a wider variety of inverse problems than current theory allows.

stat.ML↗

arXiv2023

An exact $\sinΘ$ formula for matrix perturbation analysis and its applications

In this paper, we establish a useful set of formulae for the $\sinΘ$ distance between the original and the perturbed singular subspaces. These formulae explicitly show that how the perturbation of the original matrix propagates into singular vectors and singular subspaces, thus providing a direct way of analyzing them. Following this, we derive a collection of new results on SVD perturbation related problems, including a tighter bound on the $\ell_{2,\infty}$ norm of the singular vector perturbation errors under Gaussian noise, a new stability analysis of the Principal Component Analysis and an error bound on the singular value thresholding operator. For the latter two, we consider the most general rectangular matrices with full matrix rank.

math.ST↗

arXiv2023

Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descent

It is well known that the finite step-size ($h$) in Gradient Descent (GD) implicitly regularizes solutions to flatter minima. A natural question to ask is "Does the momentum parameter $β$ play a role in implicit regularization in Heavy-ball (H.B) momentum accelerated gradient descent (GD+M)?". To answer this question, first, we show that the discrete H.B momentum update (GD+M) follows a continuous trajectory induced by a modified loss, which consists of an original loss and an implicit regularizer. Then, we show that this implicit regularizer for (GD+M) is stronger than that of (GD) by factor of $(\frac{1+β}{1-β})$, thus explaining why (GD+M) shows better generalization performance and higher test accuracy than (GD). Furthermore, we extend our analysis to the stochastic version of gradient descent with momentum (SGD+M) and characterize the continuous trajectory of the update of (SGD+M) in a pointwise sense. We explore the implicit regularization in (SGD+M) and (GD+M) through a series of experiments validating our theory.

cs.LG↗

arXiv2022

Perturbation of invariant subspaces for ill-conditioned eigensystem

Given a diagonalizable matrix $A$, we study the stability of its invariant subspaces when its matrix of eigenvectors is ill-conditioned. Let $\mathcal{X}_1$ be some invariant subspace of $A$ and $X_1$ be the matrix storing the right eigenvectors that spanned $\mathcal{X}_1$. It is generally believed that when the condition number $κ_2(X_1)$ gets large, the corresponding invariant subspace $\mathcal{X}_1$ will become unstable to perturbation. This paper proves that this is not always the case. Specifically, we show that the growth of $κ_2(X_1)$ alone is not enough to destroy the stability. As a direct application, our result ensures that when $A$ gets closer to a Jordan form, one may still estimate its invariant subspaces from the noisy data stably.

math.NA↗

arXiv2022

Sigma Delta quantization for images

In signal quantization, it is well-known that introducing adaptivity to quantization schemes can improve their stability and accuracy in quantizing bandlimited signals. However, adaptive quantization has only been designed for one-dimensional signals. The contribution of this paper is two-fold: i). we propose the first family of two-dimensional adaptive quantization schemes that maintain the same mathematical and practical merits as their one-dimensional counterparts, and ii). we show that both the traditional 1-dimensional and the new 2-dimensional quantization schemes can effectively quantize signals with jump discontinuities. These results immediately enable the usage of adaptive quantization on images. Under mild conditions, we show that the adaptivity is able to reduce the reconstruction error of images from the presently best $O(\sqrt P)$ to the much smaller $O(\sqrt s)$, where $s$ is the number of jump discontinuities in the image and $P$ ($P\gg s$) is the total number of samples. This $\sqrt{P/s}$-fold error reduction is achieved via applying a total variation norm regularized decoder, whose formulation is inspired by the mathematical super-resolution theory in the field of compressed sensing. Compared to the super-resolution setting, our error reduction is achieved without requiring adjacent spikes/discontinuities to be well-separated, which ensures its broad scope of application. We numerically demonstrate the efficacy of the new scheme on medical and natural images. We observe that for images with small pixel intensity values, the new method can significantly increase image quality over the state-of-the-art method.

cs.IT↗

arXiv2021

Deep Image Matting with Flexible Guidance Input

Image matting is an important computer vision problem. Many existing matting methods require a hand-made trimap to provide auxiliary information, which is very expensive and limits the real world usage. Recently, some trimap-free methods have been proposed, which completely get rid of any user input. However, their performance lag far behind trimap-based methods due to the lack of guidance information. In this paper, we propose a matting method that use Flexible Guidance Input as user hint, which means our method can use trimap, scribblemap or clickmap as guidance information or even work without any guidance input. To achieve this, we propose Progressive Trimap Deformation(PTD) scheme that gradually shrink the area of the foreground and background of the trimap with the training step increases and finally become a scribblemap. To make our network robust to any user scribble and click, we randomly sample points on foreground and background and perform curve fitting. Moreover, we propose Semantic Fusion Module(SFM) which utilize the Feature Pyramid Enhancement Module(FPEM) and Joint Pyramid Upsampling(JPU) in matting task for the first time. The experiments show that our method can achieve state-of-the-art results comparing with existing trimap-based and trimap-free methods.

cs.CV↗

arXiv2021

Capacity Preserving Mapping for High-dimensional Data Visualization

We provide a rigorous mathematical treatment to the crowding issue in data visualization when high dimensional data sets are projected down to low dimensions for visualization. By properly adjusting the capacity of high dimensional balls, our method makes right enough room to prepare for the embedding. A key component of the proposed method is an estimation of the correlation dimension at various scales which reflects the data density variation. The proposed adjustment to the capacity applies to any distance (Euclidean, geodesic, diffusion) and can potentially be used in many existing methods to mitigate the crowding during the dimension reduction. We demonstrate the effectiveness of the new method using synthetic and real datasets.

cs.LG↗

arXiv2021

A dual formulation of wavefield reconstruction inversion for large-scale seismic inversion

Most of the seismic inversion techniques currently proposed focus on robustness with respect to the background model choice or inaccurate physical modeling assumptions, but are not apt to large-scale 3D applications. On the other hand, methods that are computationally feasible for industrial problems, such as full waveform inversion, are notoriously bogged down by local minima and require adequate starting models. We propose a novel solution that is both scalable and less sensitive to starting model or inaccurate physics when compared to full waveform inversion. The method is based on a dual (Lagrangian) reformulation of the classical wavefield reconstruction inversion, whose robustness with respect to local minima is well documented in the literature. However, it is not suited to 3D, as it leverages expensive frequency-domain solvers for the wave equation. The proposed reformulation allows the deployment of state-of-the-art time-domain finite-difference methods, and is computationally mature for industrial scale problems.

physics.geo-ph↗

arXiv2021

On the $\ell^\infty$-norms of the Singular Vectors of Arbitrary Powers of a Difference Matrix with Applications to Sigma-Delta Quantization

Let $\| A \|_{\max} := \max_{i,j} |A_{i,j}|$ denote the maximum magnitude of entries of a given matrix $A$. In this paper we show that $$\max \left\{ \|U_r \|_{\max},\|V_r\|_{\max} \right\} \le \frac{(Cr)^{6r}}{\sqrt{N}},$$ where $U_r$ and $V_r$ are the matrices whose columns are, respectively, the left and right singular vectors of the $r$-th order finite difference matrix $D^{r}$ with $r \geq 2$, and where $D$ is the $N\times N$ finite difference matrix with $1$ on the diagonal, $-1$ on the sub-diagonal, and $0$ elsewhere. Here $C$ is a universal constant that is independent of both $N$ and $r$. Among other things, this establishes that both the right and left singular vectors of such finite difference matrices are Bounded Orthonormal Systems (BOSs) with known upper bounds on their BOS constants, objects of general interest in classical compressive sensing theory. Such finite difference matrices are also fundamental to standard $r^{\rm th}$ order Sigma-Delta quantization schemes more specifically, and as a result the new bounds provided herein on the maximum $\ell^{\infty}$-norms of their $\ell^2$-normalized singular vectors allow for several previous Sigma-Delta quantization results to be generalized and improved.

math.NA↗

arXiv2021

Linear Convergent Decentralized Optimization with Compression

Communication compression has become a key strategy to speed up distributed optimization. However, existing decentralized algorithms with compression mainly focus on compressing DGD-type algorithms. They are unsatisfactory in terms of convergence rate, stability, and the capability to handle heterogeneous data. Motivated by primal-dual algorithms, this paper proposes the first \underline{L}in\underline{EA}r convergent \underline{D}ecentralized algorithm with compression, LEAD. Our theory describes the coupled dynamics of the inexact primal and dual update as well as compression error, and we provide the first consensus error bound in such settings without assuming bounded gradients. Experiments on convex problems validate our theoretical analysis, and empirical study on deep neural nets shows that LEAD is applicable to non-convex problems.

cs.LG↗

arXiv2020

Deep Reinforcement Learning for Tropical Air Free-Cooled Data Center Control

Air free-cooled data centers (DCs) have not existed in the tropical zone due to the unique challenges of year-round high ambient temperature and relative humidity (RH). The increasing availability of servers that can tolerate higher temperatures and RH due to the regulatory bodies' prompts to raise DC temperature setpoints sheds light upon the feasibility of air free-cooled DCs in tropics. However, due to the complex psychrometric dynamics, operating the air free-cooled DC in tropics generally requires adaptive control of supply air condition to maintain the computing performance and reliability of the servers. This paper studies the problem of controlling the supply air temperature and RH in a free-cooled tropical DC below certain thresholds. To achieve the goal, we formulate the control problem as Markov decision processes and apply deep reinforcement learning (DRL) to learn the control policy that minimizes the cooling energy while satisfying the requirements on the supply air temperature and RH. We also develop a constrained DRL solution for performance improvements. Extensive evaluation based on real data traces collected from an air free-cooled testbed and comparisons among the unconstrained and constrained DRL approaches as well as two other baseline approaches show the superior performance of our proposed solutions.

eess.SY↗

arXiv2020

A Hierarchical User Intention-Habit Extract Network for Credit Loan Overdue Risk Detection

More personal consumer loan products are emerging in mobile banking APP. For ease of use, application process is always simple, which means that few application information is requested for user to fill when applying for a loan, which is not conducive to construct users' credit profile. Thus, the simple application process brings huge challenges to the overdue risk detection, as higher overdue rate will result in greater economic losses to the bank. In this paper, we propose a model named HUIHEN (Hierarchical User Intention-Habit Extract Network) that leverages the users' behavior information in mobile banking APP. Due to the diversity of users' behaviors, we divide behavior sequences into sessions according to the time interval, and use the field-aware method to extract the intra-field information of behaviors. Then, we propose a hierarchical network composed of time-aware GRU and user-item-aware GRU to capture users' short-term intentions and users' long-term habits, which can be regarded as a supplement to user profile. The proposed model can improve the accuracy without increasing the complexity of the original online application process. Experimental results demonstrate the superiority of HUIHEN and show that HUIHEN outperforms other state-of-art models on all datasets.

cs.LG↗

arXiv2019

Manifold Denoising by Nonlinear Robust Principal Component Analysis

This paper extends robust principal component analysis (RPCA) to nonlinear manifolds. Suppose that the observed data matrix is the sum of a sparse component and a component drawn from some low dimensional manifold. Is it possible to separate them by using similar ideas as RPCA? Is there any benefit in treating the manifold as a whole as opposed to treating each local region independently? We answer these two questions affirmatively by proposing and analyzing an optimization framework that separates the sparse component from the manifold under noisy data. Theoretical error bounds are provided when the tangent spaces of the manifold satisfy certain incoherence conditions. We also provide a near optimal choice of the tuning parameters for the proposed optimization formulation with the help of a new curvature estimation method. The efficacy of our method is demonstrated on both synthetic and real datasets.

cs.LG↗

arXiv2016

Sigma Delta quantization with Harmonic frames and partial Fourier ensembles

Sigma Delta quantization, a quantization method which first surfaced in the 1960s, has now been used widely in various digital products such as cameras, cell phones, radars, etc. The method samples an input signal at a rate higher than the Nyquist rate, thus achieves great robustness to quantization noise. Compressed Sensing (CS) is a frugal acquisition method that utilizes the possible sparsity of the signals to reduce the required number of samples for a lossless acquisition. One can deem the reduced number as an effective dimensionality of the set of sparse signals and accordingly, define an effective oversampling rate as the ratio between the actual sampling rate and the effective dimensionality. A natural conjecture is that the error of Sigma Delta quantization, previously shown to decay with the vanilla oversampling rate, should now decay with the effective oversampling rate when carried out in the regime of compressed sensing. Confirming this intuition is one of the main goals in this direction. The study of quantization in CS has so far been limited to proving error convergence results for Gaussian and sub-Gaussian sensing matrices, as the number of bits and/or the number of samples grow to infinity. In this paper, we provide a first result for the more realistic Fourier sensing matrices. The major idea is to randomly permute the Fourier samples before feeding them into the quantizer. We show that the random permutation can effectively increase the low frequency power of the measurements, thus enhance the quality of $ΣΔ$ quantization.

cs.IT↗

arXiv2016

From compressed sensing to compressed bit-streams: practical encoders, tractable decoders

Compressed sensing is now established as an effective method for dimension reduction when the underlying signals are sparse or compressible with respect to some suitable basis or frame. One important, yet under-addressed problem regarding the compressive acquisition of analog signals is how to perform quantization. This is directly related to the important issues of how "compressed" compressed sensing is (in terms of the total number of bits one ends up using after acquiring the signal) and ultimately whether compressed sensing can be used to obtain compressed representations of suitable signals. Building on our recent work, we propose a concrete and practicable method for performing "analog-to-information conversion". Following a compressive signal acquisition stage, the proposed method consists of a quantization stage, based on $ΣΔ$ (sigma-delta) quantization, and a subsequent encoding (compression) stage that fits within the framework of compressed sensing seamlessly. We prove that, using this method, we can convert analog compressive samples to compressed digital bitstreams and decode using tractable algorithms based on convex optimization. We prove that the proposed AIC provides a nearly optimal encoding of sparse and compressible signals. Finally, we present numerical experiments illustrating the effectiveness of the proposed analog-to-information converter.

cs.IT↗

arXiv2016

The gap between the null space property and the restricted isometry property

The null space property (NSP) and the restricted isometry property (RIP) are two properties which have received considerable attention in the compressed sensing literature. As the name suggests, NSP is a property that depends solely on the null space of the measurement procedure and as such, any two matrices which have the same null space will have NSP if either one of them does. On the other hand, RIP is a property of the measurement procedure itself, and given an RIP matrix it is straightforward to construct another matrix with the same null space that is not RIP. %Furthermore, RIP is known to imply NSP and therefore RIP is a strictly stronger assumption than NSP. We say a matrix is RIP-NSP if it has the same null space as an RIP matrix. We show that such matrices can provide robust recovery of compressible signals under Basis pursuit which in many applicable settings is comparable to the guarantee that RIP provides. More importantly, we constructively show that the RIP-NSP is stronger than NSP with the aid of this robust recovery result, which shows that RIP is fundamentally stronger than NSP.

cs.IT↗