Searcharxiv⌕ Search

arXiv subjects

Ruixiang Zhang

Publications and source records attributed to Ruixiang Zhang.

At least 73 records · Page 4Linked to original sources
arXiv2021

An Attention Free Transformer

We introduce Attention Free Transformer (AFT), an efficient variant of Transformers that eliminates the need for dot product self attention. In an AFT layer, the key and value are first combined with a set of learned position biases, the result of which is multiplied with the query in an element-wise fashion. This new operation has a memory complexity linear w.r.t. both the context size and the dimension of features, making it compatible to both large input and model sizes. We also introduce AFT-local and AFT-conv, two model variants that take advantage of the idea of locality and spatial weight sharing while maintaining global connectivity. We conduct extensive experiments on two autoregressive modeling tasks (CIFAR10 and Enwik8) as well as an image recognition task (ImageNet-1K classification). We show that AFT demonstrates competitive performance on all the benchmarks, while providing excellent efficiency at the same time.

cs.LG↗
arXiv2021

Quantitative Hilbert irreducibility and almost prime values of polynomial discriminants

We study two polynomial counting questions in arithmetic statistics via a combination of Fourier analytic and arithmetic methods. First, we obtain new quantitative forms of Hilbert's Irreducibility Theorem for degree $n$ polynomials $f$ with $\mathrm{Gal}(f) \subseteq A_n$. We study this both for monic polynomials and non-monic polynomials. Second, we study lower bounds on the number of degree $n$ monic polynomials with almost prime discriminants, as well as the closely related problem of lower bounds on the number of degree $n$ number fields with almost prime discriminants.

math.NT↗
arXiv2021

Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling

We show that the sum of the implicit generator log-density $\log p_g$ of a GAN with the logit score of the discriminator defines an energy function which yields the true data density when the generator is imperfect but the discriminator is optimal, thus making it possible to improve on the typical generator (with implicit density $p_g$). To make that practical, we show that sampling from this modified density can be achieved by sampling in latent space according to an energy-based model induced by the sum of the latent prior log-density and the discriminator output score. This can be achieved by running a Langevin MCMC in latent space and then applying the generator function, which we call Discriminator Driven Latent Sampling~(DDLS). We show that DDLS is highly efficient compared to previous methods which work in the high-dimensional pixel space and can be applied to improve on previously trained GANs of many types. We evaluate DDLS on both synthetic and real-world datasets qualitatively and quantitatively. On CIFAR-10, DDLS substantially improves the Inception Score of an off-the-shelf pre-trained SN-GAN~\citep{sngan} from $8.22$ to $9.09$ which is even comparable to the class-conditional BigGAN~\citep{biggan} model. This achieves a new state-of-the-art in unconditional image synthesis setting without introducing extra parameters or additional training.

cs.LG↗
arXiv2021

The Bochner-Riesz problem: an old approach revisited

We show that the recent techniques developed to study the Fourier restriction problem apply equally well to the Bochner-Riesz problem. This is achieved via applying a pseudo-conformal transformation and a two-parameter induction-on-scales argument. As a consequence, we improve the Bochner-Riesz problem to the best known range of the Fourier restriction problem in all high dimensions.

math.CA↗
arXiv2021

An improved result for Falconer's distance set problem in even dimensions

We show that if compact set $E\subset \mathbb{R}^d$ has Hausdorff dimension larger than $\frac{d}{2}+\frac{1}{4}$, where $d\geq 4$ is an even integer, then the distance set of $E$ has positive Lebesgue measure. This improves the previously best known result towards Falconer's distance set conjecture in even dimensions.

math.CA↗
arXiv2021

RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation

Point clouds can be represented in many forms (views), typically, point-based sets, voxel-based cells or range-based images(i.e., panoramic view). The point-based view is geometrically accurate, but it is disordered, which makes it difficult to find local neighbors efficiently. The voxel-based view is regular, but sparse, and computation grows cubically when voxel resolution increases. The range-based view is regular and generally dense, however spherical projection makes physical dimensions distorted. Both voxel- and range-based views suffer from quantization loss, especially for voxels when facing large-scale scenes. In order to utilize different view's advantages and alleviate their own shortcomings in fine-grained segmentation task, we propose a novel range-point-voxel fusion network, namely RPVNet. In this network, we devise a deep fusion framework with multiple and mutual information interactions among these three views and propose a gated fusion module (termed as GFM), which can adaptively merge the three features based on concurrent inputs. Moreover, the proposed RPV interaction mechanism is highly efficient, and we summarize it into a more general formulation. By leveraging this efficient interaction and relatively lower voxel resolution, our method is also proved to be more efficient. Finally, we evaluated the proposed model on two large-scale datasets, i.e., SemanticKITTI and nuScenes, and it shows state-of-the-art performance on both of them. Note that, our method currently ranks 1st on SemanticKITTI leaderboard without any extra tricks.

cs.CV↗
arXiv2021

Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models

AI Safety is a major concern in many deep learning applications such as autonomous driving. Given a trained deep learning model, an important natural problem is how to reliably verify the model's prediction. In this paper, we propose a novel framework -- deep verifier networks (DVN) to verify the inputs and outputs of deep discriminative models with deep generative models. Our proposed model is based on conditional variational auto-encoders with disentanglement constraints. We give both intuitive and theoretical justifications of the model. Our verifier network is trained independently with the prediction model, which eliminates the need of retraining the verifier network for a new model. We test the verifier network on out-of-distribution detection and adversarial example detection problems, as well as anomaly detection problems in structured prediction tasks such as image caption generation. We achieve state-of-the-art results in all of these problems.

cs.CV↗
arXiv2020

Improving unsupervised anomaly localization by applying multi-scale memories to autoencoders

Autoencoder and its variants have been widely applicated in anomaly detection.The previous work memory-augmented deep autoencoder proposed memorizing normality to detect anomaly, however it neglects the feature discrepancy between different resolution scales, therefore we introduce multi-scale memories to record scale-specific features and multi-scale attention fuser between the encoding and decoding module of the autoencoder for anomaly detection, namely MMAE.MMAE updates slots at corresponding resolution scale as prototype features during unsupervised learning. For anomaly detection, we accomplish anomaly removal by replacing the original encoded image features at each scale with most relevant prototype features,and fuse these features before feeding to the decoding module to reconstruct image. Experimental results on various datasets testify that our MMAE successfully removes anomalies at different scales and performs favorably on several datasets compared to similar reconstruction-based methods.

cs.CV↗
arXiv2020

Learning Structured Latent Factors from Dependent Data:A Generative Model Framework from Information-Theoretic Perspective

Learning controllable and generalizable representation of multivariate data with desired structural properties remains a fundamental problem in machine learning. In this paper, we present a novel framework for learning generative models with various underlying structures in the latent space. We represent the inductive bias in the form of mask variables to model the dependency structure in the graphical model and extend the theory of multivariate information bottleneck to enforce it. Our model provides a principled approach to learn a set of semantically meaningful latent factors that reflect various types of desired structures like capturing correlation or encoding invariance, while also offering the flexibility to automatically estimate the dependency structure from data. We show that our framework unifies many existing generative models and can be applied to a variety of tasks including multi-modal data modeling, algorithmic fairness, and invariant risk minimization.

cs.LG↗
arXiv2020

Perceptual Generative Autoencoders

Modern generative models are usually designed to match target distributions directly in the data space, where the intrinsic dimension of data can be much lower than the ambient dimension. We argue that this discrepancy may contribute to the difficulties in training generative models. We therefore propose to map both the generated and target distributions to a latent space using the encoder of a standard autoencoder, and train the generator (or decoder) to match the target distribution in the latent space. Specifically, we enforce the consistency in both the data space and the latent space with theoretically justified data and latent reconstruction losses. The resulting generative model, which we call a perceptual generative autoencoder (PGA), is then trained with a maximum likelihood or variational autoencoder (VAE) objective. With maximum likelihood, PGAs generalize the idea of reversible generative models to unrestricted neural network architectures and arbitrary number of latent dimensions. When combined with VAEs, PGAs substantially improve over the baseline VAEs in terms of sample quality. Compared to other autoencoder-based generative models using simple priors, PGAs achieve state-of-the-art FID scores on CIFAR-10 and CelebA.

cs.LG↗
arXiv2020

Polynomial Blow-up Upper Bounds for the Einstein-scalar field System Under Spherical Symmetry

For general gravitational collapse, inside the black-hole region, singularities $(r=0)$ may arise. In this article, we aim to answer how strong these singularities could be. We analyse the behaviours of various geometric quantities. In particular, we show that in the most singular scenario, the Kretschmann scalar obeys polynomial blow-up upper bounds $O(1/r^N)$. This improves previously best-known double-exponential upper bounds $O\big(\exp\exp(1/r)\big)$. Our result is sharp in the sense that there are known examples showing that no sub-polynomial upper bound could hold. Finally we do a case study on perturbations of the Schwarzschild solution.

math.AP↗
arXiv2019

Improved bounds for the Kakeya maximal conjecture in higher dimensions

We adapt Guth's polynomial partitioning argument for the Fourier restriction problem to the context of the Kakeya problem. By writing out the induction argument as a recursive algorithm, additional multiscale geometric information is made available. To take advantage of this, we prove that direction-separated tubes satisfy a multiscale version of the polynomial Wolff axioms. Altogether, this yields improved bounds for the Kakeya maximal conjecture in $\mathbb{R}^n$ with $n=5$ or $n\ge 7$ and improved bounds for the Kakeya set conjecture for an infinite sequence of dimensions.

math.CA↗
arXiv2019

Sharp $L^2$ estimate of Schrödinger maximal function in higher dimensions

We show that, for $n\geq 3$, $\lim_{t \to 0} e^{itΔ}f(x) = f(x)$ holds almost everywhere for all $f \in H^s (\mathbb{R}^n)$ provided that $s>\frac{n}{2(n+1)}$. Due to a counterexample by Bourgain, up to the endpoint, this result is sharp and fully resolves a problem raised by Carleson. Our main theorem is a fractal $L^2$ restriction estimate, which also gives improved results on the size of divergence set of Schrödinger solutions, the Falconer distance set problem and the spherical average Fourier decay rates of fractal measures. The key ingredients of the proof include multilinear Kakeya estimates, decoupling and induction on scales.

math.CA↗
arXiv2018

Fractal Uncertainty Principle with Explicit Exponent

We prove an explicit formula for the dependence of the exponent in the fractal uncertainty principle of Bourgain-Dyatlov on the dimension and on the regularity constant for the regular set. In particular, this implies an explicit essential spectral gap for convex co-compact hyperbolic surfaces when the Hausdorff dimension of the limit set is close to 1.

math.CA↗
arXiv2018

Pointwise convergence of Schrödinger solutions and multilinear refined Strichartz estimates

We obtain partial improvement toward the pointwise convergence problem of Schrödinger solutions, in the general setting of fractal measure. In particular, we show that, for $n\geq 3$, $\lim_{t \to 0} e^{itΔ}f(x) = f(x)$ almost everywhere with respect to Lebesgue measure for all $f \in H^s (\mathbb{R}^n)$ provided that $s>(n+1)/2(n+2)$. The proof uses linear refined Strichartz estimates. We also prove a multilinear refined Strichartz using decoupling and multilinear Kakeya.

math.CA↗