arXiv subjects

Siyuan Ma

Publications and source records attributed to Siyuan Ma.

68 records · Page 4Linked to original sources

arXiv2022

Sharp decay estimates for massless Dirac fields on a Schwarzschild background

We consider the explicit asymptotic profile of massless Dirac fields on a Schwarzschild background. First, we prove for the spin $s=\pm \frac{1}{2}$ components of the Dirac field a uniform bound of a positive definite energy and an integrated local energy decay estimate from a symmetric hyperbolic wave system. Based on these estimates, we further show that these components have globally pointwise decay $fv^{-3/2-s}τ^{-5/2+s}$ as both an upper and a lower bound outside the black hole, with function $f$ finite and explicitly expressed in terms of the initial data and the coordinates. This establishes the validity of the conjectured Price's law for massless Dirac fields outside a Schwarzschild black hole.

math.AP↗

arXiv2021

Gated Transformer Networks for Multivariate Time Series Classification

Deep learning model (primarily convolutional networks and LSTM) for time series classification has been studied broadly by the community with the wide applications in different domains like healthcare, finance, industrial engineering and IoT. Meanwhile, Transformer Networks recently achieved frontier performance on various natural language processing and computer vision tasks. In this work, we explored a simple extension of the current Transformer Networks with gating, named Gated Transformer Networks (GTN) for the multivariate time series classification problem. With the gating that merges two towers of Transformer which model the channel-wise and step-wise correlations respectively, we show how GTN is naturally and effectively suitable for the multivariate time series classification task. We conduct comprehensive experiments on thirteen dataset with full ablation study. Our results show that GTN is able to achieve competing results with current state-of-the-art deep learning models. We also explored the attention map for the natural interpretability of GTN on time series modeling. Our preliminary results provide a strong baseline for the Transformer Networks on multivariate time series classification task and grounds the foundation for future research.

cs.LG↗

arXiv2020

Uniform energy bound and Morawetz estimate for extreme components of spin fields in the exterior of a slowly rotating Kerr black hole I: Maxwell field

This first part of the series treats the Maxwell equations in the exterior of a slowly rotating Kerr black hole. By performing a first-order differential operator on each extreme Newman-Penrose (N-P) scalar in a Kinnersley tetrad, the resulting equation and the Teukolsky master equation for the extreme N-P component are both in the form of an inhomogeneous \textquotedblleft{spin-weighted Fackerell-Ipser equation\textquotedblright} (SWFIE) and constitute a weakly coupled system. We first prove energy estimate and integrated local energy decay (Morawetz) estimate for this type of inhomogeneous SWFIE following the method in (Dafermos and Rodnianski in Decay for solutions of the wave equation on Kerr exterior spacetimes I-II: the cases $|a|\ll M$ or axisymmetry, 2010, arXiv:1010.5132), and then utilize these estimates to achieve both a uniform bound of a positive definite energy and a Morawetz estimate for the coupled system of each extreme N-P component. The same type of estimates for the regular extreme N-P components defined in the regular Hawking-Hartle tetrad is also proved. The hierarchy here is generalized in our second part (Ma in Uniform energy bound and Morawetz estimates for extreme components of spin fields in the exterior of a slowly rotating Kerr black hole II: linearized gravity, 2017, arXiv:1708.07385) of this series to treat the extreme components of linearized gravity.

gr-qc↗

arXiv2020

Uniform energy bound and Morawetz estimate for extreme components of spin fields in the exterior of a slowly rotating Kerr black hole II: linearized gravity

This second part of the series treats spin $\pm2$ components (or extreme components), that satisfy the Teukolsky master equation, of the linearized gravity in the exterior of a slowly rotating Kerr black hole. For each of these two components, after performing a first-order differential operator once and twice, the resulting equations together with the Teukolsky master equation itself constitute a linear spin-weighted wave system. An energy and Morawetz estimate for spin $\pm 2$ components is proved by treating this system. This is a first step in a joint work (Andersson et al. in Stability for linearized gravity on the Kerr spacetime, arXiv:1903.03859, 2019) in addressing the linear stability of slowly rotating Kerr metrics.

gr-qc↗

arXiv2020

Almost Price's law in Schwarzschild and decay estimates in Kerr for Maxwell field

We consider in this work the asymptotics of a Maxwell field in Schwarzschild and Kerr spacetimes. In any subextremal Kerr spacetime, we show energy and pointwise decay estimates for all components under an assumption of a basic energy and Morawetz estimate for spin $\pm 1$ components. If restricted to slowly rotating Kerr, we utilize the basic energy and Morawetz estimates proven in an earlier work to further improve these decay estimates such that the total power of decay for all components of Maxwell field is $-7/2$. In the end, depending on if the Newman--Penrose constant vanishes or not, we prove almost sharp Price's law decay $τ^{-5+}$ (or $τ^{-4+}$) for Maxwell field and $τ^{-\ell -4+}$ (or $τ^{-\ell -3+}$) for any $\ell$ mode of the field towards a static solution on a Schwarzschild background. All estimates are uniform in the exterior of the black hole.

math.AP↗

arXiv2019

Reconciling modern machine learning practice and the bias-variance trade-off

Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias-variance trade-off, appears to be at odds with the observed behavior of methods used in the modern machine learning practice. The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns. However, in the modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered over-fit, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners. In this paper, we reconcile the classical understanding and the modern practice within a unified performance curve. This "double descent" curve subsumes the textbook U-shaped bias-variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance. We provide evidence for the existence and ubiquity of double descent for a wide spectrum of models and datasets, and we posit a mechanism for its emergence. This connection between the performance and the structure of machine learning models delineates the limits of classical analyses, and has implications for both the theory and practice of machine learning.

stat.ML↗

arXiv2019

Kernel machines that adapt to GPUs for effective large batch training

Modern machine learning models are typically trained using Stochastic Gradient Descent (SGD) on massively parallel computing resources such as GPUs. Increasing mini-batch size is a simple and direct way to utilize the parallel computing capacity. For small batch an increase in batch size results in the proportional reduction in the training time, a phenomenon known as linear scaling. However, increasing batch size beyond a certain value leads to no further improvement in training time. In this paper we develop the first analytical framework that extends linear scaling to match the parallel computing capacity of a resource. The framework is designed for a class of classical kernel machines. It automatically modifies a standard kernel machine to output a mathematically equivalent prediction function, yet allowing for extended linear scaling, i.e., higher effective parallelization and faster training time on given hardware. The resulting algorithms are accurate, principled and very fast. For example, using a single Titan Xp GPU, training on ImageNet with $1.3\times 10^6$ data points and $1000$ labels takes under an hour, while smaller datasets, such as MNIST, take seconds. As the parameters are chosen analytically, based on the theoretical bounds, little tuning beyond selecting the kernel and the kernel parameter is needed, further facilitating the practical use of these methods.

stat.ML↗

arXiv2018

Kernel Machines Beat Deep Neural Networks on Mask-based Single-channel Speech Enhancement

We apply a fast kernel method for mask-based single-channel speech enhancement. Specifically, our method solves a kernel regression problem associated to a non-smooth kernel function (exponential power kernel) with a highly efficient iterative method (EigenPro). Due to the simplicity of this method, its hyper-parameters such as kernel bandwidth can be automatically and efficiently selected using line search with subsamples of training data. We observe an empirical correlation between the regression loss (mean square error) and regular metrics for speech enhancement. This observation justifies our training target and motivates us to achieve lower regression loss by training separate kernel model per frequency subband. We compare our method with the state-of-the-art deep neural networks on mask-based HINT and TIMIT. Experimental results show that our kernel method consistently outperforms deep neural networks while requiring less training time.

cs.LG↗

arXiv2018

On exponential convergence of SGD in non-convex over-parametrized learning

Large over-parametrized models learned via stochastic gradient descent (SGD) methods have become a key element in modern machine learning. Although SGD methods are very effective in practice, most theoretical analyses of SGD suggest slower convergence than what is empirically observed. In our recent work [8] we analyzed how interpolation, common in modern over-parametrized learning, results in exponential convergence of SGD with constant step size for convex loss functions. In this note, we extend those results to a much broader non-convex function class satisfying the Polyak-Lojasiewicz (PL) condition. A number of important non-convex problems in machine learning, including some classes of neural networks, have been recently shown to satisfy the PL condition. We argue that the PL condition provides a relevant and attractive setting for many machine learning problems, particularly in the over-parametrized regime.

math.OC↗

arXiv2018

The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning

In this paper we aim to formally explain the phenomenon of fast convergence of SGD observed in modern machine learning. The key observation is that most modern learning architectures are over-parametrized and are trained to interpolate the data by driving the empirical loss (classification and regression) close to zero. While it is still unclear why these interpolated solutions perform well on test data, we show that these regimes allow for fast convergence of SGD, comparable in number of iterations to full gradient descent. For convex loss functions we obtain an exponential convergence bound for {\it mini-batch} SGD parallel to that for full gradient descent. We show that there is a critical batch size $m^*$ such that: (a) SGD iteration with mini-batch size $m\leq m^*$ is nearly equivalent to $m$ iterations of mini-batch size $1$ (\emph{linear scaling regime}). (b) SGD iteration with mini-batch $m> m^*$ is nearly equivalent to a full gradient descent iteration (\emph{saturation regime}). Moreover, for the quadratic loss, we derive explicit expressions for the optimal mini-batch and step size and explicitly characterize the two regimes above. The critical mini-batch size can be viewed as the limit for effective mini-batch parallelization. It is also nearly independent of the data size, implying $O(n)$ acceleration over GD per unit of computation. We give experimental evidence on real data which closely follows our theoretical analyses. Finally, we show how our results fit in the recent developments in training deep neural networks and discuss connections to adaptive rates for SGD and variance reduction.

cs.LG↗

arXiv2018

To understand deep learning we need to understand kernel learning

Generalization performance of classifiers in deep learning has recently become a subject of intense study. Deep models, typically over-parametrized, tend to fit the training data exactly. Despite this "overfitting", they perform well on test data, a phenomenon not yet fully understood. The first point of our paper is that strong performance of overfitted classifiers is not a unique feature of deep learning. Using six real-world and two synthetic datasets, we establish experimentally that kernel machines trained to have zero classification or near zero regression error perform very well on test data, even when the labels are corrupted with a high level of noise. We proceed to give a lower bound on the norm of zero loss solutions for smooth kernels, showing that they increase nearly exponentially with data size. We point out that this is difficult to reconcile with the existing generalization bounds. Moreover, none of the bounds produce non-trivial results for interpolating solutions. Second, we show experimentally that (non-smooth) Laplacian kernels easily fit random labels, a finding that parallels results for ReLU neural networks. In contrast, fitting noisy data requires many more epochs for smooth Gaussian kernels. Similar performance of overfitted Laplacian and Gaussian classifiers on test, suggests that generalization is tied to the properties of the kernel function rather than the optimization process. Certain key phenomena of deep learning are manifested similarly in kernel methods in the modern "overfitted" regime. The combination of the experimental and theoretical results presented in this paper indicates a need for new theoretical ideas for understanding properties of classical kernel methods. We argue that progress on understanding deep learning will be difficult until more tractable "shallow" kernel methods are better understood.

stat.ML↗

arXiv2017

Diving into the shallows: a computational perspective on large-scale shallow learning

In this paper we first identify a basic limitation in gradient descent-based optimization methods when used in conjunctions with smooth kernels. An analysis based on the spectral properties of the kernel demonstrates that only a vanishingly small portion of the function space is reachable after a polynomial number of gradient descent iterations. This lack of approximating power drastically limits gradient descent for a fixed computational budget leading to serious over-regularization/underfitting. The issue is purely algorithmic, persisting even in the limit of infinite data. To address this shortcoming in practice, we introduce EigenPro iteration, based on a preconditioning scheme using a small number of approximately computed eigenvectors. It can also be viewed as learning a new kernel optimized for gradient descent. It turns out that injecting this small (computationally inexpensive and SGD-compatible) amount of approximate second-order information leads to major improvements in convergence. For large data, this translates into significant performance boost over the standard kernel methods. In particular, we are able to consistently match or improve the state-of-the-art results recently reported in the literature with a small fraction of their computational budget. Finally, we feel that these results show a need for a broader computational perspective on modern large-scale learning to complement more traditional statistical and convergence analyses. In particular, many phenomena of large-scale high-dimensional inference are best understood in terms of optimization on infinite dimensional Hilbert spaces, where standard algorithms can sometimes have properties at odds with finite-dimensional intuition. A systematic analysis concentrating on the approximation power of such algorithms within a budget of computation may lead to progress both in theory and practice.

stat.ML↗

arXiv2016

Mode stability on the real axis

A generalization of the mode stability result of Whiting (1989) for the Teukolsky equation is proved for the case of real frequencies. The main result of the paper states that a separated solution of the Teukolsky equation governing massless test fields on the Kerr spacetime, which is purely outgoing at infinity, and purely ingoing at the horizon, must vanish. This has the consequence, that for real frequencies, there are linearly independent fundamental solutions of the radial Teukolsky equation which are purely ingoing at the horizon, and purely outgoing at infinity, respectively. This fact yields a representation formula for solutions of the inhomogenous Teukolsky equation.

gr-qc↗

arXiv2016

Re-enabling high-speed caching for LSM-trees

LSM-tree has been widely used in cloud computing systems by Google, Facebook, and Amazon, to achieve high performance for write-intensive workloads. However, in LSM-tree, random key-value queries can experience long latency and low throughput due to the interference from the compaction, a basic operation in the algorithm, to caching. LSM-tree relies on frequent compaction operations to merge data into a sorted structure. After a compaction, the original data are reorganized and written to other locations on the disk. As a result, the cached data are invalidated since their referencing addresses are changed, causing serious performance degradations. We propose dLSM in order to re-enable high-speed caching during intensive writes. dLSM is an LSM-tree with a compaction buffer on the disk, working as a cushion to minimize the cache invalidation caused by compactions. The compaction buffer maintains a series of snapshots of the frequently compacted data, which represent a consistent view of the corresponding data in the underlying LSM-tree. Being updated in a much lower rate than that of compactions, data in the compaction buffer are almost stationary. In dLSM, an object is referenced by the disk address of the corresponding block either in the compaction buffer for frequently compacted data, or in the underlying LSM-tree for infrequently compacted data. Thus, hot objects can be effectively kept in the cache without harmful invalidations. With the help of a small on-disk compaction buffer, dLSM achieves a high query performance by enabling effective caching, while retaining all merits of LSM-tree for write-intensive data processing. We have implemented dLSM based on LevelDB. Our evaluations show that with a standard DRAM cache, dLSM can achieve 5--8x performance improvement over LSM with the same cache on HDD storage.

cs.DS↗