Searcharxiv⌕ Search

arXiv subjects

Zhiyuan Zhang

Publications and source records attributed to Zhiyuan Zhang.

At least 181 records · Page 10Linked to original sources
arXiv2020

Is the Skip Connection Provable to Reform the Neural Network Loss Landscape?

The residual network is now one of the most effective structures in deep learning, which utilizes the skip connections to ``guarantee" the performance will not get worse. However, the non-convexity of the neural network makes it unclear whether the skip connections do provably improve the learning ability since the nonlinearity may create many local minima. In some previous works \cite{freeman2016topology}, it is shown that despite the non-convexity, the loss landscape of the two-layer ReLU network has good properties when the number $m$ of hidden nodes is very large. In this paper, we follow this line to study the topology (sub-level sets) of the loss landscape of deep ReLU neural networks with a skip connection and theoretically prove that the skip connection network inherits the good properties of the two-layer network and skip connections can help to control the connectedness of the sub-level sets, such that any local minima worse than the global minima of some two-layer ReLU network will be very ``shallow". The ``depth" of these local minima are at most $O(m^{(η-1)/n})$, where $n$ is the input dimension, $η<1$. This provides a theoretical explanation for the effectiveness of the skip connection in deep learning.

cs.LG↗
arXiv2020

$C^{r}-$prevalence of stable ergodicity for a class of partially hyperbolic systems

We prove that for $r \in \mathbb{N}_{\geq 2} \cup \{\infty\}$, for any dynamically coherent, center bunched and strongly pinched volume preserving $C^r$ partially hyperbolic diffeomorphism $f \colon X \to X$, if either (1) its center foliation is uniformly compact, or (2) its center-stable and center-unstable foliations are of class $C^1$, then there exists a $C^1$-open neighbourhood of $f$ in ${\rm Diff}^r(X,\mathrm{Vol})$, in which stable ergodicity is $C^r$-prevalent in Kolmogorov's sense. In particular, we verify Pugh-Shub's stable ergodicity conjecture in this region. This also provides the first result that verifies the prevalence of stable ergodicity in the measure-theoretical sense. Our theorem applies to a large class of algebraic systems. As applications, we give affirmative answers in the strongly pinched region to: 1. an open question of Pugh-Shub in \cite{PS}; 2. a generic version of an open question of Hirsch-Pugh-Shub in \cite{HPS}; and 3. a generic version of an open question of Pugh-Shub in \cite{HPS}.

math.DS↗
arXiv2019

Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection

Self-attention based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks. Self-attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context. To tackle the problem, we propose a novel model called \textbf{Explicit Sparse Transformer}. Explicit Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments. Extensive experimental results on a series of natural language processing and computer vision tasks, including neural machine translation, image captioning, and language modeling, all demonstrate the advantages of Explicit Sparse Transformer in model performance. We also show that our proposed sparse attention method achieves comparable or better results than the previous sparse attention method, but significantly reduces training and testing time. For example, the inference speed is twice that of sparsemax in Transformer model. Code will be available at \url{https://github.com/lancopku/Explicit-Sparse-Transformer}

cs.CL↗
arXiv2019

MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning

In sequence to sequence learning, the self-attention mechanism proves to be highly effective, and achieves significant improvements in many tasks. However, the self-attention mechanism is not without its own flaws. Although self-attention can model extremely long dependencies, the attention in deep layers tends to overconcentrate on a single token, leading to insufficient use of local information and difficultly in representing long sequences. In this work, we explore parallel multi-scale representation learning on sequence data, striving to capture both long-range and short-range language structures. To this end, we propose the Parallel MUlti-Scale attEntion (MUSE) and MUSE-simple. MUSE-simple contains the basic idea of parallel multi-scale sequence representation learning, and it encodes the sequence in parallel, in terms of different scales with the help from self-attention, and pointwise transformation. MUSE builds on MUSE-simple and explores combining convolution and self-attention for learning sequence representations from more different scales. We focus on machine translation and the proposed approach achieves substantial performance improvements over Transformer, especially on long sequences. More importantly, we find that although conceptually simple, its success in practice requires intricate considerations, and the multi-scale attention must build on unified semantic space. Under common setting, the proposed model achieves substantial performance and outperforms all previous models on three main machine translation tasks. In addition, MUSE has potential for accelerating inference due to its parallelism. Code will be available at https://github.com/lancopku/MUSE

cs.CL↗
arXiv2019

Understanding and Improving Layer Normalization

Layer normalization (LayerNorm) is a technique to normalize the distributions of intermediate layers. It enables smoother gradients, faster training, and better generalization accuracy. However, it is still unclear where the effectiveness stems from. In this paper, our main contribution is to take a step further in understanding LayerNorm. Many of previous studies believe that the success of LayerNorm comes from forward normalization. Unlike them, we find that the derivatives of the mean and variance are more important than forward normalization by re-centering and re-scaling backward gradients. Furthermore, we find that the parameters of LayerNorm, including the bias and gain, increase the risk of over-fitting and do not work in most cases. Experiments show that a simple version of LayerNorm (LayerNorm-simple) without the bias and gain outperforms LayerNorm on four datasets. It obtains the state-of-the-art performance on En-Vi machine translation. To address the over-fitting problem, we propose a new normalization method, Adaptive Normalization (AdaNorm), by replacing the bias and gain with a new transformation function. Experiments show that AdaNorm demonstrates better results than LayerNorm on seven out of eight datasets.

cs.LG↗
arXiv2019

ShellNet: Efficient Point Cloud Convolutional Neural Networks using Concentric Shells Statistics

Deep learning with 3D data has progressed significantly since the introduction of convolutional neural networks that can handle point order ambiguity in point cloud data. While being able to achieve good accuracies in various scene understanding tasks, previous methods often have low training speed and complex network architecture. In this paper, we address these problems by proposing an efficient end-to-end permutation invariant convolution for point cloud deep learning. Our simple yet effective convolution operator named ShellConv uses statistics from concentric spherical shells to define representative features and resolve the point order ambiguity, allowing traditional convolution to perform on such features. Based on ShellConv we further build an efficient neural network named ShellNet to directly consume the point clouds with larger receptive fields while maintaining less layers. We demonstrate the efficacy of ShellNet by producing state-of-the-art results on object classification, object part segmentation, and semantic scene segmentation while keeping the network very fast to train.

cs.CV↗
arXiv2019

Rotation Invariant Convolutions for 3D Point Clouds Deep Learning

Recent progresses in 3D deep learning has shown that it is possible to design special convolution operators to consume point cloud data. However, a typical drawback is that rotation invariance is often not guaranteed, resulting in networks being trained with data augmented with rotations. In this paper, we introduce a novel convolution operator for point clouds that achieves rotation invariance. Our core idea is to use low-level rotation invariant geometric features such as distances and angles to design a convolution operator for point cloud learning. The well-known point ordering problem is also addressed by a binning approach seamlessly built into the convolution. This convolution operator then serves as the basic building block of a neural network that is robust to point clouds under 6DoF transformations such as translation and rotation. Our experiment shows that our method performs with high accuracy in common scene understanding tasks such as object classification and segmentation. Compared to previous works, most importantly, our method is able to generalize and achieve consistent results across different scenarios in which training and testing can contain arbitrary rotations.

cs.CV↗
arXiv2019

Forecasting security's volatility using low-frequency historical data, high-frequency historical data and option-implied volatility

Low-frequency historical data, high-frequency historical data and option data are three major sources, which can be used to forecast the underlying security's volatility. In this paper, we propose two econometric models, which integrate three information sources. In GARCH-Itô-OI model, we assume that the option-implied volatility can influence the security's future volatility, and the option-implied volatility is treated as an observable exogenous variable. In GARCH-Itô-IV model, we assume that the option-implied volatility can not influence the security's volatility directly, and the relationship between the option-implied volatility and the security's volatility is constructed to extract useful information of the underlying security. After providing the quasi-maximum likelihood estimators for the parameters and establishing their asymptotic properties, we also conduct a series of simulation analysis and empirical analysis to compare the proposed models with other popular models in the literature. We find that when the sampling interval of the high-frequency data is 5 minutes, the GARCH-Itô-OI model and GARCH-Itô-IV model has better forecasting performance than other models.

q-fin.ST↗
arXiv2019

Primal Meaning Recommendation via On-line Encyclopedia

Polysemy is a very common phenomenon in modern languages. Under many circumstances, there exists a primal meaning for the expression. We define the primal meaning of an expression to be a frequently used sense of that expression from which its other frequent senses can be deduced. Many of the new appearing meanings of the expressions are either originated from a primal meaning, or are merely literal references to the original expression, e.g., apple (fruit), Apple (Inc), and Apple (movie). When constructing a knowledge base from on-line encyclopedia data, it would be more efficient to be aware of the information about the importance of the senses. In this paper, we would like to explore a way to automatically recommend the primal meaning of an expression based on the textual descriptions of the multiple senses of an expression from on-line encyclopedia websites. We propose a hybrid model that captures both the pattern of the description and the relationship between different descriptions with both weakly supervised and unsupervised models. The experiment results show that our method yields a good result with a P@1 (precision) score of 83.3 per cent, and a MAP (mean average precision) of 90.5 per cent, surpassing the UMFS-WE baseline by a big margin (P@1 is 61.1 per cent and MAP is 76.3 per cent).

cs.CL↗
arXiv2018

Electronic quantum effect on hydrogen bond geometry in water dimer

During compression of a water dimer calculated with high-precision first-principles methods, the trends of H-bond and O-H bond lengths show quantum effect of the electronic structure. We found that the H-bond length keeps decreasing, while the O-H bond length increases up to the stable point and decreases beyond it when the water dimer is further compressed. The remarkable properties are different from those observed in most previous researches which can be understood and extrapolated through classical simulation. The observations can be explained by the decrease in orbital overlap and change in the exchange repulsion interaction between water monomers. The dominant interaction between water monomers changes from electrostatic interaction to exchange repulsion at the turning point of the O-H bond length when the O...O distance is decreased. These findings highlight the quantum effect on the hydrogen bond in extreme conditions and play an important role in the recognition of the hydrogen bond structure and mechanism.

physics.chem-ph↗
arXiv2018

Zimmer's conjecture for lattice actions: the ${\rm SL}(n, \mathbb C)$-case

We prove Zimmer's conjecture for co-compact lattices in ${\rm SL}(n, \mathbb C)$: for any co-compact lattice in ${\rm SL}(n, \mathbb C)$, $n \geq 3$, any $Γ$-action on a compact manifold $M$ with dimension: (I) less than $2n-2$ if $n \neq 4$, (II) less than $5$ if $n = 4$, by $C^{1+ε}$ diffeomorphisms factors through a finite action.

math.DS↗
arXiv2018

Sequential Sampling for CGMY Processes via Decomposition of their Time Changes

We present a new and easy-to-implement sequential sampling method for CGMY processes with either finite or infinite variation, exploiting the time change representation of the CGMY model and a decomposition of its time change. We find that the time change can be decomposed into two independent components. While the first component is a \emph{finite} \emph{generalized gamma convolution} process whose increments can be sampled by either the exact double CFTP ("coupling from the past") method or an approximation scheme with high speed and accuracy, the second component can easily be made arbitrarily small in the $L^1$ sense. Simulation results show that the proposed method is advantageous over two existing methods under a model calibrated to historical option price data.

q-fin.CP↗
arXiv2018

Building an Ellipsis-aware Chinese Dependency Treebank for Web Text

Web 2.0 has brought with it numerous user-produced data revealing one's thoughts, experiences, and knowledge, which are a great source for many tasks, such as information extraction, and knowledge base construction. However, the colloquial nature of the texts poses new challenges for current natural language processing techniques, which are more adapt to the formal form of the language. Ellipsis is a common linguistic phenomenon that some words are left out as they are understood from the context, especially in oral utterance, hindering the improvement of dependency parsing, which is of great importance for tasks relied on the meaning of the sentence. In order to promote research in this area, we are releasing a Chinese dependency treebank of 319 weibos, containing 572 sentences with omissions restored and contexts reserved.

cs.CL↗
arXiv2017

On topological genericity of the mode-locking phenomenon

We study the circle homeomorphisms extensions over a strictly ergodic homeomorphism. Under a very mild restriction, we show that the fibered rotation number is locally constant on an open and dense subset. In the complement of this set, we found a dense subset in which every map is conjugate to a direct product. Our result provides a generalisation of Avila-Bochi-Damanik's result on ${\rm SL}(2,\mathbb{R})-$cocycles, and Jager-Wang-Zhou's result on quasi-periodically forced maps, to a broader setting.

math.DS↗
arXiv2017

Spin Polarization and Dispersion Effects in Emergence of Roaming Transition State for Nitrobenzene Isomerization

Since roaming was found as a new but common reaction path of isomerization, many of its properties, especially those of roaming transition state (TSR), have been studied recently on many systems. However, the mechanism of roaming is still not clear at the atomic level. In this work, we used first-principles calculations to illustrate the detailed structure of TSR in an internal isomerization process of nitrobenzene. The calculations distinctively show its nature of antiferromagnetic coupling between two roaming fragments. Moreover, the effect of dispersion is also revealed as an important issue for the stability of the TSR. Our work provides a new insight from the view of electronic structure towards the TSR and contributes to the basic understanding of the roaming systems.

physics.atm-clus↗
arXiv2017

The rigidity of pseudo-rotations on the two-torus and a question of Norton-Sullivan

We show that under certain boundedness condition, a $C^{r}$ conservative irrational pseudo-rotations on $\mathbb{T}^2$ with a generic rotation vector is $C^{r-1}$-rigid. We also obtain $C^0$-rigidity for Hölder pseudo-rotations with similar properties. These provide a partial generalisation of the main results in [B. Bramham, Invent. Math. (2015), no. 2, 561-580; A. Avila, B. Fayad, P. Le Calvez, D. Xu and Z. Zhang, arXiv: 1509.06906v1]. We then use these results to study conservative irrational pseudo-rotations on $\mathbb{T}^2$ with a generic rotation vector that is semi-conjugate to a translation via a semi-conjugacy homotopic to the identity. We show that the conservative centralizers of any such diffeomorphism is isomorphic to a uncountable subgroup of $\mathbb{R}^2/\mathbb{Z}^2$. In connection with a question of Alec Norton and Dennis Sullivan, we describe the topologically linearizable maps within this class using the topology of the conservative centralizer group. In the minimal case, we obtain a precise characterization of topological linearizability for all totally irrational vectors. We also construct a $C^\infty$ conservative and minimal totally irrational pseudo-rotation diffeomorphism that is semi-conjugate to a translation, but is topologically nonlinearizable. This gives a negative answer to the question of Norton and Sullivan in the $C^{\infty}$ category.

math.DS↗
arXiv2017

On the smooth dependence of SRB measures for partially hyperbolic systems

In this paper, we study the differentiability of SRB measures for partially hyperbolic systems. We show that for any $s \geq 1$, for any integer $\ell \geq 2$, any sufficiently large $r$, any $φ\in C^{r}(\T, \R)$ such that the map $f : \T^2 \to \T^2, f(x,y) = (\ell x, y + φ(x))$ is $C^r-$stably ergodic, there exists an open neighbourhood of $f$ in $C^r(\T^2,\T^2)$ such that any map in this neighbourhood has a unique SRB measure with $C^{s-1}$ density, which depends on the dynamics in a $C^s$ fashion. We also construct a $C^{\infty}$ mostly contracting partially hyperbolic diffeomorphism $f: \T^3 \to \T^3$ such that all $f'$ in a $C^2$ open neighbourhood of $f$ possess a unique SRB measure $μ_{f'}$ and the map $f' \mapsto μ_{f'}$ is strictly Hölder at $f$, in particular, non-differentiable. This gives a partial answer to Dolgopyat's Question 13.3 in \cite{Do1}.

math.DS↗