Searcharxiv⌕ Search

arXiv subjects

Xiao Zhang

Publications and source records attributed to Xiao Zhang.

At least 415 records · Page 23Linked to original sources
arXiv2021

Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs

Over-parameterized deep neural networks (DNNs) with sufficient capacity to memorize random noise can achieve excellent generalization performance, challenging the bias-variance trade-off in classical learning theory. Recent studies claimed that DNNs first learn simple patterns and then memorize noise; some other works showed a phenomenon that DNNs have a spectral bias to learn target functions from low to high frequencies during training. However, we show that the monotonicity of the learning bias does not always hold: under the experimental setup of deep double descent, the high-frequency components of DNNs diminish in the late stage of training, leading to the second descent of the test error. Besides, we find that the spectrum of DNNs can be applied to indicating the second descent of the test error, even though it is calculated from the training set only.

cs.LG↗
arXiv2021

Optimization Variance: Exploring Generalization Properties of DNNs

Unlike the conventional wisdom in statistical learning theory, the test error of a deep neural network (DNN) often demonstrates double descent: as the model complexity increases, it first follows a classical U-shaped curve and then shows a second descent. Through bias-variance decomposition, recent studies revealed that the bell-shaped variance is the major cause of model-wise double descent (when the DNN is widened gradually). This paper investigates epoch-wise double descent, i.e., the test error of a DNN also shows double descent as the number of training epoches increases. By extending the bias-variance analysis to epoch-wise double descent of the zero-one loss, we surprisingly find that the variance itself, without the bias, varies consistently with the test error. Inspired by this result, we propose a novel metric, optimization variance (OV), to measure the diversity of model updates caused by the stochastic gradients of random training batches drawn in the same iteration. OV can be estimated using samples from the training set only but correlates well with the (unknown) \emph{test} error, and hence early stopping may be achieved without using a validation set.

cs.LG↗
arXiv2021

Near-Room-Temperature Ferromagnetic Behavior of Single-Atom-Thick 2D Iron in Nanolaminated Ternary MAX Phases

Two dimensional (2D) ferromagnetic materials have attracted much attention in the fields of condensed matter physics and materials science, but their synthesis is still a challenge given their limitations on structural stability and susceptibility to oxidization. MAX phases nanolaminated ternary carbides or nitrides possess a unique crystal structure in which single-atom-thick A sublayers are interleaved by two dimensional MX slabs, providing nanostructured templates for designing 2D ferromagnetic materials if the non-magnetic A sublayers can be substituted replaced by magnetic elements. Here, we report three new ternary magnetic MAX phases (Ta2FeC, Ti2FeN and Nb2FeC) with A sublayers of single-atom-thick 2D iron through an isomorphous replacement reaction of MAX precursors (Ta2AlC, Ti2AlN and Nb2AlC) with a Lewis acid salts (FeCl2). All these MAX phases exhibit ferromagnetic (FM) behavior. The Curie temperature (Tc) of Ta2FeC and Nb2FeC MAX phase are 281 K and 291 K, respectively, i.e. close to room temperature. The saturation magnetization of these ternary magnetic MAX phases is almost two orders of magnitude higher than that of V2(Sn,Fe)C MAX phase whose A-site is partial substituted by Fe. Theoretical calculations on magnetic orderings of spin moments of Fe atoms in these nanolaminated magnetic MAX phases reveal that the magnetism can be mainly ascribed to intralayer exchange interaction of the 2D Fe atomic layers. Owning to the richness in composition of MAX phases, there is a large compositional space for constructing functional single-atom-thick 2D layers in materials using these nanolaminated templates.

cond-mat.mtrl-sci↗
arXiv2021

Quantum k-uniform states for heterogeneous systems from irredundant mixed orthogonal arrays

Quantum multipartite entangled states play significant roles in quantum information processing. By using difference schemes and orthogonal partitions, we construct a series of infinite classes of irredundant mixed orthogonal arrays (IrMOAs) and thus provide positive answers to two open problems. The first is the extension of the method for constructing homogeneous systems from orthogonal arrays (OAs) to heterogeneous multipartite systems with different individual levels. The second is the existence of $k$-uniform states in heterogeneous quantum systems. We present explicit constructions of two and three-uniform states for arbitrary heterogeneous multipartite systems with coprime individual levels, and characterize the entangled states in heterogeneous systems consisting of subsystems with nonprime power dimensions as well. Moreover, we obtain infinite classes of $k$-uniform states for heterogeneous multipartite systems for any $k\geq2$. The non-existence of a class of IrMOAs is also proved.

quant-ph↗
arXiv2021

Design of Robust and Efficient Edge Server Placement and Server Scheduling Policies: Extended Version

We study how to design edge server placement and server scheduling policies under workload uncertainty for 5G networks. We introduce a new metric called resource pooling factor to handle unexpected workload bursts. Maximizing this metric offers a strong enhancement on top of robust optimization against workload uncertainty. Using both real traces and synthetic traces, we show that the proposed server placement and server scheduling policies not only demonstrate better robustness against workload uncertainty than existing approaches, but also significantly reduce the cost of service providers. Specifically, in order to achieve close-to-zero workload rejection rate, the proposed server placement policy reduces the number of required edge servers by about 25% compared with the state-of-the-art approach; the proposed server scheduling policy reduces the energy consumption of edge servers by about 13% without causing much impact on the service quality.

cs.NI↗
arXiv2021

Improved Estimation of Concentration Under $\ell_p$-Norm Distance Metrics Using Half Spaces

Concentration of measure has been argued to be the fundamental cause of adversarial vulnerability. Mahloujifar et al. presented an empirical way to measure the concentration of a data distribution using samples, and employed it to find lower bounds on intrinsic robustness for several benchmark datasets. However, it remains unclear whether these lower bounds are tight enough to provide a useful approximation for the intrinsic robustness of a dataset. To gain a deeper understanding of the concentration of measure phenomenon, we first extend the Gaussian Isoperimetric Inequality to non-spherical Gaussian measures and arbitrary $\ell_p$-norms ($p \geq 2$). We leverage these theoretical insights to design a method that uses half-spaces to estimate the concentration of any empirical dataset under $\ell_p$-norm distance metrics. Our proposed algorithm is more efficient than Mahloujifar et al.'s, and our experiments on synthetic datasets and image benchmarks demonstrate that it is able to find much tighter intrinsic robustness bounds. These tighter estimates provide further evidence that rules out intrinsic dataset concentration as a possible explanation for the adversarial vulnerability of state-of-the-art classifiers.

cs.LG↗
arXiv2021

Tidal surface states as fingerprints of non-Hermitian nodal knot metals

Non-Hermitian nodal knot metals (NKMs) contains intricate complex-valued energy bands gives rise to knotted exceptional loops and new topological surface states. We introduce a formalism that connects the algebraic, geometric, and topological aspects of these surface states with their parent knots, and provide an optimized constructive ansatz for tight-binding models for non-Hermitian NKMs of arbitrary knot complexity and minimal hybridization range. Specifically, various representative non-Hermitian torus knots Hamiltonians are constructed in real-space, and their nodal topologies studied via winding numbers that avoid the explicit construction of generalized Brillouin zones. In particular, we identify the surface state boundaries as "tidal" intersections of the complex band structure in a marine landscape analogy. Beyond topological quantities based on Berry phases, we further find these tidal surface states to be intimately connected to the band vorticity and the layer structure of their dual Seifert surface, and as such provide a fingerprint for non-Hermitian NKMs.

cond-mat.mes-hall↗
arXiv2021

Non-Abelian charged nodal links in dielectric photonic crystal

A nodal link is a special form of a line degeneracy (a nodal line) between adjacent bands in the momentum space of a three-dimensional topological crystal. Unlike nodal chains or knots, a nodal link consists of two or more mutually-linked rings that do not touch each other. Recent studies on non-Abelian band topology revealed that the topological charges of the nodal links can have the properties of quaternions. However, a photonic crystal that has a nodal link with non-Abelian charges has not been reported. Here, we propose dielectric photonic crystals in forms of double diamond structures which realize the nodal links in the momentum space. By examining the evolution of the eigenstate correlations along the closed loops which enclose the nodal line(s) of the links, their non-Abelian topological charges are also analyzed. The proposed design scheme and theoretical approach in this work will allow for experimental observation of photonic non-Abelian charges in purely dielectric materials and facilitate the control of the degeneracy in complex photonic structures.

physics.optics↗
arXiv2021

Characterization of the magnetic field through the three-body loss near a narrow Feshbach resonance

The narrow s-wave Feshbach resonance of a $^6$Li Fermi gas shows strong three-body loss, which is proposed to be used to measure the minute change of a magnetic field around the resonance. However, the eddy current will cause ultracold atom experiencing a magnetic field delayed to the desired magnetic field from the current of the magnetic coils. The elimination of the eddy current effect will play a key role in any experiments that motivated to measure the magnetic field to the precision of a part per million stability. Here, we apply a method to correct the eddy current effect for precision measurement of the magnetic field. We first record the three-body loss influenced by the effect of induced eddy current, then use a certain model to obtain the time constant of the actual magnetic field by fitting the atom loss. This precisely determines the actual magnetic field according to the time response of the three-body loss. After that, we implement the desired magnetic field to the atoms so that we can analyze the three-body loss across the whole narrow Feshbach resonance. The results show that the three-body recombination is the dominated loss mechanism near the resonance. We expect this practical method of correcting the eddy current error of the magnetic field can be further applied to the future studies of quantum few- and many-body physics near a narrow Feshbach resonance.

cond-mat.quant-gas↗
arXiv2021

Multi-Task Time Series Forecasting With Shared Attention

Time series forecasting is a key component in many industrial and business decision processes and recurrent neural network (RNN) based models have achieved impressive progress on various time series forecasting tasks. However, most of the existing methods focus on single-task forecasting problems by learning separately based on limited supervised objectives, which often suffer from insufficient training instances. As the Transformer architecture and other attention-based models have demonstrated its great capability of capturing long term dependency, we propose two self-attention based sharing schemes for multi-task time series forecasting which can train jointly across multiple tasks. We augment a sequence of paralleled Transformer encoders with an external public multi-head attention function, which is updated by all data of all tasks. Experiments on a number of real-world multi-task time series forecasting tasks show that our proposed architectures can not only outperform the state-of-the-art single-task forecasting baselines but also outperform the RNN-based multi-task forecasting method.

cs.LG↗
arXiv2021

Modeling Heterogeneous Relations across Multiple Modes for Potential Crowd Flow Prediction

Potential crowd flow prediction for new planned transportation sites is a fundamental task for urban planners and administrators. Intuitively, the potential crowd flow of the new coming site can be implied by exploring the nearby sites. However, the transportation modes of nearby sites (e.g. bus stations, bicycle stations) might be different from the target site (e.g. subway station), which results in severe data scarcity issues. To this end, we propose a data driven approach, named MOHER, to predict the potential crowd flow in a certain mode for a new planned site. Specifically, we first identify the neighbor regions of the target site by examining the geographical proximity as well as the urban function similarity. Then, to aggregate these heterogeneous relations, we devise a cross-mode relational GCN, a novel relation-specific transformation model, which can learn not only the correlations but also the differences between different transportation modes. Afterward, we design an aggregator for inductive potential flow representation. Finally, an LTSM module is used for sequential flow prediction. Extensive experiments on real-world data sets demonstrate the superiority of the MOHER framework compared with the state-of-the-art algorithms.

cs.LG↗
arXiv2020

Guidance Module Network for Video Captioning

Video captioning has been a challenging and significant task that describes the content of a video clip in a single sentence. The model of video captioning is usually an encoder-decoder. We find that the normalization of extracted video features can improve the final performance of video captioning. Encoder-decoder model is usually trained using teacher-enforced strategies to make the prediction probability of each word close to a 0-1 distribution and ignore other words. In this paper, we present a novel architecture which introduces a guidance module to encourage the encoder-decoder model to generate words related to the past and future words in a caption. Based on the normalization and guidance module, guidance module net (GMNet) is built. Experimental results on commonly used dataset MSVD show that proposed GMNet can improve the performance of the encoder-decoder model on video captioning tasks.

cs.CV↗
arXiv2020

Self-Supervised Visual Representation Learning from Hierarchical Grouping

We create a framework for bootstrapping visual representation learning from a primitive visual grouping capability. We operationalize grouping via a contour detector that partitions an image into regions, followed by merging of those regions into a tree hierarchy. A small supervised dataset suffices for training this grouping primitive. Across a large unlabeled dataset, we apply this learned primitive to automatically predict hierarchical region structure. These predictions serve as guidance for self-supervised contrastive feature learning: we task a deep network with producing per-pixel embeddings whose pairwise distances respect the region hierarchy. Experiments demonstrate that our approach can serve as state-of-the-art generic pre-training, benefiting downstream tasks. We additionally explore applications to semantic region search and video-based object instance tracking.

cs.CV↗
arXiv2020

Measuring Information Transfer in Neural Networks

Quantifying the information content in a neural network model is essentially estimating the model's Kolmogorov complexity. Recent success of prequential coding on neural networks points to a promising path of deriving an efficient description length of a model. We propose a practical measure of the generalizable information in a neural network model based on prequential coding, which we term Information Transfer ($L_{IT}$). Theoretically, $L_{IT}$ is an estimation of the generalizable part of a model's information content. In experiments, we show that $L_{IT}$ is consistently correlated with generalizable information and can be used as a measure of patterns or "knowledge" in a model or a dataset. Consequently, $L_{IT}$ can serve as a useful analysis tool in deep learning. In this paper, we apply $L_{IT}$ to compare and dissect information in datasets, evaluate representation models in transfer learning, and analyze catastrophic forgetting and continual learning algorithms. $L_{IT}$ provides an information perspective which helps us discover new insights into neural network learning.

cs.LG↗
arXiv2020

Cross-Lingual Document Retrieval with Smooth Learning

Cross-lingual document search is an information retrieval task in which the queries' language differs from the documents' language. In this paper, we study the instability of neural document search models and propose a novel end-to-end robust framework that achieves improved performance in cross-lingual search with different documents' languages. This framework includes a novel measure of the relevance, smooth cosine similarity, between queries and documents, and a novel loss function, Smooth Ordinal Search Loss, as the objective. We further provide theoretical guarantee on the generalization error bound for the proposed framework. We conduct experiments to compare our approach with other document search models, and observe significant gains under commonly used ranking metrics on the cross-lingual document retrieval task in a variety of languages.

cs.IR↗
arXiv2020

Semi-supervised Autoencoding Projective Dependency Parsing

We describe two end-to-end autoencoding models for semi-supervised graph-based projective dependency parsing. The first model is a Locally Autoencoding Parser (LAP) encoding the input using continuous latent variables in a sequential manner; The second model is a Globally Autoencoding Parser (GAP) encoding the input into dependency trees as latent variables, with exact inference. Both models consist of two parts: an encoder enhanced by deep neural networks (DNN) that can utilize the contextual information to encode the input into latent variables, and a decoder which is a generative model able to reconstruct the input. Both LAP and GAP admit a unified structure with different loss functions for labeled and unlabeled data with shared parameters. We conducted experiments on WSJ and UD dependency parsing data sets, showing that our models can exploit the unlabeled data to improve the performance given a limited amount of labeled data, and outperform a previously proposed semi-supervised model.

cs.CL↗
arXiv2020

Molten salt synthesis of a nanolaminated Sc2SnC MAX Phase

The MAX phases are a family of of ternary layered material with both metal and ceramic properties, and it is also precursor ma-terials for synthesis of two-dimensional MXenes. The theory predicted that there are more than 600 stable ternary layered MAX phases. At present, there are more than 80 kinds of ternary MAX phases synthesized through experiments, and few reports on MAX phases where M is a rare earth element. In this study, a new MAX phase Sc2SnC with rare earth element Sc at the M sites was synthesized through the reaction sintering of Sc, Sn, and C mixtures. Phase composition and microstructure of Sc2SnC were confirmed by X-ray diffraction, scanning electron microscopy and X-ray energy spectrum analysis. And structural stability, mechanical and electronic properties of Sc2SnC was investigated via density functional theory. This study open a door for ex-plore more unknown ternary layered rare earth compounds Ren+1SnCn (Re=Sc, Y, La-Nd, n=1) and corresponding rare earth MXenes.

cond-mat.mtrl-sci↗