arXiv subjects

Shu-Tao Xia

Publications and source records attributed to Shu-Tao Xia.

At least 217 records · Page 12Linked to original sources

arXiv2020

Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets

Skip connections are an essential component of current state-of-the-art deep neural networks (DNNs) such as ResNet, WideResNet, DenseNet, and ResNeXt. Despite their huge success in building deeper and more powerful DNNs, we identify a surprising security weakness of skip connections in this paper. Use of skip connections allows easier generation of highly transferable adversarial examples. Specifically, in ResNet-like (with skip connections) neural networks, gradients can backpropagate through either skip connections or residual modules. We find that using more gradients from the skip connections rather than the residual modules according to a decay factor, allows one to craft adversarial examples with high transferability. Our method is termed Skip Gradient Method(SGM). We conduct comprehensive transfer attacks against state-of-the-art DNNs including ResNets, DenseNets, Inceptions, Inception-ResNet, Squeeze-and-Excitation Network (SENet) and robustly trained DNNs. We show that employing SGM on the gradient flow can greatly improve the transferability of crafted attacks in almost all cases. Furthermore, SGM can be easily combined with existing black-box attack techniques, and obtain high improvements over state-of-the-art transferability methods. Our findings not only motivate new research into the architectural vulnerability of DNNs, but also open up further challenges for the design of secure DNN architectures.

cs.LG↗

arXiv2019

Deep Flow Collaborative Network for Online Visual Tracking

The deep learning-based visual tracking algorithms such as MDNet achieve high performance leveraging to the feature extraction ability of a deep neural network. However, the tracking efficiency of these trackers is not very high due to the slow feature extraction for each frame in a video. In this paper, we propose an effective tracking algorithm to alleviate the time-consuming problem. Specifically, we design a deep flow collaborative network, which executes the expensive feature network only on sparse keyframes and transfers the feature maps to other frames via optical flow. Moreover, we raise an effective adaptive keyframe scheduling mechanism to select the most appropriate keyframe. We evaluate the proposed approach on large-scale datasets: OTB2013 and OTB2015. The experiment results show that our algorithm achieves considerable speedup and high precision as well.

cs.CV↗

arXiv2019

Bounds and Constructions of Locally Repairable Codes: Parity-check Matrix Approach

A $q$-ary $(n,k,r)$ locally repairable code (LRC) is an $[n,k,d]$ linear code over $\mathbb{F}_q$ such that every code symbol can be recovered by accessing at most $r$ other code symbols. The well-known Singleton-like bound says that $d \le n-k-\lceil k/r\rceil +2$ and an LRC is said to be optimal if it attains this bound. In this paper, we study the bounds and constructions of LRCs from the view of parity-check matrices. Firstly, a simple and unified framework based on parity-check matrix to analyze the bounds of LRCs is proposed. Several useful structural properties on $q$-ary optimal LRCs are obtained. We derive an upper bound on the minimum distance of $q$-ary optimal $(n,k,r)$-LRCs in terms of the field size $q$. Then, we focus on constructions of optimal LRCs over binary field. It is proved that there are only 5 classes of possible parameters with which optimal binary $(n,k,r)$-LRCs exist. Moreover, by employing the proposed parity-check matrix approach, we completely enumerate all these 5 classes of possible optimal binary LRCs attaining the Singleton-like bound in the sense of equivalence of linear codes.

cs.IT↗

arXiv2019

Adaptive Regularization of Labels

Recently, a variety of regularization techniques have been widely applied in deep neural networks, such as dropout, batch normalization, data augmentation, and so on. These methods mainly focus on the regularization of weight parameters to prevent overfitting effectively. In addition, label regularization techniques such as label smoothing and label disturbance have also been proposed with the motivation of adding a stochastic perturbation to labels. In this paper, we propose a novel adaptive label regularization method, which enables the neural network to learn from the erroneous experience and update the optimal label representation online. On the other hand, compared with knowledge distillation, which learns the correlation of categories using teacher network, our proposed method requires only a minuscule increase in parameters without cumbersome teacher network. Furthermore, we evaluate our method on CIFAR-10/CIFAR-100/ImageNet datasets for image recognition tasks and AGNews/Yahoo/Yelp-Full datasets for text classification tasks. The empirical results show significant improvement under all experimental settings.

cs.LG↗

arXiv2018

Dimensionality-Driven Learning with Noisy Labels

Datasets with significant proportions of noisy (incorrect) class labels present challenges for training accurate Deep Neural Networks (DNNs). We propose a new perspective for understanding DNN generalization for such datasets, by investigating the dimensionality of the deep representation subspace of training samples. We show that from a dimensionality perspective, DNNs exhibit quite distinctive learning styles when trained with clean labels versus when trained with a proportion of noisy labels. Based on this finding, we develop a new dimensionality-driven learning strategy, which monitors the dimensionality of subspaces during training and adapts the loss function accordingly. We empirically demonstrate that our approach is highly tolerant to significant proportions of noisy labels, and can effectively learn low-dimensional local subspaces that capture the data distribution.

cs.CV↗

arXiv2018

Iterative Learning with Open-set Noisy Labels

Large-scale datasets possessing clean label annotations are crucial for training Convolutional Neural Networks (CNNs). However, labeling large-scale data can be very costly and error-prone, and even high-quality datasets are likely to contain noisy (incorrect) labels. Existing works usually employ a closed-set assumption, whereby the samples associated with noisy labels possess a true class contained within the set of known classes in the training data. However, such an assumption is too restrictive for many applications, since samples associated with noisy labels might in fact possess a true class that is not present in the training data. We refer to this more complex scenario as the \textbf{open-set noisy label} problem and show that it is nontrivial in order to make accurate predictions. To address this problem, we propose a novel iterative learning framework for training CNNs on datasets with open-set noisy labels. Our approach detects noisy labels and learns deep discriminative features in an iterative fashion. To benefit from the noisy label detection, we design a Siamese network to encourage clean labels and noisy labels to be dissimilar. A reweighting module is also applied to simultaneously emphasize the learning from clean labels and reduce the effect caused by noisy labels. Experiments on CIFAR-10, ImageNet and real-world noisy (web-search) datasets demonstrate that our proposed model can robustly train CNNs in the presence of a high proportion of open-set as well as closed-set noisy labels.

cs.CV↗

arXiv2017

Accelerated Stochastic Greedy Coordinate Descent by Soft Thresholding Projection onto Simplex

In this paper we study the well-known greedy coordinate descent (GCD) algorithm to solve $\ell_1$-regularized problems and improve GCD by the two popular strategies: Nesterov's acceleration and stochastic optimization. Firstly, we propose a new rule for greedy selection based on an $\ell_1$-norm square approximation which is nontrivial to solve but convex, then an efficient algorithm called "SOft ThreshOlding PrOjection (SOTOPO)" is proposed to exactly solve the $\ell_1$-regularized $\ell_1$-norm square approximation problem, which is induced by the new rule. Based on the new rule and the SOTOPO algorithm, the Nesterov's acceleration and stochastic optimization strategies are then successfully applied to the GCD algorithm. The resulted algorithm called accelerated stochastic greedy coordinate descent (ASGCD) has the optimal convergence rate $O(\sqrt{1/ε})$, meanwhile, it reduces the iteration complexity of greedy selection up to a factor of sample size. Both theoretically and empirically, we show that ASGCD has better performance for high-dimensional and dense problems with sparse solution.

math.OC↗

arXiv2017

Locally Repairable Codes with Multiple $(r_{i}, δ_{i})$-Localities

In distributed storage systems, locally repairable codes (LRCs) are introduced to realize low disk I/O and repair cost. In order to tolerate multiple node failures, the LRCs with \emph{$(r, δ)$-locality} are further proposed. Since hot data is not uncommon in a distributed storage system, both Zeh \emph{et al.} and Kadhe \emph{et al.} focus on the LRCs with \emph{multiple localities or unequal localities} (ML-LRCs) recently, which said that the localities among the code symbols can be different. ML-LRCs are attractive and useful in reducing repair cost for hot data. In this paper, we generalize the ML-LRCs to the $(r,δ)$-locality case of multiple node failures, and define an LRC with multiple $(r_{i}, δ_{i})_{i\in [s]}$ localities ($s\ge 2$), where $r_{1}\leq r_{2}\leq\dots\leq r_{s}$ and $δ_{1}\geqδ_{2}\geq\dots\geqδ_{s}\geq2$. Such codes ensure that some hot data could be repaired more quickly and have better failure-tolerance in certain cases because of relatively smaller $r_{i}$ and larger $δ_{i}$. Then, we derive a Singleton-like upper bound on the minimum distance for the proposed LRCs by employing the regenerating-set technique. Finally, we obtain a class of explicit and structured constructions of optimal ML-LRCs, and further extend them to the cases of multiple $(r_{i}, δ)_{i\in [s]}$ localities.

cs.IT↗

arXiv2017

On the Weight Hierarchy of Locally Repairable Codes

An $(n,k,r)$ \emph{locally repairable code} (LRC) is an $[n,k,d]$ linear code where every code symbol can be repaired from at most $r$ other code symbols. An LRC is said to be optimal if the minimum distance attains the Singleton-like bound $d \le n-k-\lceil k/r \rceil +2$. The \emph{generalized Hamming weights} (GHWs) of linear codes are fundamental parameters which have many useful applications. Generally it is difficult to determine the GHWs of linear codes. In this paper, we study the GHWs of LRCs. Firstly, we obtain a generalized Singleton-like bound on the $i$-th $(1\le i \le k)$ GHWs of general $(n,k,r)$ LRCs. Then, it is shown that for an optimal $(n,k,r)$ LRC with $r \mid k$, its weight hierarchy can be completely determined, and the $i$-th GHW of an optimal $(n,k,r)$ LRC with $r \mid k$ attains the proposed generalized Singleton-like bound for all $1\le i \le k$. For an optimal $(n,k,r)$ LRC with $r \nmid k$, we give lower bounds on the GHWs of the LRC and its dual code. Finally, two general bounds on linear codes in terms of the GHWs are presented. Moreover, it is also shown that some previous results on the bounds of minimum distances of linear codes can also be explained or refined in terms of the GHWs.

cs.IT↗

arXiv2017

On Optimal Ternary Locally Repairable Codes

In an $[n,k,d]$ linear code, a code symbol is said to have locality $r$ if it can be repaired by accessing at most $r$ other code symbols. For an $(n,k,r)$ \emph{locally repairable code} (LRC), the minimum distance satisfies the well-known Singleton-like bound $d\le n-k-\lceil k/r\rceil +2$. In this paper, we study optimal ternary LRCs meeting this Singleton-like bound by employing a parity-check matrix approach. It is proved that there are only $8$ classes of possible parameters with which optimal ternary LRCs exist. Moreover, we obtain explicit constructions of optimal ternary LRCs for all these $8$ classes of parameters, where the minimum distance could only be 2, 3, 4, 5 and 6.

cs.IT↗

arXiv2016

Constructions of Optimal Cyclic $(r,δ)$ Locally Repairable Codes

A code is said to be a $r$-local locally repairable code (LRC) if each of its coordinates can be repaired by accessing at most $r$ other coordinates. When some of the $r$ coordinates are also erased, the $r$-local LRC can not accomplish the local repair, which leads to the concept of $(r,δ)$-locality. A $q$-ary $[n, k]$ linear code $\cC$ is said to have $(r, δ)$-locality ($δ\ge 2$) if for each coordinate $i$, there exists a punctured subcode of $\cC$ with support containing $i$, whose length is at most $r + δ- 1$, and whose minimum distance is at least $δ$. The $(r, δ)$-LRC can tolerate $δ-1$ erasures in total, which degenerates to a $r$-local LRC when $δ=2$. A $q$-ary $(r,δ)$ LRC is called optimal if it meets the Singleton-like bound for $(r,δ)$-LRCs. A class of optimal $q$-ary cyclic $r$-local LRCs with lengths $n\mid q-1$ were constructed by Tamo, Barg, Goparaju and Calderbank based on the $q$-ary Reed-Solomon codes. In this paper, we construct a class of optimal $q$-ary cyclic $(r,δ)$-LRCs ($δ\ge 2$) with length $n\mid q-1$, which generalizes the results of Tamo \emph{et al.} Moreover, we construct a new class of optimal $q$-ary cyclic $r$-local LRCs with lengths $n\mid q+1$ and a new class of optimal $q$-ary cyclic $(r,δ)$-LRCs ($δ\ge 2$) with lengths $n\mid q+1$. The constructed optimal LRCs with length $n=q+1$ have the best-known length $q+1$ for the given finite field with size $q$ when the minimum distance is larger than $4$.

cs.IT↗

arXiv2016

Unifying Decision Trees Split Criteria Using Tsallis Entropy

The construction of efficient and effective decision trees remains a key topic in machine learning because of their simplicity and flexibility. A lot of heuristic algorithms have been proposed to construct near-optimal decision trees. ID3, C4.5 and CART are classical decision tree algorithms and the split criteria they used are Shannon entropy, Gain Ratio and Gini index respectively. All the split criteria seem to be independent, actually, they can be unified in a Tsallis entropy framework. Tsallis entropy is a generalization of Shannon entropy and provides a new approach to enhance decision trees' performance with an adjustable parameter $q$. In this paper, a Tsallis Entropy Criterion (TEC) algorithm is proposed to unify Shannon entropy, Gain Ratio and Gini index, which generalizes the split criteria of decision trees. More importantly, we reveal the relations between Tsallis entropy with different $q$ and other split criteria. Experimental results on UCI data sets indicate that the TEC algorithm achieves statistically significant improvement over the classical algorithms.

stat.ML↗

arXiv2016

Nonextensive information theoretical machine

In this paper, we propose a new discriminative model named \emph{nonextensive information theoretical machine (NITM)} based on nonextensive generalization of Shannon information theory. In NITM, weight parameters are treated as random variables. Tsallis divergence is used to regularize the distribution of weight parameters and maximum unnormalized Tsallis entropy distribution is used to evaluate fitting effect. On the one hand, it is showed that some well-known margin-based loss functions such as $\ell_{0/1}$ loss, hinge loss, squared hinge loss and exponential loss can be unified by unnormalized Tsallis entropy. On the other hand, Gaussian prior regularization is generalized to Student-t prior regularization with similar computational complexity. The model can be solved efficiently by gradient-based convex optimization and its performance is illustrated on standard datasets.

cs.LG↗

arXiv2016

Alternating direction algorithms for $\ell_0$ regularization in compressed sensing

In this paper we propose three iterative greedy algorithms for compressed sensing, called \emph{iterative alternating direction} (IAD), \emph{normalized iterative alternating direction} (NIAD) and \emph{alternating direction pursuit} (ADP), which stem from the iteration steps of alternating direction method of multiplier (ADMM) for $\ell_0$-regularized least squares ($\ell_0$-LS) and can be considered as the alternating direction versions of the well-known iterative hard thresholding (IHT), normalized iterative hard thresholding (NIHT) and hard thresholding pursuit (HTP) respectively. Firstly, relative to the general iteration steps of ADMM, the proposed algorithms have no splitting or dual variables in iterations and thus the dependence of the current approximation on past iterations is direct. Secondly, provable theoretical guarantees are provided in terms of restricted isometry property, which is the first theoretical guarantee of ADMM for $\ell_0$-LS to the best of our knowledge. Finally, they outperform the corresponding IHT, NIHT and HTP greatly when reconstructing both constant amplitude signals with random signs (CARS signals) and Gaussian signals.

cs.IT↗

arXiv2016

Bayesian linear regression with Student-t assumptions

As an automatic method of determining model complexity using the training data alone, Bayesian linear regression provides us a principled way to select hyperparameters. But one often needs approximation inference if distribution assumption is beyond Gaussian distribution. In this paper, we propose a Bayesian linear regression model with Student-t assumptions (BLRS), which can be inferred exactly. In this framework, both conjugate prior and expectation maximization (EM) algorithm are generalized. Meanwhile, we prove that the maximum likelihood solution is equivalent to the standard Bayesian linear regression with Gaussian assumptions (BLRG). The $q$-EM algorithm for BLRS is nearly identical to the EM algorithm for BLRG. It is showed that $q$-EM for BLRS can converge faster than EM for BLRG for the task of predicting online news popularity.

cs.LG↗

arXiv2014

Deterministic Constructions of Binary Measurement Matrices from Finite Geometry

Deterministic constructions of measurement matrices in compressed sensing (CS) are considered in this paper. The constructions are inspired by the recent discovery of Dimakis, Smarandache and Vontobel which says that parity-check matrices of good low-density parity-check (LDPC) codes can be used as {provably} good measurement matrices for compressed sensing under $\ell_1$-minimization. The performance of the proposed binary measurement matrices is mainly theoretically analyzed with the help of the analyzing methods and results from (finite geometry) LDPC codes. Particularly, several lower bounds of the spark (i.e., the smallest number of columns that are linearly dependent, which totally characterizes the recovery performance of $\ell_0$-minimization) of general binary matrices and finite geometry matrices are obtained and they improve the previously known results in most cases. Simulation results show that the proposed matrices perform comparably to, sometimes even better than, the corresponding Gaussian random matrices. Moreover, the proposed matrices are sparse, binary, and most of them have cyclic or quasi-cyclic structure, which will make the hardware realization convenient and easy.

cs.IT↗

arXiv2014

Deterministic Construction of Binary Measurement Matrices with Various Sizes

We introduce a general framework to deterministically construct binary measurement matrices for compressed sensing. The proposed matrices are composed of (circulant) permutation submatrix blocks and zero submatrix blocks, thus making their hardware realization convenient and easy. Firstly, using the famous Johnson bound for binary constant weight codes, we derive a new lower bound for the coherence of binary matrices with uniform column weights. Afterwards, a large class of binary base matrices with coherence asymptotically achieving this new bound are presented. Finally, by choosing proper rows and columns from these base matrices, we construct the desired measurement matrices with various sizes and they show empirically comparable performance to that of the corresponding Gaussian matrices

cs.IT↗

arXiv2014

Subspace Thresholding Pursuit: A Reconstruction Algorithm for Compressed Sensing

We propose a new iterative greedy algorithm for reconstructions of sparse signals with or without noisy perturbations in compressed sensing. The proposed algorithm, called \emph{subspace thresholding pursuit} (STP) in this paper, is a simple combination of subspace pursuit and iterative hard thresholding. Firstly, STP has the theoretical guarantee comparable to that of $\ell_1$ minimization in terms of restricted isometry property. Secondly, with a tuned parameter, on the one hand, when reconstructing Gaussian signals, it can outperform other state-of-the-art reconstruction algorithms greatly; on the other hand, when reconstructing constant amplitude signals with random signs, it can outperform other state-of-the-art iterative greedy algorithms and even outperform $\ell_1$ minimization if the undersampling ratio is not very large. In addition, we propose a simple but effective method to improve the empirical performance further if the undersampling ratio is large. Finally, it is showed that other iterative greedy algorithms can improve their empirical performance by borrowing the idea of STP.

cs.IT↗