arXiv subjects

Xiaojun Chen

Publications and source records attributed to Xiaojun Chen.

At least 91 records · Page 5Linked to original sources

arXiv2022

A Survey for Efficient Open Domain Question Answering

Open domain question answering (ODQA) is a longstanding task aimed at answering factual questions from a large knowledge corpus without any explicit evidence in natural language processing (NLP). Recent works have predominantly focused on improving the answering accuracy and achieved promising progress. However, higher accuracy often comes with more memory consumption and inference latency, which might not necessarily be efficient enough for direct deployment in the real world. Thus, a trade-off between accuracy, memory consumption and processing speed is pursued. In this paper, we provide a survey of recent advances in the efficiency of ODQA models. We walk through the ODQA models and conclude the core techniques on efficiency. Quantitative analysis on memory cost, processing speed, accuracy and overall comparison are given. We hope that this work would keep interested scholars informed of the advances and open challenges in ODQA efficiency research, and thus contribute to the further development of ODQA efficiency.

cs.CL↗

arXiv2022

Anderson Acceleration for Nonsmooth Fixed Point Problems

We give new convergence results of Anderson acceleration for the composite $\max$ fixed point problem. We prove that Anderson(1) and EDIIS(1) are q-linear convergent with a smaller q-factor than existing q-factors. Moreover, we propose a smoothing approximation of the composite max function in the contractive fixed point problem. We show that the smoothing approximation is a contraction mapping with the same fixed point as the composite $\max$ fixed point problem. Our results rigorously confirm that the nonsmoothness does not affect the convergence rate of Anderson acceleration method when we use the proposed smoothing approximation for the composite $\max$ fixed point problem. Numerical results for constrained minimax problems, complementarity problems and nonsmooth differential equations are presented to show the efficiency and good performance of the proposed Anderson acceleration method with smoothing approximation.

math.OC↗

arXiv2022

An Optimal Control Problem with Terminal Stochastic Linear Complementarity Constraints

In this paper, we investigate an optimal control problem with terminal stochastic linear complementarity constraints (SLCC), and its discrete approximation using the relaxation, the sample average approximation (SAA) and the implicit Euler time-stepping scheme. We show the existence of feasible solutions and optimal solutions to the optimal control problem and its discrete approximation under the conditions that the expectation of the stochastic matrix in the SLCC is a Z-matrix or an adequate matrix. Moreover, we prove that the solution sequence generated by the discrete approximation converges to a solution of the original optimal control problem with probability 1 as $ε\downarrow 0$, $ν\to \infty $ and $h\downarrow 0$, where $ε$ is the relaxation parameter, $ν$ is the sample size and $h$ is the mesh size. We also provide asymptotics of the SAA optimal value and error bounds of the time-stepping method. A numerical example is used to illustrate the existence of optimal solutions, the discretization scheme and error estimation.

math.OC↗

arXiv2022

PrUE: Distilling Knowledge from Sparse Teacher Networks

Although deep neural networks have enjoyed remarkable success across a wide variety of tasks, their ever-increasing size also imposes significant overhead on deployment. To compress these models, knowledge distillation was proposed to transfer knowledge from a cumbersome (teacher) network into a lightweight (student) network. However, guidance from a teacher does not always improve the generalization of students, especially when the size gap between student and teacher is large. Previous works argued that it was due to the high certainty of the teacher, resulting in harder labels that were difficult to fit. To soften these labels, we present a pruning method termed Prediction Uncertainty Enlargement (PrUE) to simplify the teacher. Specifically, our method aims to decrease the teacher's certainty about data, thereby generating soft predictions for students. We empirically investigate the effectiveness of the proposed method with experiments on CIFAR-10/100, Tiny-ImageNet, and ImageNet. Results indicate that student networks trained with sparse teachers achieve better performance. Besides, our method allows researchers to distill knowledge from deeper networks to improve students further. Our code is made public at: \url{https://github.com/wangshaopu/prue}.

cs.CV↗

arXiv2022

Identifying Electrocardiogram Abnormalities Using a Handcrafted-Rule-Enhanced Neural Network

A large number of people suffer from life-threatening cardiac abnormalities, and electrocardiogram (ECG) analysis is beneficial to determining whether an individual is at risk of such abnormalities. Automatic ECG classification methods, especially the deep learning based ones, have been proposed to detect cardiac abnormalities using ECG records, showing good potential to improve clinical diagnosis and help early prevention of cardiovascular diseases. However, the predictions of the known neural networks still do not satisfactorily meet the needs of clinicians, and this phenomenon suggests that some information used in clinical diagnosis may not be well captured and utilized by these methods. In this paper, we introduce some rules into convolutional neural networks, which help present clinical knowledge to deep learning based ECG analysis, in order to improve automated ECG diagnosis performance. Specifically, we propose a Handcrafted-Rule-enhanced Neural Network (called HRNN) for ECG classification with standard 12-lead ECG input, which consists of a rule inference module and a deep learning module. Experiments on two large-scale public ECG datasets show that our new approach considerably outperforms existing state-of-the-art methods. Further, our proposed approach not only can improve the diagnosis performance, but also can assist in detecting mislabelled ECG samples. Our codes are available at https://github.com/alwaysbyx/ecg_processing.

cs.AI↗

arXiv2022

CLTS+: A New Chinese Long Text Summarization Dataset with Abstractive Summaries

The abstractive methods lack of creative ability is particularly a problem in automatic text summarization. The summaries generated by models are mostly extracted from the source articles. One of the main causes for this problem is the lack of dataset with abstractiveness, especially for Chinese. In order to solve this problem, we paraphrase the reference summaries in CLTS, the Chinese Long Text Summarization dataset, correct errors of factual inconsistencies, and propose the first Chinese Long Text Summarization dataset with a high level of abstractiveness, CLTS+, which contains more than 180K article-summary pairs and is available online. Additionally, we introduce an intrinsic metric based on co-occurrence words to evaluate the dataset we constructed. We analyze the extraction strategies used in CLTS+ summaries against other datasets to quantify the abstractiveness and difficulty of our new data and train several baselines on CLTS+ to verify the utility of it for improving the creative ability of models.

cs.CL↗

arXiv2022

An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity

The leaky ReLU network with a group sparse regularization term has been widely used in the recent years. However, training such a network yields a nonsmooth nonconvex optimization problem and there exists a lack of approaches to compute a stationary point deterministically. In this paper, we first resolve the multi-layer composite term in the original optimization problem by introducing auxiliary variables and additional constraints. We show the new model has a nonempty and bounded solution set and its feasible set satisfies the Mangasarian-Fromovitz constraint qualification. Moreover, we show the relationship between the new model and the original problem. Remarkably, we propose an inexact augmented Lagrangian algorithm for solving the new model and show the convergence of the algorithm to a KKT point. Numerical experiments demonstrate that our algorithm is more efficient for training sparse leaky ReLU neural networks than some well-known algorithms.

math.OC↗

arXiv2022

Group sparse optimization for inpainting of random fields on the sphere

We propose a group sparse optimization model for inpainting of a square-integrable isotropic random field on the unit sphere, where the field is represented by spherical harmonics with random complex coefficients. In the proposed optimization model, the variable is an infinite-dimensional complex vector and the objective function is a real-valued function defined by a hybrid of the $\ell_2$ norm and non-Liptchitz $\ell_p (0<p<1)$ norm that preserves rotational invariance property and group structure of the random complex coefficients. We show that the infinite-dimensional optimization problem is equivalent to a convexly-constrained finite-dimensional optimization problem. Moreover, we propose a smoothing penalty algorithm to solve the finite-dimensional problem via unconstrained optimization problems. We provide an approximation error bound of the inpainted random field defined by a scaled KKT point of the constrained optimization problem in the square-integrable space on the sphere with probability measure. Finally, we conduct numerical experiments on band-limited random fields on the sphere and images from Earth topography data to show the promising performance of the smoothing penalty algorithm for inpainting of random fields on the sphere.

math.OC↗

arXiv2022

Linearly-constrained nonsmooth optimization for training autoencoders

A regularized minimization model with $l_1$-norm penalty (RP) is introduced for training the autoencoders that belong to a class of two-layer neural networks. We show that the RP can act as an exact penalty model which shares the same global minimizers, local minimizers, and d(irectional)-stationary points with the original regularized model under mild conditions. We construct a bounded box region that contains at least one global minimizer of the RP, and propose a linearly constrained regularized minimization model with $l_1$-norm penalty (LRP) for training autoencoders. A smoothing proximal gradient algorithm is designed to solve the LRP. Convergence of the algorithm to a generalized d-stationary point of the RP and LRP is delivered. Comprehensive numerical experiments convincingly illustrate the efficiency as well as the robustness of the proposed algorithm.

math.OC↗

arXiv2022

Deep Unsupervised Hashing with Latent Semantic Components

Deep unsupervised hashing has been appreciated in the regime of image retrieval. However, most prior arts failed to detect the semantic components and their relationships behind the images, which makes them lack discriminative power. To make up the defect, we propose a novel Deep Semantic Components Hashing (DSCH), which involves a common sense that an image normally contains a bunch of semantic components with homology and co-occurrence relationships. Based on this prior, DSCH regards the semantic components as latent variables under the Expectation-Maximization framework and designs a two-step iterative algorithm with the objective of maximum likelihood of training data. Firstly, DSCH constructs a semantic component structure by uncovering the fine-grained semantics components of images with a Gaussian Mixture Modal~(GMM), where an image is represented as a mixture of multiple components, and the semantics co-occurrence are exploited. Besides, coarse-grained semantics components, are discovered by considering the homology relationships between fine-grained components, and the hierarchy organization is then constructed. Secondly, DSCH makes the images close to their semantic component centers at both fine-grained and coarse-grained levels, and also makes the images share similar semantic components close to each other. Extensive experiments on three benchmark datasets demonstrate that the proposed hierarchical semantic components indeed facilitate the hashing model to achieve superior performance.

cs.CV↗

arXiv2022

Optimality conditions for nonsmooth nonconvex-nonconcave min-max problems and generative adversarial networks

This paper considers a class of nonsmooth nonconvex-nonconcave min-max problems in machine learning and games. We first provide sufficient conditions for the existence of global minimax points and local minimax points. Next, we establish the first-order and second-order optimality conditions for local minimax points by using directional derivatives. These conditions reduce to smooth min-max problems with Fr{é}chet derivatives. We apply our theoretical results to generative adversarial networks (GANs) in which two neural networks contest with each other in a game. Examples are used to illustrate applications of the new theory for training GANs.

math.OC↗

arXiv2022

Twisted bi-symplectic structure on Koszul twisted Calabi-Yau algebras

For a Koszul Artin-Schelter regular algebra (also called twisted Calabi-Yau algebra), we show that it has a "twisted" bi-symplectic structure, which may be viewed as a noncommutative and twisted analogue of the shifted symplectic structure introduced by Pantev, Toën, Vaquié and Vezzosi. This structure gives a quasi-isomorphism between the tangent complex and the twisted cotangent complex of the algebra, and may be viewed as a DG enhancement of Van den Bergh's noncommutative Poincaré duality; it also induces a twisted symplectic structure on its derived representation schemes.

math.RA↗

arXiv2021

Sparse Solutions of a Class of Constrained Optimization Problems

In this paper, we consider a well-known sparse optimization problem that aims to find a sparse solution of a possibly noisy underdetermined system of linear equations. Mathematically, it can be modeled in a unified manner by minimizing $\|\bf{x}\|_p^p$ subject to $\|A\bf{x}-\bf{b}\|_q\leqσ$ for given $A \in \mathbb{R}^{m \times n}$, $\bf{b}\in\mathbb{R}^m$, $σ\geq0$, $0\leq p\leq 1$ and $q \geq 1$. We then study various properties of the optimal solutions of this problem. Specifically, without any condition on the matrix $A$, we provide upper bounds in cardinality and infinity norm for the optimal solutions, and show that all optimal solutions must be on the boundary of the feasible set when $0<p<1$. Moreover, for $q \in \{1,\infty\}$, we show that the problem with $0<p<1$ has a finite number of optimal solutions and prove that there exists $0<p^*<1$ such that the solution set of the problem with any $0<p<p^*$ is contained in the solution set of the problem with $p=0$ and there further exists $0<\bar{p}<p^*$ such that the solution set of the problem with any $0<p\leq\bar{p}$ remains unchanged. An estimation of such $p^*$ is also provided. In addition, to solve the constrained nonconvex non-Lipschitz $L_p$-$L_1$ problem ($0<p<1$ and $q=1$), we propose a smoothing penalty method and show that, under some mild conditions, any cluster point of the sequence generated is a KKT point of our problem. Some numerical examples are given to implicitly illustrate the theoretical results and show the efficiency of the proposed algorithm for the constrained $L_p$-$L_1$ problem under different noises.

math.OC↗

arXiv2021

Deep Self-Adaptive Hashing for Image Retrieval

Hashing technology has been widely used in image retrieval due to its computational and storage efficiency. Recently, deep unsupervised hashing methods have attracted increasing attention due to the high cost of human annotations in the real world and the superiority of deep learning technology. However, most deep unsupervised hashing methods usually pre-compute a similarity matrix to model the pairwise relationship in the pre-trained feature space. Then this similarity matrix would be used to guide hash learning, in which most of the data pairs are treated equivalently. The above process is confronted with the following defects: 1) The pre-computed similarity matrix is inalterable and disconnected from the hash learning process, which cannot explore the underlying semantic information. 2) The informative data pairs may be buried by the large number of less-informative data pairs. To solve the aforementioned problems, we propose a Deep Self-Adaptive Hashing (DSAH) model to adaptively capture the semantic information with two special designs: Adaptive Neighbor Discovery (AND) and Pairwise Information Content (PIC). Firstly, we adopt the AND to initially construct a neighborhood-based similarity matrix, and then refine this initial similarity matrix with a novel update strategy to further investigate the semantic structure behind the learned representation. Secondly, we measure the priorities of data pairs with PIC and assign adaptive weights to them, which is relies on the assumption that more dissimilar data pairs contain more discriminative information for hash learning. Extensive experiments on several datasets demonstrate that the above two technologies facilitate the deep hashing model to achieve superior performance.

cs.CV↗

arXiv2021

Efficient Byzantine-Resilient Stochastic Gradient Desce

Distributed Learning often suffers from Byzantine failures, and there have been a number of works studying the problem of distributed stochastic optimization under Byzantine failures, where only a portion of workers, instead of all the workers in a distributed learning system, compute stochastic gradients at each iteration. These methods, albeit workable under Byzantine failures, have the shortcomings of either a sub-optimal convergence rate or high computation cost. To this end, we propose a new Byzantine-resilient stochastic gradient descent algorithm (BrSGD for short) which is provably robust against Byzantine failures. BrSGD obtains the optimal statistical performance and efficient computation simultaneously. In particular, BrSGD can achieve an order-optimal statistical error rate for strongly convex loss functions. The computation complexity of BrSGD is O(md), where d is the model dimension and m is the number of machines. Experimental results show that BrSGD can obtain competitive results compared with non-Byzantine machines in terms of effectiveness and convergence.

cs.DC↗

arXiv2021

A Riemannian smoothing steepest descent method for non-Lipschitz optimization on submanifolds

In this paper, we propose a Riemannian smoothing steepest descent method to minimize a nonconvex and non-Lipschitz function on submanifolds. The generalized subdifferentials on Riemannian manifold and the Riemannian gradient sub-consistency are defined and discussed. We prove that any accumulation point of the sequence generated by the Riemannian smoothing steepest descent method is a stationary point associated with the smoothing function employed in the method, which is necessary for the local optimality of the original non-Lipschitz problem. Under the Riemannian gradient sub-consistency condition, we also prove that any accumulation point is a Riemannian limiting stationary point of the original non-Lipschitz problem. Numerical experiments are conducted to demonstrate the efficiency of the proposed method.

math.OC↗

arXiv2021

On the Łojasiewicz Exponent of the Quadratic Sphere Constrained Optimization Problem

In this paper, we prove that the global version of the $Ł$ojasiewicz gradient inequality holds for quadratic sphere constrained optimization problem with exponent $θ=\frac{3}{4}$. An example from Ting Kei Pong shows that $θ=\frac{3}{4}$ is tight. This is the first $Ł$ojasiewicz gradient inequality established for the sphere constrained optimization problem with a linear term.

math.OC↗

arXiv2021

Pure Characteristics Demand Models and Distributionally Robust Mathematical Programs with Stochastic Complementarity Constraints

We formulate pure characteristics demand models under uncertainties of probability distributions as distributionally robust mathematical programs with stochastic complementarity constraints (DRMP-SCC). For any fixed first-stage variable and a random realization, the second-stage problem of DRMP-SCC is a monotone linear complementarity problem (LCP). To deal with uncertainties of probability distributions of the involved random variables in the stochastic LCP, we use the distributionally robust approach. Moreover, we propose an approximation problem with regularization and discretization to solve DRMP-SCC, which is a two-stage nonconvex-nonconcave minimax optimization problem. We prove the convergence of the approximation problem to DRMP-SCC regarding the optimal solution sets, optimal values and stationary points as the regularization parameter goes to zero and the sample size goes to infinity. Finally, preliminary numerical results for investigating distributional robustness of pure characteristics demand models are reported to illustrate the effectiveness and efficiency of our approaches.

math.OC↗