arXiv subjects

Lei Li

Publications and source records attributed to Lei Li.

At least 541 records · Page 30Linked to original sources

arXiv2022

Deep Computational Model for the Inference of Ventricular Activation Properties

Patient-specific cardiac computational models are essential for the efficient realization of precision medicine and in-silico clinical trials using digital twins. Cardiac digital twins can provide non-invasive characterizations of cardiac functions for individual patients, and therefore are promising for the patient-specific diagnosis and therapy stratification. However, current workflows for both the anatomical and functional twinning phases, referring to the inference of model anatomy and parameter from clinical data, are not sufficiently efficient, robust, and accurate. In this work, we propose a deep learning based patient-specific computational model, which can fuse both anatomical and electrophysiological information for the inference of ventricular activation properties, i.e., conduction velocities and root nodes. The activation properties can provide a quantitative assessment of cardiac electrophysiological function for the guidance of interventional procedures. We employ the Eikonal model to generate simulated electrocardiogram (ECG) with ground truth properties to train the inference model, where specific patient information has also been considered. For evaluation, we test the model on the simulated data and obtain generally promising results with fast computational time.

cs.CV↗

arXiv2022

Distributional Correlation--Aware Knowledge Distillation for Stock Trading Volume Prediction

Traditional knowledge distillation in classification problems transfers the knowledge via class correlations in the soft label produced by teacher models, which are not available in regression problems like stock trading volume prediction. To remedy this, we present a novel distillation framework for training a light-weight student model to perform trading volume prediction given historical transaction data. Specifically, we turn the regression model into a probabilistic forecasting model, by training models to predict a Gaussian distribution to which the trading volume belongs. The student model can thus learn from the teacher at a more informative distributional level, by matching its predicted distributions to that of the teacher. Two correlational distillation objectives are further introduced to encourage the student to produce consistent pair-wise relationships with the teacher model. We evaluate the framework on a real-world stock volume dataset with two different time window settings. Experiments demonstrate that our framework is superior to strong baseline models, compressing the model size by $5\times$ while maintaining $99.6\%$ prediction accuracy. The extensive analysis further reveals that our framework is more effective than vanilla distillation methods under low-resource scenarios.

q-fin.TR↗

arXiv2022

Completely Spin-Decoupled Geometric Phase of Metasurface

Metasurfaces have provided unprecedented degree of freedom (DOF) in manipulating electromagnetic (EM) waves. Geometric phase can be readily obtained by rotating the meta-atom of metasurfaces. Nevertheless, such geometric phases are usually spin-coupled, with the same magnitude but opposite signs for left_ and right_handed circularly polarized (LCP,RCP) waves. To achieve independent control on LCP and RCP waves, it is crucial to obtain spin-decoupled geometric phases. In this paper, we propose to obtain completely spin-decoupled geometric phases by engineering surface current paths on meta-atoms. Based on the rotational Doppler effect, the rotation manner is firstly analyzed and it is found that the essence of generating geometric phase lies in the rotation of surface current paths on meta-atoms. Since the induced surface currents paths under LCP and RCP waves always start oppositely and are mirror-symmetrical with each other, it is natural that the geometric phases be with the same magnitude and opposite signs when the meta-atoms are rotated. To obtain spin-decoupled geometric phases, the start point of induced surface current under one spin should be rotated by an angle while that under the other spin by another different angle. In this way, LCP and RCP waves can acquire different geometric phase changes and spin-decoupled geometric phase can be imparted by metasurfaces. Proof-of-principle prototypes were designed, fabricated and measured. Both the simulation and experiment results verify spin-decoupled geometric phases. This work provides a robust means of obtaining spin-dependent geometric phase and will further adds up to the metasurface DOF in manipulating EM waves.

physics.class-ph↗

arXiv2022

AutoCellLibX: Automated Standard Cell Library Extension Based on Pattern Mining

Custom standard cell libraries can improve the final quality of the corresponding VLSI designs but properly customizing standard cell libraries remains challenging due to the complex characteristics of the VLSI designs. This paper presents an automatic standard-cell library extension framework, AutoCellLibX. It can find a set of standard cell cluster pattern candidates from the post-technology mapping gate-level netlist, with the consideration of standard cell characteristics and technology mapping constraints, based on our high-efficiency frequent subgraph mining algorithm. Meanwhile, to maximize the area benefit of standard cell customization for the given gate-level netlist, AutoCellLibX includes our proposed pattern combination algorithm which can iteratively find a set of gate-level patterns from numerous candidates as the extension part of the given initial standard cell library. To the best of our knowledge, AutoCellLibX is the first automated standard cell extension framework that closes the optimization loop between the analysis of gate-level netlist and standard cell library customization for VLSI design productivity. The experiments with FreePDK45 library and benchmarks from various domains show that AutoCellLibX can generate the library extension with up to 5 custom standard cells within 1.1 hours for each of the 31 benchmark designs and the resultant extension of the standard cell library can save design area by 4.49% averagely.

cs.AR↗

arXiv2022

Weak precompactness in Banach lattices

We show that the solid hull of every weakly precompact set of a Banach lattice $E$ is weakly precompact if and only if every order interval in $E$ is weakly precompact, or equivalently, if and only if every disjoint weakly compact set is weakly precompact. Some results on the domination property for weakly precompact positive operators are obtained. Among other things, we show that, for a pair of Banach lattices $E$ and $F$ with $E$ $σ$-Dedekind complete, every positive operator from $E$ to $F$ dominated by a weakly precompact operator is weakly precompact if and only if either the norm of $E^{\prime}$ is order continuous or else every order interval in $F$ is weakly precompact.

math.FA↗

arXiv2022

Phase control of localization in the nonlinear two-mode system from harmonic mixing driving: Perturbative analysis and symmetry consideration

In this paper, we present a rigorous analysis of symmetry and underlying physics of the nonlinear two-mode system driven by a harmonic mixing field, by means of multiple scale asymptotic analysis method. The effective description in the framework of the second-order perturbative theory provides an accurate picture for understanding the Floquet eigenspectrum and dynamical features of the nonlinear two-mode system, showing full agreement with the prediction of symmetry considerations. We find that two types of symmetries play significant role in the dynamical features of this model, the mechanism behind which can be interpreted in terms of the effective description. The results are of relevance for the phase control of the atomic localization in Bose-Einstein condensates or switch of the optical signals in nonlinear mediums.

quant-ph↗

arXiv2022

On uniform-in-time diffusion approximation for stochastic gradient descent

The diffusion approximation of stochastic gradient descent (SGD) in current literature is only valid on a finite time interval. In this paper, we establish the uniform-in-time diffusion approximation of SGD, by only assuming that the expected loss is strongly convex and some other mild conditions, without assuming the convexity of each random loss function. The main technique is to establish the exponential decay rates of the derivatives of the solution to the backward Kolmogorov equation. The uniform-in-time approximation allows us to study asymptotic behaviors of SGD via the continuous stochastic differential equation (SDE) even when the random objective function $f(\cdot;ξ)$ is not strongly convex.

stat.ML↗

arXiv2022

On the Impact of Noises in Crowd-Sourced Data for Speech Translation

Training speech translation (ST) models requires large and high-quality datasets. MuST-C is one of the most widely used ST benchmark datasets. It contains around 400 hours of speech-transcript-translation data for each of the eight translation directions. This dataset passes several quality-control filters during creation. However, we find that MuST-C still suffers from three major quality issues: audio-text misalignment, inaccurate translation, and unnecessary speaker's name. What are the impacts of these data quality issues for model development and evaluation? In this paper, we propose an automatic method to fix or filter the above quality issues, using English-German (En-De) translation as an example. Our experiments show that ST models perform better on clean test sets, and the rank of proposed models remains consistent across different test sets. Besides, simply removing misaligned data points from the training set does not lead to a better ST model.

cs.CL↗

arXiv2022

CSI Sensing from Heterogeneous User Feedbacks: A Constrained Phase Retrieval Approach

This paper investigates the downlink channel state information (CSI) sensing in 5G heterogeneous networks composed of user equipments (UEs) with different feedback capabilities. We aim to enhance the CSI accuracy of UEs only affording the low-resolution Type-I codebook. While existing works have demonstrated that the task can be accomplished by solving a phase retrieval (PR) formulation based on the feedback of precoding matrix indicator (PMI) and channel quality indicator (CQI), they need many feedback rounds. In this paper, we propose a novel CSI sensing scheme that can significantly reduce the feedback overhead. Our scheme involves a novel parameter dimension reduction design by exploiting the spatial consistency of wireless channels among nearby UEs, and a constrained PR (CPR) formulation that characterizes the feasible region of CSI by the PMI information. To address the computational challenge due to the non-convexity and the large number of constraints of CPR, we develop a two-stage algorithm that firstly identifies and removes inactive constraints, followed by a fast first-order algorithm. The study is further extended to multi-carrier systems. Extensive tests over DeepMIMO and QuaDriGa datasets showcase that our designs greatly outperform existing methods and achieve the high-resolution Type-II codebook performance with a few rounds of feedback.

cs.IT↗

arXiv2022

Provably Confidential Language Modelling

Large language models are shown to memorize privacy information such as social security numbers in training data. Given the sheer scale of the training corpus, it is challenging to screen and filter these privacy data, either manually or automatically. In this paper, we propose Confidentially Redacted Training (CRT), a method to train language generation models while protecting the confidential segments. We borrow ideas from differential privacy (which solves a related but distinct problem) and show that our method is able to provably prevent unintended memorization by randomizing parts of the training process. Moreover, we show that redaction with an approximately correct screening policy amplifies the confidentiality guarantee. We implement the method for both LSTM and GPT language models. Our experimental results show that the models trained by CRT obtain almost the same perplexity while preserving strong confidentiality.

cs.CL↗

arXiv2022

A splitting Hamiltonian Monte Carlo method for efficient sampling

We propose a splitting Hamiltonian Monte Carlo (SHMC) algorithm, which can be computationally efficient when combined with the random mini-batch strategy. By splitting the potential energy into numerically nonstiff and stiff parts, one makes a proposal using the nonstiff part of $U$, followed by a Metropolis rejection step using the stiff part that is often easy to compute. The splitting allows efficient sampling from systems with singular potentials (or distributions with degenerate points) and/or with multiple potential barriers. In our SHMC algorithm, the proposal only based on the nonstiff part in the splitting is generated by the Hamiltonian dynamics, which can be potentially more efficient than the overdamped Langevin dynamics. We also use random batch strategies to reduce the computational cost to $\mathcal{O}(1)$ per time step in generating the proposals for problems arising from many-body systems and Bayesian inference, and prove that the errors of the Hamiltonian induced by the random batch approximation is $\mathcal{O}(\sqrt{Δt})$ in the strong and $\mathcal{O}(Δt)$ in the weak sense, where $Δt$ is the time step. Numerical experiments are conducted to verify the theoretical results and the computational efficiency of the proposed algorithms in practice.

math.NA↗

arXiv2022

Towards Making the Most of BERT in Neural Machine Translation

GPT-2 and BERT demonstrate the effectiveness of using pre-trained language models (LMs) on various natural language processing tasks. However, LM fine-tuning often suffers from catastrophic forgetting when applied to resource-rich tasks. In this work, we introduce a concerted training framework (CTNMT) that is the key to integrate the pre-trained LMs to neural machine translation (NMT). Our proposed CTNMT consists of three techniques: a) asymptotic distillation to ensure that the NMT model can retain the previous pre-trained knowledge; b) a dynamic switching gate to avoid catastrophic forgetting of pre-trained knowledge; and c) a strategy to adjust the learning paces according to a scheduled policy. Our experiments in machine translation show CTNMT gains of up to 3 BLEU score on the WMT14 English-German language pair which even surpasses the previous state-of-the-art pre-training aided NMT by 1.4 BLEU score. While for the large WMT14 English-French task with 40 millions of sentence-pairs, our base model still significantly improves upon the state-of-the-art Transformer big model by more than 1 BLEU score. The code and model can be downloaded from https://github.com/bytedance/neurst/ tree/master/examples/ctnmt.

cs.CL↗

arXiv2022

LightSeq2: Accelerated Training for Transformer-based Models on GPUs

Transformer-based neural models are used in many AI applications. Training these models is expensive, as it takes huge GPU resources and long duration. It is challenging because typical data like sentences have variable lengths, and Transformer's computation patterns are more complex than convolutional neural networks. Existing systems either only focus on model inference or optimization for only BERT-like encoder models. In this paper, we present LightSeq2, a system to accelerate training for a general family of Transformer models on GPUs. We propose a series of GPU optimization techniques tailored to the specific computation flow and memory access patterns of Transformer models. LightSeq2 supports many model architectures, including BERT (encoder-only), GPT (decoder-only), Transformer (encoder-decoder), and vision Transformer. Our experiments for a variety of models and benchmarks show that LightSeq2 is consistently faster (1.4-3.5x) than previous systems on different GPUs. In particular, it gains 308% training speedup compared with existing systems on a large public machine translation benchmark (WMT14 English-German).

cs.CL↗

arXiv2022

On the Learning of Non-Autoregressive Transformers

Non-autoregressive Transformer (NAT) is a family of text generation models, which aims to reduce the decoding latency by predicting the whole sentences in parallel. However, such latency reduction sacrifices the ability to capture left-to-right dependencies, thereby making NAT learning very challenging. In this paper, we present theoretical and empirical analyses to reveal the challenges of NAT learning and propose a unified perspective to understand existing successes. First, we show that simply training NAT by maximizing the likelihood can lead to an approximation of marginal distributions but drops all dependencies between tokens, where the dropped information can be measured by the dataset's conditional total correlation. Second, we formalize many previous objectives in a unified framework and show that their success can be concluded as maximizing the likelihood on a proxy distribution, leading to a reduced information loss. Empirical studies show that our perspective can explain the phenomena in NAT learning and guide the design of new training methods.

cs.CL↗

arXiv2022

MTG: A Benchmark Suite for Multilingual Text Generation

We introduce MTG, a new benchmark suite for training and evaluating multilingual text generation. It is the first-proposed multilingual multiway text generation dataset with the largest human-annotated data (400k). It includes four generation tasks (story generation, question generation, title generation and text summarization) across five languages (English, German, French, Spanish and Chinese). The multiway setup enables testing knowledge transfer capabilities for a model across languages and tasks. Using MTG, we train and analyze several popular multilingual generation models from different aspects. Our benchmark suite fosters model performance enhancement with more human-annotated parallel data. It provides comprehensive evaluations with diverse generation scenarios. Code and data are available at \url{https://github.com/zide05/MTG}.

cs.CL↗

arXiv2022

Decoupling Predictions in Distributed Learning for Multi-Center Left Atrial MRI Segmentation

Distributed learning has shown great potential in medical image analysis. It allows to use multi-center training data with privacy protection. However, data distributions in local centers can vary from each other due to different imaging vendors, and annotation protocols. Such variation degrades the performance of learning-based methods. To mitigate the influence, two groups of methods have been proposed for different aims, i.e., the global methods and the personalized methods. The former are aimed to improve the performance of a single global model for all test data from unseen centers (known as generic data); while the latter target multiple models for each center (denoted as local data). However, little has been researched to achieve both goals simultaneously. In this work, we propose a new framework of distributed learning that bridges the gap between two groups, and improves the performance for both generic and local data. Specifically, our method decouples the predictions for generic data and local data, via distribution-conditioned adaptation matrices. Results on multi-center left atrial (LA) MRI segmentation showed that our method demonstrated superior performance over existing methods on both generic and local data. Our code is available at https://github.com/key1589745/decouple_predict

eess.IV↗

arXiv2022

Symmetry-breaking-induced multifunctionalities of two-dimensional chromium-based materials for nanoelectronics and clean energy conversion

Structural symmetry-breaking that could lead to exotic physical properties plays a crucial role in determining the functions of a system, especially for two-dimensional (2D) materials. Here we demonstrate that multiple functionalities of 2D chromium-based materials could be achieved by breaking inversion symmetry via replacing Y atoms in one face of pristine CrY (Y=P, As, Sb) monolayers with N atoms, i.e., forming Janus Cr2NY monolayers. The functionalities include spin-gapless, very low work function, inducing carrier doping and catalytic activity, which are predominately ascribed to the large intrinsic dipole of Janus Cr2NY monolayers, making them having great potentials in various applications. Specifically, Cr2NSb is found to be a spin-gapless semiconductor, Cr2NP and Cr2NHPF could simultaneously induce n- and p-type carrier doping for two graphene sheets with different concentrations (forming intrinsic p-n vertical junction), and Cr2NY exhibits excellent electrocatalytic hydrogen evolution activity, even superior to benchmark Pt. The results confirm that breaking symmetry is a promising approach for the rational design of multifunctional 2D materials.

physics.comp-ph↗

arXiv2022

Ergodicity and long-time behavior of the Random Batch Method for interacting particle systems

We study the geometric ergodicity and the long time behavior of the Random Batch Method for interacting particle systems, which exhibits superior numerical performance in recent large-scale scientific computing experiments. We show that for both the interacting particle system (IPS) and the random batch interacting particle system (RB-IPS), the distribution laws converge to their respective invariant distributions exponentially, and the convergence rate does not depend on the number of particles $N$, the time step $τ$ for batch divisions or the batch size $p$. Moreover, the Wasserstein distance between the invariant distributions of the IPS and the RB-IPS is bounded by $O(\sqrtτ)$, showing that the RB-IPS can be used to sample the invariant distribution of the IPS accurately with greatly reduced computational cost.

math.PR↗