arXiv subjects

Zheng Li

Publications and source records attributed to Zheng Li.

At least 361 records · Page 20Linked to original sources

arXiv2022

DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization

Large-scale pre-trained sequence-to-sequence models like BART and T5 achieve state-of-the-art performance on many generative NLP tasks. However, such models pose a great challenge in resource-constrained scenarios owing to their large memory requirements and high latency. To alleviate this issue, we propose to jointly distill and quantize the model, where knowledge is transferred from the full-precision teacher model to the quantized and distilled low-precision student model. Empirical analyses show that, despite the challenging nature of generative tasks, we were able to achieve a 16.5x model footprint compression ratio with little performance drop relative to the full-precision counterparts on multiple summarization and QA datasets. We further pushed the limit of compression ratio to 27.7x and presented the performance-efficiency trade-off for generative tasks using pre-trained models. To the best of our knowledge, this is the first work aiming to effectively distill and quantize sequence-to-sequence pre-trained models for language generation tasks.

cs.CL↗

arXiv2022

Online Knowledge Distillation for Efficient Pose Estimation

Existing state-of-the-art human pose estimation methods require heavy computational resources for accurate predictions. One promising technique to obtain an accurate yet lightweight pose estimator is knowledge distillation, which distills the pose knowledge from a powerful teacher model to a less-parameterized student model. However, existing pose distillation works rely on a heavy pre-trained estimator to perform knowledge transfer and require a complex two-stage learning procedure. In this work, we investigate a novel Online Knowledge Distillation framework by distilling Human Pose structure knowledge in a one-stage manner to guarantee the distillation efficiency, termed OKDHP. Specifically, OKDHP trains a single multi-branch network and acquires the predicted heatmaps from each, which are then assembled by a Feature Aggregation Unit (FAU) as the target heatmaps to teach each branch in reverse. Instead of simply averaging the heatmaps, FAU which consists of multiple parallel transformations with different receptive fields, leverages the multi-scale information, thus obtains target heatmaps with higher-quality. Specifically, the pixel-wise Kullback-Leibler (KL) divergence is utilized to minimize the discrepancy between the target heatmaps and the predicted ones, which enables the student network to learn the implicit keypoint relationship. Besides, an unbalanced OKDHP scheme is introduced to customize the student networks with different compression rates. The effectiveness of our approach is demonstrated by extensive experiments on two common benchmark datasets, MPII and COCO.

cs.CV↗

arXiv2022

RETE: Retrieval-Enhanced Temporal Event Forecasting on Unified Query Product Evolutionary Graph

With the increasing demands on e-commerce platforms, numerous user action history is emerging. Those enriched action records are vital to understand users' interests and intents. Recently, prior works for user behavior prediction mainly focus on the interactions with product-side information. However, the interactions with search queries, which usually act as a bridge between users and products, are still under investigated. In this paper, we explore a new problem named temporal event forecasting, a generalized user behavior prediction task in a unified query product evolutionary graph, to embrace both query and product recommendation in a temporal manner. To fulfill this setting, there involves two challenges: (1) the action data for most users is scarce; (2) user preferences are dynamically evolving and shifting over time. To tackle those issues, we propose a novel Retrieval-Enhanced Temporal Event (RETE) forecasting framework. Unlike existing methods that enhance user representations via roughly absorbing information from connected entities in the whole graph, RETE efficiently and dynamically retrieves relevant entities centrally on each user as high-quality subgraphs, preventing the noise propagation from the densely evolutionary graph structures that incorporate abundant search queries. And meanwhile, RETE autoregressively accumulates retrieval-enhanced user representations from each time step, to capture evolutionary patterns for joint query and product prediction. Empirically, extensive experiments on both the public benchmark and four real-world industrial datasets demonstrate the effectiveness of the proposed RETE method.

cs.IR↗

arXiv2022

Perspective: Ultrafast Imaging of Molecular Dynamics Using Ultrafast Low-Frequency Lasers, X-ray Free Electron Laser and Electron Pulses

The requirement of high space-time resolution and brightness is a great challenge for imaging atomic motion and making molecular movies. Important breakthroughs in ultrabright tabletop laser, x-ray and electron sources have enabled the direct imaging of evolving molecular structures in chemical processes. And recent experimental advances in preparing ultrafast laser and electron pulses equipped molecular imaging with femtosecond time resolution. This Perspectives present an overview of versatile imaging methods of molecular dynamics. High-order harmonic generation imaging and photoelectron diffraction imaging are based on laser-induced ionization and rescattering processes. Coulomb explosion imaging retrieves molecular structural information by detecting the momentum vectors of fragmented ions. Diffraction imaging encodes molecular structural and electronic information in reciprocal space. We also present various applications of these ultrafast imaging methods in resolving laser-induced nuclear and electronic dynamics.

physics.atom-ph↗

arXiv2022

Opportunities of Hybrid Model-based Reinforcement Learning for Cell Therapy Manufacturing Process Control

Driven by the key challenges of cell therapy manufacturing, including high complexity, high uncertainty, and very limited process observations, we propose a hybrid model-based reinforcement learning (RL) to efficiently guide process control. We first create a probabilistic knowledge graph (KG) hybrid model characterizing the risk- and science-based understanding of biomanufacturing process mechanisms and quantifying inherent stochasticity, e.g., batch-to-batch variation. It can capture the key features, including nonlinear reactions, nonstationary dynamics, and partially observed state. This hybrid model can leverage existing mechanistic models and facilitate learning from heterogeneous process data. A computational sampling approach is used to generate posterior samples quantifying model uncertainty. Then, we introduce hybrid model-based Bayesian RL, accounting for both inherent stochasticity and model uncertainty, to guide optimal, robust, and interpretable dynamic decision making. Cell therapy manufacturing examples are used to empirically demonstrate that the proposed framework can outperform the classical deterministic mechanistic model assisted process optimization.

eess.SY↗

arXiv2022

Just Enough, Just in Time, Just for "Me": Fundamental Principles for Engineering IoT-native Software Systems

By seamlessly integrating everyday objects and by changing the way we interact with our surroundings, Internet of Things (IoT) is drastically improving the life quality of households and enhancing the productivity of businesses. Given the unique IoT characteristics, IoT applications have emerged distinctively from the mainstream application types. Inspired by the outlook of a programmable world, we further foresee an IoT-native trend in designing, developing, deploying, and maintaining software systems. However, although the challenges of IoT software projects are frequently discussed, addressing those challenges are still in the "crossing the chasm" period. By participating in a few various IoT projects, we gradually distilled three fundamental principles for engineering IoT-native software systems, such as just enough, just in time, and just for "me". These principles target the challenges that are associated with the most typical features of IoT environments, ranging from resource limits to technology heterogeneity of IoT devices. We expect this research to trigger dedicated efforts, techniques and theories for the topic IoT-native software engineering.

cs.SE↗

arXiv2021

Tri-Hexagonal charge order in kagome metal CsV$_{3}$Sb$_{5}$ revealed by $^{121}$Sb NQR

We report $^{121}$Sb nuclear quadrupole resonance (NQR) measurements on kagome superconductor CsV$_3$Sb$_5$ with $T_{\rm c}=2.5$ K. $^{121}$Sb NQR spectra split after a charge density wave (CDW) transition at $94$ K, which demonstrates a commensurate CDW state. The coexistence of the high temperature phase and the CDW phase between $91$ K and $94$ K manifests that it is a first order phase transition. The CDW order exhibits Tri-Hexagonal deformation with a lateral shift between the adjacent kagome layers, which is consistent with $2 \times 2 \times 2$ superlattice modulation. The superconducting state coexists with CDW order and shows a conventional s-wave behavior in the bulk state.

cond-mat.str-el↗

arXiv2021

Selective quantum Zeno effect of ultracold atom-molecule scattering in dynamic magnetic fields

We demonstrated that final states of ultracold scattering between atom and molecule can be selectively produced using dynamic magnetic fields of multiple frequencies. The mechanism of the dynamic magnetic field control is based on a generalized quantum Zeno effect for the selected scattering channels. In particular, we use an atom-molecule spin flip scattering to show that the transition to the selected final spin projection of the molecule in the inelastic scattering can be suppressed by dynamic modulation of coupling between the Floquet engineered initial and final states.

quant-ph↗

arXiv2021

Ghost imaging with non-Gaussian quantum light

Non-local point-to-point correlations between two photons have been used to produce "ghost" images without placing the camera towards the object. Here we theoretically demonstrated and analyzed the advantage of non-Gaussian quantum light in improving the image quality of ghost imaging system over traditional Gaussian light source. For any squeezing degree, the signal-to-noise ratio (SNR) of the ghost image can be enhanced by the non-Gaussian operations of photon addition and subtraction on the two-mode squeezed light source. We find striking evidence that using non-Gaussian coherent operations, the SNR can be promoted to a high level even within the extremely weak squeezing regime. The resulting insight provides new experimental recipes of quantum imaging using non-Gaussian light for illumination.

quant-ph↗

arXiv2021

Student Helping Teacher: Teacher Evolution via Self-Knowledge Distillation

Knowledge distillation usually transfers the knowledge from a pre-trained cumbersome teacher network to a compact student network, which follows the classical teacher-teaching-student paradigm. Based on this paradigm, previous methods mostly focus on how to efficiently train a better student network for deployment. Different from the existing practices, in this paper, we propose a novel student-helping-teacher formula, Teacher Evolution via Self-Knowledge Distillation (TESKD), where the target teacher (for deployment) is learned with the help of multiple hierarchical students by sharing the structural backbone. The diverse feedback from multiple students allows the teacher to improve itself through the shared feature representations. The effectiveness of our proposed framework is demonstrated by extensive experiments with various network settings on two standard benchmarks including CIFAR-100 and ImageNet. Notably, when trained together with our proposed method, ResNet-18 achieves 79.15% and 71.14% accuracy on CIFAR-100 and ImageNet, outperforming the baseline results by 4.74% and 1.43%, respectively. The code is available at: https://github.com/zhengli427/TESKD.

cs.CV↗

arXiv2021

3D Object Detection Combining Semantic and Geometric Features from Point Clouds

In this paper, we investigate the combination of voxel-based methods and point-based methods, and propose a novel end-to-end two-stage 3D object detector named SGNet for point clouds scenes. The voxel-based methods voxelize the scene to regular grids, which can be processed with the current advanced feature learning frameworks based on convolutional layers for semantic feature learning. Whereas the point-based methods can better extract the geometric feature of the point due to the coordinate reservations. The combination of the two is an effective solution for 3D object detection from point clouds. However, most current methods use a voxel-based detection head with anchors for final classification and localization. Although the preset anchors cover the entire scene, it is not suitable for point clouds detection tasks with larger scenes and multiple categories due to the limitation of voxel size. In this paper, we propose a voxel-to-point module (VTPM) that captures semantic and geometric features. The VTPM is a Voxel-Point-Based Module that finally implements 3D object detection in point space, which is more conducive to the detection of small-size objects and avoids the presets of anchors in inference stage. In addition, a Confidence Adjustment Module (CAM) with the center-boundary-aware confidence attention is proposed to solve the misalignment between the predicted confidence and proposals in the regions of the interest (RoI) selection. The SGNet proposed in this paper has achieved state-of-the-art results for 3D object detection in the KITTI dataset, especially in the detection of small-size objects such as cyclists. Actually, as of September 19, 2021, for KITTI dataset, SGNet ranked 1st in 3D and BEV detection on cyclists with easy difficulty level, and 2nd in the 3D detection of moderate cyclists.

cs.CV↗

arXiv2021

Membership Leakage in Label-Only Exposures

Machine learning (ML) has been widely adopted in various privacy-critical applications, e.g., face recognition and medical image analysis. However, recent research has shown that ML models are vulnerable to attacks against their training data. Membership inference is one major attack in this domain: Given a data sample and model, an adversary aims to determine whether the sample is part of the model's training set. Existing membership inference attacks leverage the confidence scores returned by the model as their inputs (score-based attacks). However, these attacks can be easily mitigated if the model only exposes the predicted label, i.e., the final model decision. In this paper, we propose decision-based membership inference attacks and demonstrate that label-only exposures are also vulnerable to membership leakage. In particular, we develop two types of decision-based attacks, namely transfer attack, and boundary attack. Empirical evaluation shows that our decision-based attacks can achieve remarkable performance, and even outperform the previous score-based attacks in some cases. We further present new insights on the success of membership inference based on quantitative and qualitative analysis, i.e., member samples of a model are more distant to the model's decision boundary than non-member samples. Finally, we evaluate multiple defense mechanisms against our decision-based attacks and show that our two types of attacks can bypass most of these defenses.

cs.LG↗

arXiv2021

QUEACO: Borrowing Treasures from Weakly-labeled Behavior Data for Query Attribute Value Extraction

We study the problem of query attribute value extraction, which aims to identify named entities from user queries as diverse surface form attribute values and afterward transform them into formally canonical forms. Such a problem consists of two phases: {named entity recognition (NER)} and {attribute value normalization (AVN)}. However, existing works only focus on the NER phase but neglect equally important AVN. To bridge this gap, this paper proposes a unified query attribute value extraction system in e-commerce search named QUEACO, which involves both two phases. Moreover, by leveraging large-scale weakly-labeled behavior data, we further improve the extraction performance with less supervision cost. Specifically, for the NER phase, QUEACO adopts a novel teacher-student network, where a teacher network that is trained on the strongly-labeled data generates pseudo-labels to refine the weakly-labeled data for training a student network. Meanwhile, the teacher network can be dynamically adapted by the feedback of the student's performance on strongly-labeled data to maximally denoise the noisy supervisions from the weak labels. For the AVN phase, we also leverage the weakly-labeled query-to-attribute behavior data to normalize surface form attribute values from queries into canonical forms from products. Extensive experiments on a real-world large-scale E-commerce dataset demonstrate the effectiveness of QUEACO.

cs.CL↗

arXiv2021

Quantum state tomography of molecules by ultrafast diffraction

Ultrafast electron diffraction and time-resolved serial crystallography are the basis of the ongoing revolution in capturing at the atomic level of detail the structural dynamics of molecules. However, most experiments employ the classical "ball-and-stick" depictions, and the information of molecular quantum states, such as the density matrix, is missing. Here, we introduce a framework for the preparation and ultrafast coherent diffraction from rotational wave packets of molecules, and we establish a new variant of quantum state tomography for ultrafast electron diffraction to characterize the molecular quantum states. The ability to reconstruct the density matrix of molecules of arbitrary degrees of freedom will provide us with an unprecedentedly clear view of the quantum states of molecules, and enable the visualization of effects dictated by the quantum dynamics of molecules.

quant-ph↗

arXiv2021

OLR 2021 Challenge: Datasets, Rules and Baselines

This paper introduces the sixth Oriental Language Recognition (OLR) 2021 Challenge, which intends to improve the performance of language recognition systems and speech recognition systems within multilingual scenarios. The data profile, four tasks, two baselines, and the evaluation principles are introduced in this paper. In addition to the Language Identification (LID) tasks, multilingual Automatic Speech Recognition (ASR) tasks are introduced to OLR 2021 Challenge for the first time. The challenge this year focuses on more practical and challenging problems, with four tasks: (1) constrained LID, (2) unconstrained LID, (3) constrained multilingual ASR, (4) unconstrained multilingual ASR. Baselines for LID tasks and multilingual ASR tasks are provided, respectively. The LID baseline system is an extended TDNN x-vector model constructed with Pytorch. A transformer-based end-to-end model is provided as the multilingual ASR baseline system. These recipes will be online published, and available for participants to construct their own LID or ASR systems. The baseline results demonstrate that those tasks are rather challenging and deserve more effort to achieve better performance.

cs.CL↗

arXiv2021

Throughput Maximization of UAV Networks

In this paper we study the deployment of multiple unmanned aerial vehicles (UAVs) to form a temporal UAV network for the provisioning of emergent communications to affected people in a disaster zone, where each UAV is equipped with a lightweight base station device and thus can act as an aerial base station for users. Unlike most existing studies that assumed that a UAV can serve all users in its communication range, we observe that both computation and communication capabilities of a single lightweight UAV are very limited, due to various constraints on its size, weight, and power supply. Thus, a single UAV can only provide communication services to a limited number of users. We study a novel problem of deploying $K$ UAVs in the top of a disaster area such that the sum of the data rates of users served by the UAVs is maximized, subject to that (i) the number of users served by each UAV is no greater than its service capacity; and (ii) the communication network induced by the $K$ UAVs is connected. We then propose a $\frac{1-1/e}{\lfloor \sqrt{K} \rfloor}$-approximation algorithm for the problem, improving the current best result of the problem by five times (the best approximation ratio so far is $\frac{1-1/e}{5( \sqrt{K} +1)}$), where $e$ is the base of the natural logarithm. We finally evaluate the algorithm performance via simulation experiments. Experimental results show that the proposed algorithm is very promising. Especially, the solution delivered by the proposed algorithm is up to 12% better than those by existing algorithms.

cs.DC↗

arXiv2021

Disentangle pathways in strong field molecular photoionization byangular distribution of dissociation fragments

In strong field ionization, the pump pulse not only photoionizes the molecule, but also drives efficient population exchanges between its ionic ground and excited states.In this study, we investigated the population dynamics accompanying strong field molecular photoionization, using angular distribution of dissociative fragments after ionization.Our results reveal that the first and higher order processes of the post-ionization population redistribution mechanism (PPRM) in the ion core can be disentangled and classified by {its} angle-resolved kinetic energy release (KER) spectra.We demonstrate that the imprints of PPRM in the KER spectra can be used to determine the branching ratio of the population exchange pathways of different orders, by exploiting the pump intensity dependent variation of the spectra.

physics.chem-ph↗

arXiv2021

Oriental Language Recognition (OLR) 2020: Summary and Analysis

The fifth Oriental Language Recognition (OLR) Challenge focuses on language recognition in a variety of complex environments to promote its development. The OLR 2020 Challenge includes three tasks: (1) cross-channel language identification, (2) dialect identification, and (3) noisy language identification. We choose Cavg as the principle evaluation metric, and the Equal Error Rate (EER) as the secondary metric. There were 58 teams participating in this challenge and one third of the teams submitted valid results. Compared with the best baseline, the Cavg values of Top 1 system for the three tasks were relatively reduced by 82%, 62% and 48%, respectively. This paper describes the three tasks, the database profile, and the final results. We also outline the novel approaches that improve the performance of language recognition systems most significantly, such as the utilization of auxiliary information.

cs.SD↗