Searcharxiv⌕ Search

arXiv subjects

Jun Li

Publications and source records attributed to Jun Li.

At least 505 records · Page 28Linked to original sources
arXiv2023

Frozen Language Model Helps ECG Zero-Shot Learning

The electrocardiogram (ECG) is one of the most commonly used non-invasive, convenient medical monitoring tools that assist in the clinical diagnosis of heart diseases. Recently, deep learning (DL) techniques, particularly self-supervised learning (SSL), have demonstrated great potential in the classification of ECG. SSL pre-training has achieved competitive performance with only a small amount of annotated data after fine-tuning. However, current SSL methods rely on the availability of annotated data and are unable to predict labels not existing in fine-tuning datasets. To address this challenge, we propose Multimodal ECG-Text Self-supervised pre-training (METS), the first work to utilize the auto-generated clinical reports to guide ECG SSL pre-training. We use a trainable ECG encoder and a frozen language model to embed paired ECG and automatically machine-generated clinical reports separately. The SSL aims to maximize the similarity between paired ECG and auto-generated report while minimize the similarity between ECG and other reports. In downstream classification tasks, METS achieves around 10% improvement in performance without using any annotated data via zero-shot classification, compared to other supervised and SSL baselines that rely on annotated data. Furthermore, METS achieves the highest recall and F1 scores on the MIT-BIH dataset, despite MIT-BIH containing different classes of ECG compared to the pre-trained dataset. The extensive experiments have demonstrated the advantages of using ECG-Text multimodal self-supervised learning in terms of generalizability, effectiveness, and efficiency.

cs.LG↗
arXiv2023

Amplitude-Varying Perturbation for Balancing Privacy and Utility in Federated Learning

While preserving the privacy of federated learning (FL), differential privacy (DP) inevitably degrades the utility (i.e., accuracy) of FL due to model perturbations caused by DP noise added to model updates. Existing studies have considered exclusively noise with persistent root-mean-square amplitude and overlooked an opportunity of adjusting the amplitudes to alleviate the adverse effects of the noise. This paper presents a new DP perturbation mechanism with a time-varying noise amplitude to protect the privacy of FL and retain the capability of adjusting the learning performance. Specifically, we propose a geometric series form for the noise amplitude and reveal analytically the dependence of the series on the number of global aggregations and the $(ε,δ)$-DP requirement. We derive an online refinement of the series to prevent FL from premature convergence resulting from excessive perturbation noise. Another important aspect is an upper bound developed for the loss function of a multi-layer perceptron (MLP) trained by FL running the new DP mechanism. Accordingly, the optimal number of global aggregations is obtained, balancing the learning and privacy. Extensive experiments are conducted using MLP, supporting vector machine, and convolutional neural network models on four public datasets. The contribution of the new DP mechanism to the convergence and accuracy of privacy-preserving FL is corroborated, compared to the state-of-the-art Gaussian noise mechanism with a persistent noise amplitude.

cs.LG↗
arXiv2023

Noisy intermediate-scale quantum computers

Quantum computers have made extraordinary progress over the past decade, and significant milestones have been achieved along the path of pursuing universal fault-tolerant quantum computers. Quantum advantage, the tipping point heralding the quantum era, has been accomplished along with several waves of breakthroughs. Quantum hardware has become more integrated and architectural compared to its toddler days. The controlling precision of various physical systems is pushed beyond the fault-tolerant threshold. Meanwhile, quantum computation research has established a new norm by embracing industrialization and commercialization. The joint power of governments, private investors, and tech companies has significantly shaped a new vibrant environment that accelerates the development of this field, now at the beginning of the noisy intermediate-scale quantum era. Here, we first discuss the progress achieved in the field of quantum computation by reviewing the most important algorithms and advances in the most promising technical routes, and then summarizing the next-stage challenges. Furthermore, we illustrate our confidence that solid foundations have been built for the fault-tolerant quantum computer and our optimism that the emergence of quantum killer applications essential for human society shall happen in the future.

quant-ph↗
arXiv2023

An Improved Pair Method to Probe the Dust Extinction Law

Dust extinction law is crucial to recover the intrinsic energy distribution of celestial objects and infer the characteristics of interstellar dust. Based on the traditional pair method, an improved pair method is proposed to model the dust extinguished spectral energy distribution (SED) of an individual star. Instead of the mathematically parameterizing extinction curves, the extinction curves in this work are directly from the silicate-graphite dust model, so that the dust extinction law can be obtained and the dust properties can be analyzed simultaneously. The ATLAS9 stellar model atmosphere is adopted for the intrinsic SEDs in this work, while the silicate-graphite dust model with a dust size distribution of $dn/da \sim a^{-α}{\rm exp}(-a/a_c),~0.005 < a < 5~μ{\rm m}$ for each component is adopted for the model extinction curves. One typical extinction tracer in the dense region (V410 Anon9) and one in the diffuse region (Cyg OB2 \#12) of the MW are chosen to test the reliability and the practicability of the improved pair method in different stellar environments. The results are consistent with their interstellar environments and are in agreement with the previous observations and studies, which prove that the improved pair method is effective and applicable in different stellar environments. In addition to the reliable extinction results, the derived parameters in the dust model can be used to analyze the dust properties, which cannot be achieved by other methods with the mathematical extinction models. With the improved pair method, the stellar parameters can also be inferred and the extinction law beyond the wavelengths of observed data can be predicted based on the dust model as well.

astro-ph.IM↗
arXiv2023

EfficientFace: An Efficient Deep Network with Feature Enhancement for Accurate Face Detection

In recent years, deep convolutional neural networks (CNN) have significantly advanced face detection. In particular, lightweight CNNbased architectures have achieved great success due to their lowcomplexity structure facilitating real-time detection tasks. However, current lightweight CNN-based face detectors trading accuracy for efficiency have inadequate capability in handling insufficient feature representation, faces with unbalanced aspect ratios and occlusion. Consequently, they exhibit deteriorated performance far lagging behind the deep heavy detectors. To achieve efficient face detection without sacrificing accuracy, we design an efficient deep face detector termed EfficientFace in this study, which contains three modules for feature enhancement. To begin with, we design a novel cross-scale feature fusion strategy to facilitate bottom-up information propagation, such that fusing low-level and highlevel features is further strengthened. Besides, this is conducive to estimating the locations of faces and enhancing the descriptive power of face features. Secondly, we introduce a Receptive Field Enhancement module to consider faces with various aspect ratios. Thirdly, we add an Attention Mechanism module for improving the representational capability of occluded faces. We have evaluated EfficientFace on four public benchmarks and experimental results demonstrate the appealing performance of our method. In particular, our model respectively achieves 95.1% (Easy), 94.0% (Medium) and 90.1% (Hard) on validation set of WIDER Face dataset, which is competitive with heavyweight models with only 1/15 computational costs of the state-of-the-art MogFace detector.

cs.CV↗
arXiv2023

InOR-Net: Incremental 3D Object Recognition Network for Point Cloud Representation

3D object recognition has successfully become an appealing research topic in the real-world. However, most existing recognition models unreasonably assume that the categories of 3D objects cannot change over time in the real-world. This unrealistic assumption may result in significant performance degradation for them to learn new classes of 3D objects consecutively, due to the catastrophic forgetting on old learned classes. Moreover, they cannot explore which 3D geometric characteristics are essential to alleviate the catastrophic forgetting on old classes of 3D objects. To tackle the above challenges, we develop a novel Incremental 3D Object Recognition Network (i.e., InOR-Net), which could recognize new classes of 3D objects continuously via overcoming the catastrophic forgetting on old classes. Specifically, a category-guided geometric reasoning is proposed to reason local geometric structures with distinctive 3D characteristics of each class by leveraging intrinsic category information. We then propose a novel critic-induced geometric attention mechanism to distinguish which 3D geometric characteristics within each class are beneficial to overcome the catastrophic forgetting on old classes of 3D objects, while preventing the negative influence of useless 3D characteristics. In addition, a dual adaptive fairness compensations strategy is designed to overcome the forgetting brought by class imbalance, by compensating biased weights and predictions of the classifier. Comparison experiments verify the state-of-the-art performance of the proposed InOR-Net model on several public point cloud datasets.

cs.CV↗
arXiv2023

Measuring the primordial curvature perturbations from the scalar induced gravitational waves

The scalar induced gravitational waves are produced from primordial curvature perturbations in the second order of perturbations. We constrain the fractional energy density of scalar induced gravitational waves from gravitational waves observations. If there is no detection of the scalar induced gravitational waves, the fractional energy density of scalar induced gravitational waves is constrained by some upper limits. Depends on these upper limits, we can obtain the constraints on the power spectrum of the primordial curvature perturbations. For a power-law scalar power spectrum, the constraints on the power spectrum are affected by adding the upper limit of scalar induced gravitational waves from Square Kilometer Array (SKA). In the standard model, the mean values of the scalar amplitude and the spectral index shift to lower values when SKA is added to the combination of Cosmic Microwave Background (CMB) and Baryon Acoustic Oscillation (BAO) datasets, namely $\ln(10^{10}A_s)=3.038\pm0.013$ and $n_s=0.9589^{+0.0021}_{-0.0011}$ at $68\%$ confidence level. We also consider the effects of the existing ground-based gravitational-wave detectors, the existing Pulsar Timing Arrays (PTAs) and Five-hundred-meter Aperture Spherical radio Telescope (FAST), while the constraints from CMB+BAO datasets are totally within their upper limits of scalar induced gravitational waves. Furthermore, we characterize the scalar fluctuation spectrum in terms of the spectral index $n_s$ and its first two derivatives. We calculate corresponding power spectrum of scalar induced gravitational waves theoretically and give the constraints on the running of the spectral index and the running of the running of the spectral index.

astro-ph.CO↗
arXiv2023

A Multi-modal Deformable Land-air Robot for Complex Environments

Single locomotion robots often struggle to adapt in highly variable or uncertain environments, especially in emergencies. In this paper, a multi-modal deformable robot is introduced that can both fly and drive. Compatibility issues with multi-modal locomotive fusion for this hybrid land-air robot are solved using proposed design conceptions, including power settings, energy selection, and designs of deformable structure. The robot can also automatically transform between land and air modes during 3D planning and tracking. Meanwhile, we proposed a algorithms for evaluation the performance of land-air robots. A series of comparisons and experiments were conducted to demonstrate the robustness and reliability of the proposed structure in complex field environments.

cs.RO↗
arXiv2023

AMD: Adaptive Masked Distillation for Object Detection

As a general model compression paradigm, feature-based knowledge distillation allows the student model to learn expressive features from the teacher counterpart. In this paper, we mainly focus on designing an effective feature-distillation framework and propose a spatial-channel adaptive masked distillation (AMD) network for object detection. More specifically, in order to accurately reconstruct important feature regions, we first perform attention-guided feature masking on the feature map of the student network, such that we can identify the important features via spatially adaptive feature masking instead of random masking in the previous methods. In addition, we employ a simple and efficient module to allow the student network channel to be adaptive, improving its model capability in object perception and detection. In contrast to the previous methods, more crucial object-aware features can be reconstructed and learned from the proposed network, which is conducive to accurate object detection. The empirical experiments demonstrate the superiority of our method: with the help of our proposed distillation method, the student networks report 41.3%, 42.4%, and 42.7% mAP scores when RetinaNet, Cascade Mask-RCNN and RepPoints are respectively used as the teacher framework for object detection, which outperforms the previous state-of-the-art distillation methods including FGD and MGD.

cs.CV↗
arXiv2023

Control-enhanced quantum metrology under Markovian noise

Quantum metrology is supposed to significantly improve the precision of parameter estimation by utilizing suitable quantum resources. However, the predicted precision can be severely distorted by realistic noises. Here, we propose a control-enhanced quantum metrology scheme to defend against these noises for improving the metrology performance. Our scheme can automatically alter the parameter encoding dynamics with adjustable controls, thus leading to optimal resultant states that are less sensitive to the noises under consideration. As a demonstration, we numerically apply it to the problem of frequency estimation under several typical Markovian noise channels. Through comparing our control-enhanced scheme with the standard scheme and the ancilla-assisted scheme, we show that our scheme performs better and can improve the estimation precision up to around one order of magnitude. Furthermore, we conduct a proof-of-principle experiment in nuclear magnetic resonance system to verify the effectiveness of the proposed scheme. The research here is helpful for current quantum platforms to harness the power of quantum metrology in realistic noise environments.

quant-ph↗
arXiv2023

Reversible Column Networks

We propose a new neural network design paradigm Reversible Column Network (RevCol). The main body of RevCol is composed of multiple copies of subnetworks, named columns respectively, between which multi-level reversible connections are employed. Such architectural scheme attributes RevCol very different behavior from conventional networks: during forward propagation, features in RevCol are learned to be gradually disentangled when passing through each column, whose total information is maintained rather than compressed or discarded as other network does. Our experiments suggest that CNN-style RevCol models can achieve very competitive performances on multiple computer vision tasks such as image classification, object detection and semantic segmentation, especially with large parameter budget and large dataset. For example, after ImageNet-22K pre-training, RevCol-XL obtains 88.2% ImageNet-1K accuracy. Given more pre-training data, our largest model RevCol-H reaches 90.0% on ImageNet-1K, 63.8% APbox on COCO detection minival set, 61.0% mIoU on ADE20k segmentation. To our knowledge, it is the best COCO detection and ADE20k segmentation result among pure (static) CNN models. Moreover, as a general macro architecture fashion, RevCol can also be introduced into transformers or other neural networks, which is demonstrated to improve the performances in both computer vision and NLP tasks. We release code and models at https://github.com/megvii-research/RevCol

cs.CV↗
arXiv2023

Structure Flow-Guided Network for Real Depth Super-Resolution

Real depth super-resolution (DSR), unlike synthetic settings, is a challenging task due to the structural distortion and the edge noise caused by the natural degradation in real-world low-resolution (LR) depth maps. These defeats result in significant structure inconsistency between the depth map and the RGB guidance, which potentially confuses the RGB-structure guidance and thereby degrades the DSR quality. In this paper, we propose a novel structure flow-guided DSR framework, where a cross-modality flow map is learned to guide the RGB-structure information transferring for precise depth upsampling. Specifically, our framework consists of a cross-modality flow-guided upsampling network (CFUNet) and a flow-enhanced pyramid edge attention network (PEANet). CFUNet contains a trilateral self-attention module combining both the geometric and semantic correlations for reliable cross-modality flow learning. Then, the learned flow maps are combined with the grid-sampling mechanism for coarse high-resolution (HR) depth prediction. PEANet targets at integrating the learned flow map as the edge attention into a pyramid network to hierarchically learn the edge-focused guidance feature for depth edge refinement. Extensive experiments on real and synthetic DSR datasets verify that our approach achieves excellent performance compared to state-of-the-art methods.

cs.CV↗
arXiv2023

Recurrent Structure Attention Guidance for Depth Super-Resolution

Image guidance is an effective strategy for depth super-resolution. Generally, most existing methods employ hand-crafted operators to decompose the high-frequency (HF) and low-frequency (LF) ingredients from low-resolution depth maps and guide the HF ingredients by directly concatenating them with image features. However, the hand-designed operators usually cause inferior HF maps (e.g., distorted or structurally missing) due to the diverse appearance of complex depth maps. Moreover, the direct concatenation often results in weak guidance because not all image features have a positive effect on the HF maps. In this paper, we develop a recurrent structure attention guided (RSAG) framework, consisting of two important parts. First, we introduce a deep contrastive network with multi-scale filters for adaptive frequency-domain separation, which adopts contrastive networks from large filters to small ones to calculate the pixel contrasts for adaptive high-quality HF predictions. Second, instead of the coarse concatenation guidance, we propose a recurrent structure attention block, which iteratively utilizes the latest depth estimation and the image features to jointly select clear patterns and boundaries, aiming at providing refined guidance for accurate depth recovery. In addition, we fuse the features of HF maps to enhance the edge structures in the decomposed LF maps. Extensive experiments show that our approach obtains superior performance compared with state-of-the-art depth super-resolution methods.

cs.CV↗
arXiv2023

Privacy-Preserving Joint Edge Association and Power Optimization for the Internet of Vehicles via Federated Multi-Agent Reinforcement Learning

Proactive edge association is capable of improving wireless connectivity at the cost of increased handover (HO) frequency and energy consumption, while relying on a large amount of private information sharing required for decision making. In order to improve the connectivity-cost trade-off without privacy leakage, we investigate the privacy-preserving joint edge association and power allocation (JEAPA) problem in the face of the environmental uncertainty and the infeasibility of individual learning. Upon modelling the problem by a decentralized partially observable Markov Decision Process (Dec-POMDP), it is solved by federated multi-agent reinforcement learning (FMARL) through only sharing encrypted training data for federatively learning the policy sought. Our simulation results show that the proposed solution strikes a compelling trade-off, while preserving a higher privacy level than the state-of-the-art solutions.

cs.LG↗
arXiv2023

How Does Traffic Environment Quantitatively Affect the Autonomous Driving Prediction?

An accurate trajectory prediction is crucial for safe and efficient autonomous driving in complex traffic environments. In recent years, artificial intelligence has shown strong capabilities in improving prediction accuracy. However, its characteristics of inexplicability and uncertainty make it challenging to determine the traffic environmental effect on prediction explicitly, posing significant challenges to safety-critical decision-making. To address these challenges, this study proposes a trajectory prediction framework with the epistemic uncertainty estimation ability that outputs high uncertainty when confronting unforeseeable or unknown scenarios. The proposed framework is used to analyze the environmental effect on the prediction algorithm performance. In the analysis, the traffic environment is considered in terms of scenario features and shifts, respectively, where features are divided into kinematic features of a target agent, features of its surrounding traffic participants, and other features. In addition, feature correlation and importance analyses are performed to study the above features' influence on the prediction error and epistemic uncertainty. Further, a cross-dataset case study is conducted using multiple intersection datasets to investigate the impact of unavoidable distributional shifts in the real world on trajectory prediction. The results indicate that the deep ensemble-based method has advantages in improving prediction robustness and estimating epistemic uncertainty. The consistent conclusions are obtained by the feature correlation and importance analyses, including the conclusion that kinematic features of the target agent have relatively strong effects on the prediction error and epistemic uncertainty. Furthermore, the prediction failure caused by distributional shifts and the potential of the deep ensemble-based method are analyzed.

cs.CV↗
arXiv2023

Partially Concatenated Calderbank-Shor-Steane Codes Achieving the Quantum Gilbert-Varshamov Bound Asymptotically

In this paper, we utilize a concatenation scheme to construct new families of quantum error correction codes achieving the quantum Gilbert-Varshamov (GV) bound asymptotically. We concatenate alternant codes with any linear code achieving the classical GV bound to construct Calderbank-Shor-Steane (CSS) codes. We show that the concatenated code can achieve the quantum GV bound asymptotically and can approach the Hashing bound for asymmetric Pauli channels. By combing Steane's enlargement construction of CSS codes, we derive a family of enlarged stabilizer codes achieving the quantum GV bound for enlarged CSS codes asymptotically. As applications, we derive two families of fast encodable and decodable CSS codes with parameters $\mathscr{Q}_1=[[N,Ω(\sqrt{N}),Ω( \sqrt{N})]],$ and $\mathscr{Q}_2=[[N,Ω(N/\log N),Ω(N/\log N)/Ω(\log N)]].$ We show that $\mathscr{Q}_1$ can be encoded very efficiently by circuits of size $O(N)$ and depth $O(\sqrt{N})$. For an input error syndrome, $\mathscr{Q}_1$ can correct any adversarial error of weight up to half the minimum distance bound in $O(N)$ time. $\mathscr{Q}_1$ can also be decoded in parallel in $O(\sqrt{N})$ time by using $O(\sqrt{N})$ classical processors. For an input error syndrome, we proved that $\mathscr{Q}_2$ can correct a linear number of ${X}$-errors with high probability and an almost linear number of ${Z}$-errors in $O(N )$ time. Moreover, $\mathscr{Q}_2$ can be decoded in parallel in $O(\log(N))$ time by using $O(N)$ classical processors.

quant-ph↗
arXiv2022

Curriculum Temperature for Knowledge Distillation

Most existing distillation methods ignore the flexible role of the temperature in the loss function and fix it as a hyper-parameter that can be decided by an inefficient grid search. In general, the temperature controls the discrepancy between two distributions and can faithfully determine the difficulty level of the distillation task. Keeping a constant temperature, i.e., a fixed level of task difficulty, is usually sub-optimal for a growing student during its progressive learning stages. In this paper, we propose a simple curriculum-based technique, termed Curriculum Temperature for Knowledge Distillation (CTKD), which controls the task difficulty level during the student's learning career through a dynamic and learnable temperature. Specifically, following an easy-to-hard curriculum, we gradually increase the distillation loss w.r.t. the temperature, leading to increased distillation difficulty in an adversarial manner. As an easy-to-use plug-in technique, CTKD can be seamlessly integrated into existing knowledge distillation frameworks and brings general improvements at a negligible additional computation cost. Extensive experiments on CIFAR-100, ImageNet-2012, and MS-COCO demonstrate the effectiveness of our method. Our code is available at https://github.com/zhengli97/CTKD.

cs.CV↗
arXiv2022

Robust Sum-Rate Maximization in Transmissive RMS Transceiver-Enabled SWIPT Networks

In this paper, we propose a state-of-the-art downlink communication transceiver design for transmissive reconfigurable metasurface (RMS)-enabled simultaneous wireless information and power transfer (SWIPT) networks. Specifically, a feed antenna is deployed in the transmissive RMS-based transceiver, which can be used to implement beamforming. According to the relationship between wavelength and propagation distance, the spatial propagation models of plane and spherical waves are built. Then, in the case of imperfect channel state information (CSI), we formulate a robust system sum-rate maximization problem that jointly optimizes RMS transmissive coefficient, transmit power allocation, and power splitting ratio design while taking account of the non-linear energy harvesting model and outage probability criterion. Since the coupling of optimization variables, the whole optimization problem is non-convex and cannot be solved directly. Therefore, the alternating optimization (AO) framework is implemented to decompose the non-convex original problem. In detail, the whole problem is divided into three sub-problems to solve. For the non-convexity of the objective function, successive convex approximation (SCA) is used to transform it, and penalty function method and difference-of-convex (DC) programming are applied to deal with the non-convex constraints. Finally, we alternately solve the three sub-problems until the entire optimization problem converges. Numerical results show that our proposed algorithm has convergence and better performance than other benchmark algorithms.

eess.SP↗