arXiv subjects

Wei Li

Publications and source records attributed to Wei Li.

At least 775 records · Page 43Linked to original sources

arXiv2023

Siamese DETR

Recent self-supervised methods are mainly designed for representation learning with the base model, e.g., ResNets or ViTs. They cannot be easily transferred to DETR, with task-specific Transformer modules. In this work, we present Siamese DETR, a Siamese self-supervised pretraining approach for the Transformer architecture in DETR. We consider learning view-invariant and detection-oriented representations simultaneously through two complementary tasks, i.e., localization and discrimination, in a novel multi-view learning framework. Two self-supervised pretext tasks are designed: (i) Multi-View Region Detection aims at learning to localize regions-of-interest between augmented views of the input, and (ii) Multi-View Semantic Discrimination attempts to improve object-level discrimination for each region. The proposed Siamese DETR achieves state-of-the-art transfer performance on COCO and PASCAL VOC detection using different DETR variants in all setups. Code is available at https://github.com/Zx55/SiameseDETR.

cs.CV↗

arXiv2023

Correlational Image Modeling for Self-Supervised Visual Pre-Training

We introduce Correlational Image Modeling (CIM), a novel and surprisingly effective approach to self-supervised visual pre-training. Our CIM performs a simple pretext task: we randomly crop image regions (exemplars) from an input image (context) and predict correlation maps between the exemplars and the context. Three key designs enable correlational image modeling as a nontrivial and meaningful self-supervisory task. First, to generate useful exemplar-context pairs, we consider cropping image regions with various scales, shapes, rotations, and transformations. Second, we employ a bootstrap learning framework that involves online and target encoders. During pre-training, the former takes exemplars as inputs while the latter converts the context. Third, we model the output correlation maps via a simple cross-attention block, within which the context serves as queries and the exemplars offer values and keys. We show that CIM performs on par or better than the current state of the art on self-supervised and transfer benchmarks.

cs.CV↗

arXiv2023

Frame-Level Multi-Label Playing Technique Detection Using Multi-Scale Network and Self-Attention Mechanism

Instrument playing technique (IPT) is a key element of musical presentation. However, most of the existing works for IPT detection only concern monophonic music signals, yet little has been done to detect IPTs in polyphonic instrumental solo pieces with overlapping IPTs or mixed IPTs. In this paper, we formulate it as a frame-level multi-label classification problem and apply it to Guzheng, a Chinese plucked string instrument. We create a new dataset, Guzheng\_Tech99, containing Guzheng recordings and onset, offset, pitch, IPT annotations of each note. Because different IPTs vary a lot in their lengths, we propose a new method to solve this problem using multi-scale network and self-attention. The multi-scale network extracts features from different scales, and the self-attention mechanism applied to the feature maps at the coarsest scale further enhances the long-range feature extraction. Our approach outperforms existing works by a large margin, indicating its effectiveness in IPT detection.

cs.SD↗

arXiv2023

Maximizing the Smallest Eigenvalue of Grounded Laplacian Matrix

For a connected graph $\mathcal{G}=(V,E)$ with $n$ nodes, $m$ edges, and Laplacian matrix $\boldsymbol{\mathit{L}}$, a grounded Laplacian matrix $\boldsymbol{\mathit{L}}(S)$ of $\mathcal{G}$ is a $(n-k) \times (n-k)$ principal submatrix of $\boldsymbol{\mathit{L}}$, obtained from $\boldsymbol{\mathit{L}}$ by deleting $k$ rows and columns corresponding to $k$ selected nodes forming a set $S\subseteq V$. The smallest eigenvalue $λ(S)$ of $\boldsymbol{\mathit{L}}(S)$ plays a pivotal role in various dynamics defined on $\mathcal{G}$. For example, $λ(S)$ characterizes the convergence rate of leader-follower consensus, as well as the effectiveness of a pinning scheme for the pinning control problem, with larger $λ(S)$ corresponding to smaller convergence time or better effectiveness of a pinning scheme. In this paper, we focus on the problem of optimally selecting a subset $S$ of fixed $k \ll n$ nodes, in order to maximize the smallest eigenvalue $λ(S)$ of the grounded Laplacian matrix $\boldsymbol{\mathit{L}}(S)$. We show that this optimization problem is NP-hard and that the objective function is non-submodular but monotone. Due to the difficulty to obtain the optimal solution, we first propose a na\"ıve heuristic algorithm selecting one optimal node at each time for $k$ iterations. Then we propose a fast heuristic scalable algorithm to approximately solve this problem, using derivative matrix, matrix perturbations, and Laplacian solvers as tools. Our na\"ıve heuristic algorithm takes $\tilde{O}(knm)$ time, while the fast greedy heuristic has a nearly linear time complexity of $\tilde{O}(km)$. We also conduct numerous experiments on different networks sized up to one million nodes, demonstrating the superiority of our algorithm in terms of efficiency and effectiveness.

cs.IT↗

arXiv2023

(Non)-projective bounds on gravitational EFT

In this paper we study both projective and non-projective constraints on four-dimensional gravitational effective fields theories implied from unitarity, causality and crossing, assuming perturbative UV completions in $M_{\rm pl}$. We derive bounds on the Wilson coefficients of $R^3$ and $D^{2n}R^4$ from its dispersive representation, utilizing both numerical semi-definite programming and analytic geometry analysis. From the former, we derive projective bounds on ratios of couplings and observe accumulation point spectrum populating the boundary of the allowed region. For the latter we consider the non-projective geometry of the EFThedron, which we relate to the known $L$-moment problem in the literature. This allows us to move beyond positivity and incorporate the upper bound from unitarity of the imaginary parts of partial waves. This leads to sharp bounds on individual coefficients, which are of order unity when normalized with respect to the UV scale. Finally, the non-projective geometry also allows us to derive optimal bounds reflecting assumptions of low-spin dominance, improving previous results. We complement the analytic analysis with a simple linear programming approach that validates the bounds.

hep-th↗

arXiv2023

SeqCo-DETR: Sequence Consistency Training for Self-Supervised Object Detection with Transformers

Self-supervised pre-training and transformer-based networks have significantly improved the performance of object detection. However, most of the current self-supervised object detection methods are built on convolutional-based architectures. We believe that the transformers' sequence characteristics should be considered when designing a transformer-based self-supervised method for the object detection task. To this end, we propose SeqCo-DETR, a novel Sequence Consistency-based self-supervised method for object DEtection with TRansformers. SeqCo-DETR defines a simple but effective pretext by minimizes the discrepancy of the output sequences of transformers with different image views as input and leverages bipartite matching to find the most relevant sequence pairs to improve the sequence-level self-supervised representation learning performance. Furthermore, we provide a mask-based augmentation strategy incorporated with the sequence consistency strategy to extract more representative contextual information about the object for the object detection task. Our method achieves state-of-the-art results on MS COCO (45.8 AP) and PASCAL VOC (64.1 AP), demonstrating the effectiveness of our approach.

cs.CV↗

arXiv2023

An ASR-free Fluency Scoring Approach with Self-Supervised Learning

A typical fluency scoring system generally relies on an automatic speech recognition (ASR) system to obtain time stamps in input speech for either the subsequent calculation of fluency-related features or directly modeling speech fluency with an end-to-end approach. This paper describes a novel ASR-free approach for automatic fluency assessment using self-supervised learning (SSL). Specifically, wav2vec2.0 is used to extract frame-level speech features, followed by K-means clustering to assign a pseudo label (cluster index) to each frame. A BLSTM-based model is trained to predict an utterance-level fluency score from frame-level SSL features and the corresponding cluster indexes. Neither speech transcription nor time stamp information is required in the proposed system. It is ASR-free and can potentially avoid the ASR errors effect in practice. Experimental results carried out on non-native English databases show that the proposed approach significantly improves the performance in the "open response" scenario as compared to previous methods and matches the recently reported performance in the "read aloud" scenario.

eess.AS↗

arXiv2023

Leveraging phone-level linguistic-acoustic similarity for utterance-level pronunciation scoring

Recent studies on pronunciation scoring have explored the effect of introducing phone embeddings as reference pronunciation, but mostly in an implicit manner, i.e., addition or concatenation of reference phone embedding and actual pronunciation of the target phone as the phone-level pronunciation quality representation. In this paper, we propose to use linguistic-acoustic similarity to explicitly measure the deviation of non-native production from its native reference for pronunciation assessment. Specifically, the deviation is first estimated by the cosine similarity between reference phone embedding and corresponding acoustic embedding. Next, a phone-level Goodness of pronunciation (GOP) pre-training stage is introduced to guide this similarity-based learning for better initialization of the aforementioned two embeddings. Finally, a transformer-based hierarchical pronunciation scorer is used to map a sequence of phone embeddings, acoustic embeddings along with their similarity measures to predict the final utterance-level score. Experimental results on the non-native databases suggest that the proposed system significantly outperforms the baselines, where the acoustic and phone embeddings are simply added or concatenated. A further examination shows that the phone embeddings learned in the proposed approach are able to capture linguistic-acoustic attributes of native pronunciation as reference.

eess.AS↗

arXiv2023

Multifunctional Charge Transfer Plasmon Sensors

Charge transfer plasmon (CTP) modes arise when metallic nanoparticles are connected by a conductive junction. These ultatunable plasmonic modes can be actively tuned and broadly modulated from visible to IR regimes, implying their potentials for applications in sensing. This review showcases recent developments in theory and applications of charge transfer plasmon resonances (CTPRs) in various configurations of conductively linked plasmonic nanodimers. In particular, we give a due attention to exploiting ultratunable spectral features of charge transfer plasmon resonances for developing multifunctional plasmonic sensors for bulk, surface, gas and molecular sensing applications. We have also briefly discussed that the charge and energy transfer between two plasmonic nanoparticles linked by sub-nanometer length self-assembled monolayers are of interest for single-molecule detection and molecular electronics. In addition to the well-established surface plasmon resonance and localized surface plasmon resonance based sensing schemes, CTPR sensors may open up a new route in efforts to develop multifunctional sensing technologies.

physics.optics↗

arXiv2023

Ground-state phase diagram of the extended two-leg $t$-$J$ ladder

Inspired by the observation of a robust $d$-wave superconducting phase driven by tuning the next-nearest-neighbor (NNN) electron hopping in recent density matrix renormalization group (DMRG) studies of six- and eight-leg $t$-$J$ model, we systematically study the phase diagram of the two-leg $t$-$J$ ladder with the NNN couplings (including NNN hopping and spin interaction) in a large region of doping level, by means of the DMRG calculations. Upon doping from half filling, we identify the Luther-Emery liquid (LEL) phase, which can be distinguished as the pairing-dominant and charge density-dominant regime by comparing the Luttinger parameter $K_ρ$. With the growing NNN couplings, pairing correlations are enhanced and correspondingly $K_ρ$ increases, driving the system from the density-dominant to the pairing-dominant regime. In the Tomonaga-Luttinger liquid (TLL) phase in the larger doping region, we identify two TLL regimes with different features of charge density correlation. At the quarter filling ($1/2$ doping level), we find that the strong dimer orders of bond energy in the open system actually decay algebraically and thus do not indicate a spontaneous translational symmetry breaking. Our results show that in the LEL phase of the two-leg ladder, the NNN couplings seem to play the similar role as that on the wider $t$-$J$ cylinder, and studies on this more accessible system can be helpful towards understanding the emergence of the remarkable $d$-wave superconducting phase on the wider system.

cond-mat.str-el↗

arXiv2023

DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training

Large-scale pre-trained multi-modal models (e.g., CLIP) demonstrate strong zero-shot transfer capability in many discriminative tasks. Their adaptation to zero-shot image-conditioned text generation tasks has drawn increasing interest. Prior arts approach to zero-shot captioning by either utilizing the existing large language models (e.g., GPT-2) or pre-training the encoder-decoder network in an end-to-end manner. In this work, we propose a simple framework, named DeCap, for zero-shot captioning. We introduce a lightweight visual-aware language decoder. This decoder is both data-efficient and computation-efficient: 1) it only requires the text data for training, easing the burden on the collection of paired data. 2) it does not require end-to-end training. When trained with text-only data, the decoder takes the text embedding extracted from the off-the-shelf CLIP encoder as a prefix embedding. The challenge is that the decoder is trained on the text corpus but at the inference stage, it needs to generate captions based on visual inputs. The modality gap issue is widely observed in multi-modal contrastive models that prevents us from directly taking the visual embedding as the prefix embedding. We propose a training-free mechanism to reduce the modality gap. We project the visual embedding into the CLIP text embedding space, while the projected embedding retains the information of the visual input. Taking the projected embedding as the prefix embedding, the decoder generates high-quality descriptions that match the visual input. The experiments show that DeCap outperforms other zero-shot captioning methods and unpaired captioning methods on the typical image captioning benchmarks, i.e., MSCOCO and NoCaps.

cs.CV↗

arXiv2023

Insight-HXMT and GECAM-C observations of the brightest-of-all-time GRB 221009A

GRB 221009A is the brightest gamma-ray burst ever detected since the discovery of this kind of energetic explosions. However, an accurate measurement of the prompt emission properties of this burst is very challenging due to its exceptional brightness. With joint observations of \textit{Insight}-HXMT and GECAM-C, we made an unprecedentedly accurate measurement of the emission during the first $\sim$1800 s of GRB 221009A, including its precursor, main emission (ME, which dominates the burst in flux), flaring emission and early afterglow, in the hard X-ray to soft gamma-ray band from $\sim$ 10 keV to $\sim$ 6 MeV. Based on the GECAM-C unsaturated data of the ME, we measure a record-breaking isotropic equivalent energy ($E_{\rm iso}$) of $\bf \sim 1.5 \times 10^{55}$ erg, which is about eight times the total rest-mass energy of the Sun. The early afterglow data require a significant jet break between 650 s and 1100 s, most likely at $\sim950$ s from the afterglow starting time $T_{AG}$, which corresponds to a jet opening angle of $\sim {0.7^\circ} \ (η_γn)^{1/8}$, where $n$ is the ambient medium density in units of $\rm cm^{-3}$ and $η_γ$ is the ratio between $γ$-ray energy and afterglow kinetic energy. The beaming-corrected total $γ$-ray energy $E_γ$ is $\sim 1.15 \times10^{51} \ (η_γn)^{1/4}$ erg, which is typical for long GRBs. These results suggest that this GRB may have a special central engine, which could launch and collimate a very narrowly beamed jet with an ordinary energy budget, leading to exceptionally luminous gamma-ray radiation per unit solid angle. Alternatively, more GRBs might have such a narrow and bright beam, which are missed by an unfavorable viewing angle or have been detected without distance measurement.

astro-ph.HE↗

arXiv2023

Zero-Shot Text-to-Parameter Translation for Game Character Auto-Creation

Recent popular Role-Playing Games (RPGs) saw the great success of character auto-creation systems. The bone-driven face model controlled by continuous parameters (like the position of bones) and discrete parameters (like the hairstyles) makes it possible for users to personalize and customize in-game characters. Previous in-game character auto-creation systems are mostly image-driven, where facial parameters are optimized so that the rendered character looks similar to the reference face photo. This paper proposes a novel text-to-parameter translation method (T2P) to achieve zero-shot text-driven game character auto-creation. With our method, users can create a vivid in-game character with arbitrary text description without using any reference photo or editing hundreds of parameters manually. In our method, taking the power of large-scale pre-trained multi-modal CLIP and neural rendering, T2P searches both continuous facial parameters and discrete facial parameters in a unified framework. Due to the discontinuous parameter representation, previous methods have difficulty in effectively learning discrete facial parameters. T2P, to our best knowledge, is the first method that can handle the optimization of both discrete and continuous parameters. Experimental results show that T2P can generate high-quality and vivid game characters with given text prompts. T2P outperforms other SOTA text-to-3D generation methods on both objective evaluations and subjective evaluations.

cs.CV↗

arXiv2023

Phase-Field DeepONet: Physics-informed deep operator neural network for fast simulations of pattern formation governed by gradient flows of free-energy functionals

Recent advances in scientific machine learning have shed light on the modeling of pattern-forming systems. However, simulations of real patterns still incur significant computational costs, which could be alleviated by leveraging large image datasets. Physics-informed machine learning and operator learning are two new emerging and promising concepts for this application. Here, we propose "Phase-Field DeepONet", a physics-informed operator neural network framework that predicts the dynamic responses of systems governed by gradient flows of free-energy functionals. Examples used to validate the feasibility and accuracy of the method include the Allen-Cahn and Cahn-Hilliard equations, as special cases of reactive phase-field models for nonequilibrium thermodynamics of chemical mixtures. This is achieved by incorporating the minimizing movement scheme into the framework, which optimizes and controls how the total free energy of a system evolves, instead of solving the governing equations directly. The trained operator neural networks can work as explicit time-steppers that take the current state as the input and output the next state. This could potentially facilitate fast real-time predictions of pattern-forming dynamical systems, such as phase-separating Li-ion batteries, emulsions, colloidal displays, or biological patterns.

cs.CE↗

arXiv2023

Positive-Negative Equal Contrastive Loss for Semantic Segmentation

The contextual information is critical for various computer vision tasks, previous works commonly design plug-and-play modules and structural losses to effectively extract and aggregate the global context. These methods utilize fine-label to optimize the model but ignore that fine-trained features are also precious training resources, which can introduce preferable distribution to hard pixels (i.e., misclassified pixels). Inspired by contrastive learning in unsupervised paradigm, we apply the contrastive loss in a supervised manner and re-design the loss function to cast off the stereotype of unsupervised learning (e.g., imbalance of positives and negatives, confusion of anchors computing). To this end, we propose Positive-Negative Equal contrastive loss (PNE loss), which increases the latent impact of positive embedding on the anchor and treats the positive as well as negative sample pairs equally. The PNE loss can be directly plugged right into existing semantic segmentation frameworks and leads to excellent performance with neglectable extra computational costs. We utilize a number of classic segmentation methods (e.g., DeepLabV3, HRNetV2, OCRNet, UperNet) and backbone (e.g., ResNet, HRNet, Swin Transformer) to conduct comprehensive experiments and achieve state-of-the-art performance on three benchmark datasets (e.g., Cityscapes, COCO-Stuff and ADE20K). Our code will be publicly available soon.

cs.CV↗

arXiv2023

GANet: Goal Area Network for Motion Forecasting

Predicting the future motion of road participants is crucial for autonomous driving but is extremely challenging due to staggering motion uncertainty. Recently, most motion forecasting methods resort to the goal-based strategy, i.e., predicting endpoints of motion trajectories as conditions to regress the entire trajectories, so that the search space of solution can be reduced. However, accurate goal coordinates are hard to predict and evaluate. In addition, the point representation of the destination limits the utilization of a rich road context, leading to inaccurate prediction results in many cases. Goal area, i.e., the possible destination area, rather than goal coordinate, could provide a more soft constraint for searching potential trajectories by involving more tolerance and guidance. In view of this, we propose a new goal area-based framework, named Goal Area Network (GANet), for motion forecasting, which models goal areas rather than exact goal coordinates as preconditions for trajectory prediction, performing more robustly and accurately. Specifically, we propose a GoICrop (Goal Area of Interest) operator to effectively extract semantic lane features in goal areas and model actors' future interactions, which benefits a lot for future trajectory estimations. GANet ranks the 1st on the leaderboard of Argoverse Challenge among all public literature (till the paper submission), and its source codes will be released.

cs.CV↗

arXiv2023

AIIR-MIX: Multi-Agent Reinforcement Learning Meets Attention Individual Intrinsic Reward Mixing Network

Deducing the contribution of each agent and assigning the corresponding reward to them is a crucial problem in cooperative Multi-Agent Reinforcement Learning (MARL). Previous studies try to resolve the issue through designing an intrinsic reward function, but the intrinsic reward is simply combined with the environment reward by summation in these studies, which makes the performance of their MARL framework unsatisfactory. We propose a novel method named Attention Individual Intrinsic Reward Mixing Network (AIIR-MIX) in MARL, and the contributions of AIIR-MIX are listed as follows:(a) we construct a novel intrinsic reward network based on the attention mechanism to make teamwork more effective. (b) we propose a Mixing network that is able to combine intrinsic and extrinsic rewards non-linearly and dynamically in response to changing conditions of the environment. We compare AIIR-MIX with many State-Of-The-Art (SOTA) MARL methods on battle games in StarCraft II. And the results demonstrate that AIIR-MIX performs admirably and can defeat the current advanced methods on average test win rate. To validate the effectiveness of AIIR-MIX, we conduct additional ablation studies. The results show that AIIR-MIX can dynamically assign each agent a real-time intrinsic reward in accordance with their actual contribution.

cs.LG↗

arXiv2023

A Lightweight and Detector-free 3D Single Object Tracker on Point Clouds

Recent works on 3D single object tracking treat the task as a target-specific 3D detection task, where an off-the-shelf 3D detector is commonly employed for the tracking. However, it is non-trivial to perform accurate target-specific detection since the point cloud of objects in raw LiDAR scans is usually sparse and incomplete. In this paper, we address this issue by explicitly leveraging temporal motion cues and propose DMT, a Detector-free Motion-prediction-based 3D Tracking network that completely removes the usage of complicated 3D detectors and is lighter, faster, and more accurate than previous trackers. Specifically, the motion prediction module is first introduced to estimate a potential target center of the current frame in a point-cloud-free manner. Then, an explicit voting module is proposed to directly regress the 3D box from the estimated target center. Extensive experiments on KITTI and NuScenes datasets demonstrate that our DMT can still achieve better performance (~10% improvement over the NuScenes dataset) and a faster tracking speed (i.e., 72 FPS) than state-of-the-art approaches without applying any complicated 3D detectors. Our code is released at \url{https://github.com/jimmy-dq/DMT}

cs.CV↗