arXiv subjects

Dong Wang

Publications and source records attributed to Dong Wang.

At least 343 records · Page 19Linked to original sources

arXiv2024

MaLa-ASR: Multimedia-Assisted LLM-Based ASR

As more and more information-rich data like video become available, utilizing multi-modal auxiliary information to enhance audio tasks has sparked widespread research interest. The recent surge in research on LLM-based audio models provides fresh perspectives for tackling audio tasks. Given that LLM can flexibly ingest multiple inputs, we propose MaLa-ASR, an LLM-based ASR model that can integrate textual keywords extracted from presentation slides to improve recognition of conference content. MaLa-ASR yields average WERs of 9.4% and 11.7% on the L95 and S95 subsets of the SlideSpeech corpus, representing a significant relative WER drop of 27.9% and 44.7% over the baseline model reported in SlideSpeech. MaLa-ASR underscores LLM's strong performance in speech tasks and the capability to integrate auxiliary information conveniently. By adding keywords to the input prompt, the biased word error rate (B-WER) reduces relatively by 46.0% and 44.2%, establishing a new SOTA on this dataset.

eess.AS↗

arXiv2024

SE/BN Adapter: Parametric Efficient Domain Adaptation for Speaker Recognition

Deploying a well-optimized pre-trained speaker recognition model in a new domain often leads to a significant decline in performance. While fine-tuning is a commonly employed solution, it demands ample adaptation data and suffers from parameter inefficiency, rendering it impractical for real-world applications with limited data available for model adaptation. Drawing inspiration from the success of adapters in self-supervised pre-trained models, this paper introduces a SE/BN adapter to address this challenge. By freezing the core speaker encoder and adjusting the feature maps' weights and activation distributions, we introduce a novel adapter utilizing trainable squeeze-and-excitation (SE) blocks and batch normalization (BN) layers, termed SE/BN adapter. Our experiments, conducted using VoxCeleb for pre-training and 4 genres from CN-Celeb for adaptation, demonstrate that the SE/BN adapter offers significant performance improvement over the baseline and competes with the vanilla fine-tuning approach by tuning just 1% of the parameters.

cs.SD↗

arXiv2024

Zero-Shot Fake Video Detection by Audio-Visual Consistency

Recent studies have advocated the detection of fake videos as a one-class detection task, predicated on the hypothesis that the consistency between audio and visual modalities of genuine data is more significant than that of fake data. This methodology, which solely relies on genuine audio-visual data while negating the need for forged counterparts, is thus delineated as a `zero-shot' detection paradigm. This paper introduces a novel zero-shot detection approach anchored in content consistency across audio and video. By employing pre-trained ASR and VSR models, we recognize the audio and video content sequences, respectively. Then, the edit distance between the two sequences is computed to assess whether the claimed video is genuine. Experimental results indicate that, compared to two mainstream approaches based on semantic consistency and temporal consistency, our approach achieves superior generalizability across various deepfake techniques and demonstrates strong robustness against audio-visual perturbations. Finally, state-of-the-art performance gains can be achieved by simply integrating the decision scores of these three systems.

cs.SD↗

arXiv2024

A Comprehensive Investigation on Speaker Augmentation for Speaker Recognition

Data augmentation (DA) has played a pivotal role in the success of deep speaker recognition. Current DA techniques primarily focus on speaker-preserving augmentation, which does not change the speaker trait of the speech and does not create new speakers. Recent research has shed light on the potential of speaker augmentation, which generates new speakers to enrich the training dataset. In this study, we delve into two speaker augmentation approaches: speed perturbation (SP) and vocal tract length perturbation (VTLP). Despite the empirical utilization of both methods, a comprehensive investigation into their efficacy is lacking. Our study, conducted using two public datasets, VoxCeleb and CN-Celeb, revealed that both SP and VTLP are proficient at generating new speakers, leading to significant performance improvements in speaker recognition. Furthermore, they exhibit distinct properties in sensitivity to perturbation factors and data complexity, hinting at the potential benefits of their fusion. Our research underscores the substantial potential of speaker augmentation, highlighting the importance of in-depth exploration and analysis.

cs.SD↗

arXiv2024

Personalized Binomial DAGs Learning with Network Structured Covariates

The causal dependence in data is often characterized by Directed Acyclic Graphical (DAG) models, widely used in many areas. Causal discovery aims to recover the DAG structure using observational data. This paper focuses on causal discovery with multi-variate count data. We are motivated by real-world web visit data, recording individual user visits to multiple websites. Building a causal diagram can help understand user behavior in transitioning between websites, inspiring operational strategy. A challenge in modeling is user heterogeneity, as users with different backgrounds exhibit varied behaviors. Additionally, social network connections can result in similar behaviors among friends. We introduce personalized Binomial DAG models to address heterogeneity and network dependency between observations, which are common in real-world applications. To learn the proposed DAG model, we develop an algorithm that embeds the network structure into a dimension-reduced covariate, learns each node's neighborhood to reduce the DAG search space, and explores the variance-mean relation to determine the ordering. Simulations show our algorithm outperforms state-of-the-art competitors in heterogeneous data. We demonstrate its practical usefulness on a real-world web visit dataset.

cs.LG↗

arXiv2024

SRRT: Exploring Search Region Regulation for Visual Object Tracking

The dominant trackers generate a fixed-size rectangular region based on the previous prediction or initial bounding box as the model input, i.e., search region. While this manner obtains promising tracking efficiency, a fixed-size search region lacks flexibility and is likely to fail in some cases, e.g., fast motion and distractor interference. Trackers tend to lose the target object due to the limited search region or experience interference from distractors due to the excessive search region. Drawing inspiration from the pattern humans track an object, we propose a novel tracking paradigm, called Search Region Regulation Tracking (SRRT) that applies a small eyereach when the target is captured and zooms out the search field when the target is about to be lost. SRRT applies a proposed search region regulator to estimate an optimal search region dynamically for each frame, by which the tracker can flexibly respond to transient changes in the location of object occurrences. To adapt the object's appearance variation during online tracking, we further propose a lockingstate determined updating strategy for reference frame updating. The proposed SRRT is concise without bells and whistles, yet achieves evident improvements and competitive results with other state-of-the-art trackers on eight benchmarks. On the large-scale LaSOT benchmark, SRRT improves SiamRPN++ and TransT with absolute gains of 4.6% and 3.1% in terms of AUC. The code and models will be released.

cs.CV↗

arXiv2024

Evaluating extractable work of quantum batteries via entropic uncertainty relations

In this study, we investigate the effectiveness of entropic uncertainty relations (EURs) in discerning the energy variation in quantum batteries (QBs) modelled by battery-charger-field in the presence of bosonic and fermionic reservoirs. Our results suggest that the extractable works (exergy and ergotropy) have versatile characteristics in different scenarios, resulting in a complex relationship between tightness and extractable work. It is worth noting that the tightness of the lower bound of entropic uncertainty can be a good indicator for energy conversion efficiency in charging QBs. Furthermore, we disclose how the EUR including uncertainty and lower bound contributes to energy conversion efficiency in the QB system. It is believed that these findings will be beneficial for better understanding the role of quantum uncertainty in evaluating quantum battery performance.

quant-ph↗

arXiv2024

Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters

Continual learning can empower vision-language models to continuously acquire new knowledge, without the need for access to the entire historical dataset. However, mitigating the performance degradation in large-scale models is non-trivial due to (i) parameter shifts throughout lifelong learning and (ii) significant computational burdens associated with full-model tuning. In this work, we present a parameter-efficient continual learning framework to alleviate long-term forgetting in incremental learning with vision-language models. Our approach involves the dynamic expansion of a pre-trained CLIP model, through the integration of Mixture-of-Experts (MoE) adapters in response to new tasks. To preserve the zero-shot recognition capability of vision-language models, we further introduce a Distribution Discriminative Auto-Selector (DDAS) that automatically routes in-distribution and out-of-distribution inputs to the MoE Adapter and the original CLIP, respectively. Through extensive experiments across various settings, our proposed method consistently outperforms previous state-of-the-art approaches while concurrently reducing parameter training burdens by 60%. Our code locates at https://github.com/JiazuoYu/MoE-Adapters4CL

cs.CV↗

arXiv2024

MOSEAC: Streamlined Variable Time Step Reinforcement Learning

Traditional reinforcement learning (RL) methods typically employ a fixed control loop, where each cycle corresponds to an action. This rigidity poses challenges in practical applications, as the optimal control frequency is task-dependent. A suboptimal choice can lead to high computational demands and reduced exploration efficiency. Variable Time Step Reinforcement Learning (VTS-RL) addresses these issues by using adaptive frequencies for the control loop, executing actions only when necessary. This approach, rooted in reactive programming principles, reduces computational load and extends the action space by including action durations. However, VTS-RL's implementation is often complicated by the need to tune multiple hyperparameters that govern exploration in the multi-objective action-duration space (i.e., balancing task performance and number of time steps to achieve a goal). To overcome these challenges, we introduce the Multi-Objective Soft Elastic Actor-Critic (MOSEAC) method. This method features an adaptive reward scheme that adjusts hyperparameters based on observed trends in task rewards during training. This scheme reduces the complexity of hyperparameter tuning, requiring a single hyperparameter to guide exploration, thereby simplifying the learning process and lowering deployment costs. We validate the MOSEAC method through simulations in a Newtonian kinematics environment, demonstrating high task and training performance with fewer time steps, ultimately lowering energy consumption. This validation shows that MOSEAC streamlines RL algorithm deployment by automatically tuning the agent control loop frequency using a single parameter. Its principles can be applied to enhance any RL algorithm, making it a versatile solution for various applications.

cs.LG↗

arXiv2024

Learning Manipulation by Predicting Interaction

Representation learning approaches for robotic manipulation have boomed in recent years. Due to the scarcity of in-domain robot data, prevailing methodologies tend to leverage large-scale human video datasets to extract generalizable features for visuomotor policy learning. Despite the progress achieved, prior endeavors disregard the interactive dynamics that capture behavior patterns and physical interaction during the manipulation process, resulting in an inadequate understanding of the relationship between objects and the environment. To this end, we propose a general pre-training pipeline that learns Manipulation by Predicting the Interaction (MPI) and enhances the visual representation.Given a pair of keyframes representing the initial and final states, along with language instructions, our algorithm predicts the transition frame and detects the interaction object, respectively. These two learning objectives achieve superior comprehension towards "how-to-interact" and "where-to-interact". We conduct a comprehensive evaluation of several challenging robotic tasks.The experimental results demonstrate that MPI exhibits remarkable improvement by 10% to 64% compared with previous state-of-the-art in real-world robot platforms as well as simulation environments. Code and checkpoints are publicly shared at https://github.com/OpenDriveLab/MPI.

cs.RO↗

arXiv2024

Subspace-Configurable Networks

While the deployment of deep learning models on edge devices is increasing, these models often lack robustness when faced with dynamic changes in sensed data. This can be attributed to sensor drift, or variations in the data compared to what was used during offline training due to factors such as specific sensor placement or naturally changing sensing conditions. Hence, achieving the desired robustness necessitates the utilization of either an invariant architecture or specialized training approaches, like data augmentation techniques. Alternatively, input transformations can be treated as a domain shift problem, and solved by post-deployment model adaptation. In this paper, we train a parameterized subspace of configurable networks, where an optimal network for a particular parameter setting is part of this subspace. The obtained subspace is low-dimensional and has a surprisingly simple structure even for complex, non-invertible transformations of the input, leading to an exceptionally high efficiency of subspace-configurable networks (SCNs) when limited storage and computing resources are at stake.

cs.LG↗

arXiv2024

Iterative Thresholding Methods for Longest Minimal Length Partitions

In this paper, we introduce two iterative methods for longest minimal length partition problem, which asks whether the disc (ball) is the set maximizing the total perimeter of the shortest partition that divides the total region into sub-regions with given volume proportions, under a volume constraint. The objective functional is approximated by a short-time heat flow using indicator functions of regions and Gaussian convolution. The problem is then represented as a constrained max-min optimization problem. Auction dynamics is used to find the shortest partition in a fixed region, and threshold dynamics is used to update the region. Numerical experiments in two-dimensional and three-dimensional cases are shown with different numbers of partitions, unequal volume proportions, and different initial shapes. The results of both methods are consistent with the conjecture that the disc in two dimensions and the ball in three dimensions are the solution of the longest minimal length partition problem.

math.NA↗

arXiv2024

Suppression of Coherent Synchrotron Radiation-Induced Emittance Growth in a Multi-Bend Deflection Line

Preserving beam quality during the transportation of high-brightness electron beams is a significant and widespread challenge in the design of modern accelerators. The importance of this issue stems from the fact that the quality of the beam at the accelerator's output is crucial for various applications, including particle colliders, free-electron lasers, and synchrotron radiation sources. The coherent synchrotron radiation (CSR) effect can degrade beam quality when a bunch is deflected. Therefore, developing a structure that effectively suppresses the CSR effect, especially for the short bunches is critically important. This involves protecting both the transverse emittance and the longitudinal profile to ensure the production of a high-quality beam. In this study, an optimization based on the reverse lattice of the beamline is proposed. This method can simplify the optimization process. Based on this approach, the Quadruple Bend Achromat (QBA) deflection structure has been designed and optimized. Then we have derived a general solution to completely suppress the impact of steady-state CSR on the transverse plane for different topologies of QBA. Furthermore, a general condition is proposed for suppressing displacements caused by CSR in sequence drifts for isochronous structures. Simultaneously, QBA has proven to be the simplest structure that can simultaneously suppress both types of CSR effects. Simulation results for bunches with a peak current of up to $3000A$ show almost no change in transverse emittance for a large angle deflection.

physics.acc-ph↗

arXiv2024

Formation and Evolution of Transient Prominence Bubbles Driven by Erupting Mini-filaments

Prominence bubbles, the dark arch-shaped "voids" below quiescent prominences, are generally believed to be caused by the interaction between the prominences and the slowly-emerging or quasi-stable underlying magnetic loops. However, this scenario could not explain some short-lived bubbles with extremely dynamic properties of evolution. Based on high-resolution H$α$ observations, here we propose that bubbles should be classified into two categories according to their dynamic properties: quasi-steady (Type-I) bubbles and transient (Type-II) bubbles. Type-I bubbles could remain relatively stable and last for several hours, indicating the existence of a quasi-stable magnetic topology, while Type-II bubbles grow and collapse quickly within one hour without stability duration, which are usually associated with erupting mini-filaments. Analysis of several typical Type-II bubbles from different views, especially including an on-disk event, reveals that Type-II bubbles quickly appear and expand at a velocity of $\thicksim$5--25 km s$^{-1}$ accompanied by an erupting mini-filament below. The mini-filament's rising velocity is slightly larger than that of the Type-II bubbles' boundary, which will lead to the collision with each other in a short time, subsequent collapse of Type-II bubbles, and formation of a large plume into the above prominence. We also speculate that only if the angle between the axis of the erupting mini-filament and the line-of-sight is large enough, the interaction between the erupting mini-filament and the overlying prominence could trigger a Type-II bubble with a typical arch-shaped but quickly-expanding bright boundary.

astro-ph.SR↗

arXiv2024

Task Aligned Meta-learning based Augmented Graph for Cold-Start Recommendation

The cold-start problem is a long-standing challenge in recommender systems due to the lack of user-item interactions, which significantly hurts the recommendation effect over new users and items. Recently, meta-learning based methods attempt to learn globally shared prior knowledge across all users, which can be rapidly adapted to new users and items with very few interactions. Though with significant performance improvement, the globally shared parameter may lead to local optimum. Besides, they are oblivious to the inherent information and feature interactions existing in the new users and items, which are critical in cold-start scenarios. In this paper, we propose a Task aligned Meta-learning based Augmented Graph (TMAG) to address cold-start recommendation. Specifically, a fine-grained task aligned constructor is proposed to cluster similar users and divide tasks for meta-learning, enabling consistent optimization direction. Besides, an augmented graph neural network with two graph enhanced approaches is designed to alleviate data sparsity and capture the high-order user-item interactions. We validate our approach on three real-world datasets in various cold-start scenarios, showing the superiority of TMAG over state-of-the-art methods for cold-start recommendation.

cs.IR↗

arXiv2024

Other Tokens Matter: Exploring Global and Local Features of Vision Transformers for Object Re-Identification

Object Re-Identification (Re-ID) aims to identify and retrieve specific objects from images captured at different places and times. Recently, object Re-ID has achieved great success with the advances of Vision Transformers (ViT). However, the effects of the global-local relation have not been fully explored in Transformers for object Re-ID. In this work, we first explore the influence of global and local features of ViT and then further propose a novel Global-Local Transformer (GLTrans) for high-performance object Re-ID. We find that the features from last few layers of ViT already have a strong representational ability, and the global and local information can mutually enhance each other. Based on this fact, we propose a Global Aggregation Encoder (GAE) to utilize the class tokens of the last few Transformer layers and learn comprehensive global features effectively. Meanwhile, we propose the Local Multi-layer Fusion (LMF) which leverages both the global cues from GAE and multi-layer patch tokens to explore the discriminative local representations. Extensive experiments demonstrate that our proposed method achieves superior performance on four object Re-ID benchmarks.

cs.CV↗

arXiv2024

Odd dipole screening in disordered matter

Disordered solids, straddling the solid-fluid boundary, lack a comprehensive continuum mechanical description. They exhibit a complex microstructure wherein multiple meta-stable states exist. Deforming disordered solids induces particles rearrangements enabling the system to transition between meta-stable states. A dramatic consequence of these transitions is that quasistatic deformation cycles modify the reference state, facilitating the storage and release of mechanical energy. Here we develop a continuum mechanical theory of disordered solids, which accounts for the absence of a reference state and the lack of conserved potential energy. Our theory, which introduces a new modulus describing non-conservative mechanical screening, reduces to classical elasticity in the absence of screening. We analytically derive predictions for the deformation field for various perturbations and geometries. While our theory applies to general disordered solids, we focus on a two-dimensional disordered granular system and predict accurately the non-affine displacement fields observed in experiments for both small and large deformations, along with the observable vanishing shear modulus. The new proposed moduli satisfy universal relations that are independent of the specific experimental realization. Our work thus forms the basis of an entirely new family of continuum descriptions of the mechanics of disordered solids.

cond-mat.soft↗

arXiv2024

Off-Policy Primal-Dual Safe Reinforcement Learning

Primal-dual safe RL methods commonly perform iterations between the primal update of the policy and the dual update of the Lagrange Multiplier. Such a training paradigm is highly susceptible to the error in cumulative cost estimation since this estimation serves as the key bond connecting the primal and dual update processes. We show that this problem causes significant underestimation of cost when using off-policy methods, leading to the failure to satisfy the safety constraint. To address this issue, we propose conservative policy optimization, which learns a policy in a constraint-satisfying area by considering the uncertainty in cost estimation. This improves constraint satisfaction but also potentially hinders reward maximization. We then introduce local policy convexification to help eliminate such suboptimality by gradually reducing the estimation uncertainty. We provide theoretical interpretations of the joint coupling effect of these two ingredients and further verify them by extensive experiments. Results on benchmark tasks show that our method not only achieves an asymptotic performance comparable to state-of-the-art on-policy methods while using much fewer samples, but also significantly reduces constraint violation during training. Our code is available at https://github.com/ZifanWu/CAL.

cs.LG↗