arXiv subjects

Wei Zou

Publications and source records attributed to Wei Zou.

At least 55 records · Page 3Linked to original sources

arXiv2024

Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization

Audio-Visual Source Localization (AVSL) aims to locate sounding objects within video frames given the paired audio clips. Existing methods predominantly rely on self-supervised contrastive learning of audio-visual correspondence. Without any bounding-box annotations, they struggle to achieve precise localization, especially for small objects, and suffer from blurry boundaries and false positives. Moreover, the naive semi-supervised method is poor in fully leveraging the information of abundant unlabeled data. In this paper, we propose a novel semi-supervised learning framework for AVSL, namely Dual Mean-Teacher (DMT), comprising two teacher-student structures to circumvent the confirmation bias issue. Specifically, two teachers, pre-trained on limited labeled data, are employed to filter out noisy samples via the consensus between their predictions, and then generate high-quality pseudo-labels by intersecting their confidence maps. The sufficient utilization of both labeled and unlabeled data and the proposed unbiased framework enable DMT to outperform current state-of-the-art methods by a large margin, with CIoU of 90.4% and 48.8% on Flickr-SoundNet and VGG-Sound Source, obtaining 8.9%, 9.6% and 4.6%, 6.4% improvements over self- and semi-supervised methods respectively, given only 3% positional-annotations. We also extend our framework to some existing AVSL methods and consistently boost their performance.

cs.CV↗

arXiv2024

From LLM to Conversational Agent: A Memory Enhanced Architecture with Fine-Tuning of Large Language Models

This paper introduces RAISE (Reasoning and Acting through Scratchpad and Examples), an advanced architecture enhancing the integration of Large Language Models (LLMs) like GPT-4 into conversational agents. RAISE, an enhancement of the ReAct framework, incorporates a dual-component memory system, mirroring human short-term and long-term memory, to maintain context and continuity in conversations. It entails a comprehensive agent construction scenario, including phases like Conversation Selection, Scene Extraction, CoT Completion, and Scene Augmentation, leading to the LLMs Training phase. This approach appears to enhance agent controllability and adaptability in complex, multi-turn dialogues. Our preliminary evaluations in a real estate sales context suggest that RAISE has some advantages over traditional agents, indicating its potential for broader applications. This work contributes to the AI field by providing a robust framework for developing more context-aware and versatile conversational agents.

cs.CL↗

arXiv2023

DUMA: a Dual-Mind Conversational Agent with Fast and Slow Thinking

Inspired by the dual-process theory of human cognition, we introduce DUMA, a novel conversational agent framework that embodies a dual-mind mechanism through the utilization of two generative Large Language Models (LLMs) dedicated to fast and slow thinking respectively. The fast thinking model serves as the primary interface for external interactions and initial response generation, evaluating the necessity for engaging the slow thinking model based on the complexity of the complete response. When invoked, the slow thinking model takes over the conversation, engaging in meticulous planning, reasoning, and tool utilization to provide a well-analyzed response. This dual-mind configuration allows for a seamless transition between intuitive responses and deliberate problem-solving processes based on the situation. We have constructed a conversational agent to handle online inquiries in the real estate industry. The experiment proves that our method balances effectiveness and efficiency, and has a significant improvement compared to the baseline.

cs.CL↗

arXiv2023

Exotic swarming dynamics of high-dimensional swarmalators

Swarmalators are oscillators that can swarm as well as sync via a dynamic balance between their spatial proximity and phase similarity. We present a generalized D-dimensional swarmalator model, which is more realistic and versatile, that captures the self-organizing behaviors of a plethora of real-world collectives. This allows for modeling complicated processes such as flocking, schooling of fish, cell sorting during embryonic development, residential segregation, and opinion dynamics in social groups. We demonstrate its versatility by capturing the manoeuvers of the school of fish and traveling waves of gene expression, both qualitatively and quantitatively, embryonic cell sorting, microrobot collectives, and various life stages of slime mold by a suitable extension of the original model to incorporate appropriate features besides a gallery of its intrinsic self-organizations for various interactions. We expect this high-dimensional model to be potentially useful in describing swarming systems in a wide range of disciplines including physics of active matter, developmental biology, sociology, and engineering.

nlin.AO↗

arXiv2023

ChatHome: Development and Evaluation of a Domain-Specific Language Model for Home Renovation

This paper presents the development and evaluation of ChatHome, a domain-specific language model (DSLM) designed for the intricate field of home renovation. Considering the proven competencies of large language models (LLMs) like GPT-4 and the escalating fascination with home renovation, this study endeavors to reconcile these aspects by generating a dedicated model that can yield high-fidelity, precise outputs relevant to the home renovation arena. ChatHome's novelty rests on its methodology, fusing domain-adaptive pretraining and instruction-tuning over an extensive dataset. This dataset includes professional articles, standard documents, and web content pertinent to home renovation. This dual-pronged strategy is designed to ensure that our model can assimilate comprehensive domain knowledge and effectively address user inquiries. Via thorough experimentation on diverse datasets, both universal and domain-specific, including the freshly introduced "EvalHome" domain dataset, we substantiate that ChatHome not only amplifies domain-specific functionalities but also preserves its versatility.

cs.CL↗

arXiv2023

Solvable Dynamics of Coupled High-Dimensional Generalized Limit-Cycle Oscillators

We introduce a new model consisting of globally coupled high-dimensional generalized limit-cycle oscillators, which explicitly incorporates the role of amplitude dynamics of individual units in the collective dynamics. In the limit of weak coupling, our model reduces to the $D$-dimensional Kuramoto phase model, akin to a similar classic construction of the well-known Kuramoto phase model from weakly coupled two-dimensional limit-cycle oscillators. For the practically important case of $D=3$, the incoherence of the model is rigorously proved to be stable for negative coupling $(K<0)$ but unstable for positive coupling $(K>0)$; the locked states are shown to exist if $K>0$; in particular, the onset of amplitude death is theoretically predicted. For $D\geq2$, the discrete and continuous spectra for both locked states and amplitude death are governed by two general formulas. Our proposed $D$-dimensional model is physically more reasonable, because it is no longer constrained by fixed amplitude dynamics, which puts the recent studies of the $D$-dimensional Kuramoto phase model on a stronger footing by providing a more general framework for $D$-dimensional limit-cycle oscillators.

nlin.AO↗

arXiv2022

Metrics to find a surrogate endpoint of OS in metastatic oncology trials: a simulation study

Surrogate endpoint (SE) for overall survival in cancer patients is essential to improving the efficiency of oncology drug development. In practice, we may discover a new patient level association with survival, based on one or more clinical or biological features, in a discovery cohort; and then measure the trial level association across studies in a meta-analysis to validate the SE. To understand how well various patient level metrics would indicate the eventual trial level association, we considered causal biological trajectories based on bi-exponential functions, modeled the strength of their impact on survival hazards via a parameter α, and simulated the trajectories and survival times in randomized trials simultaneously. We set an early time point in the trials when the trajectory measurement became the SE value. From simulated discovery cohorts, we compared patient level metrics including C index, integrated brier score, and log hazard ratio between SE values and survival times. We assembled multiple simulated studies to enable meta-analyses to estimate the trial level association. Across all the simulation scenarios considered here, we found tight correlations among the three patient level metrics and similar correlations between any of them and the trial level metric. Despite the continual increase in α, both patient and trial level metrics often plateaued together; their association always decreased quickly as α increased. This suggests that incorporating additional biological factors into a composite SE is likely to have diminishing returns on improving both patient level and trial level association.

stat.AP↗

arXiv2022

DSLA: Dynamic smooth label assignment for efficient anchor-free object detection

Anchor-free detectors basically formulate object detection as dense classification and regression. For popular anchor-free detectors, it is common to introduce an individual prediction branch to estimate the quality of localization. The following inconsistencies are observed when we delve into the practices of classification and quality estimation. Firstly, for some adjacent samples which are assigned completely different labels, the trained model would produce similar classification scores. This violates the training objective and leads to performance degradation. Secondly, it is found that detected bounding boxes with higher confidences contrarily have smaller overlaps with the corresponding ground-truth. Accurately localized bounding boxes would be suppressed by less accurate ones in the Non-Maximum Suppression (NMS) procedure. To address the inconsistency problems, the Dynamic Smooth Label Assignment (DSLA) method is proposed. Based on the concept of centerness originally developed in FCOS, a smooth assignment strategy is proposed. The label is smoothed to a continuous value in [0, 1] to make a steady transition between positive and negative samples. Intersection-of-Union (IoU) is predicted dynamically during training and is coupled with the smoothed label. The dynamic smooth label is assigned to supervise the classification branch. Under such supervision, quality estimation branch is naturally merged into the classification branch, which simplifies the architecture of anchor-free detector. Comprehensive experiments are conducted on the MS COCO benchmark. It is demonstrated that, DSLA can significantly boost the detection accuracy by alleviating the above inconsistencies for anchor-free detectors. Our codes are released at https://github.com/YonghaoHe/DSLA.

cs.CV↗

arXiv2022

An efficient dosimetry method with a Faraday cup for small animal, small-field proton irradiation under conventional and ultra-high dose rates

Introduction: We developed and evaluated a method for dose calibration and monitoring under conventional and ultra-high dose rates for small animal experiments with small-field proton beams using a Faraday cup. Methods: We determined a relationship between dose and optical density (OD) of EBT-XD Gafchromic film using scanned 10x10 cm2 proton pencil beams delivered at clinical dose rates; the dose was measured with an Advanced Markus chamber. On a small animal proton irradiation platform, double-scattered pencil beams with 5 or 8 mm diameter brass collimation at conventional and ultra-high dose rates were delivered to the EBT-XD films. The proton fluence charges were collected by a Faraday cup placed downstream from the film. The average of the irradiated film ODs was related to the Faraday cup charges. A conversion from the Faraday cup charge to the average dose of the small-field proton beam was then obtained. Results: The relationship between the small-field average profile dose and Faraday cup charge was established for 10 and 15 Gy mice FLASH experiments. The film OD was found to be independent of dose rate. At small-animal treatments, the Faraday cup readings were conveniently used to QA and monitor the delivered dose and dose rates to the mice under conventional and ultra-high dose rates. Conclusion: The dose calibration and monitoring method with Faraday cup for small animal proton FLASH experiments is time-efficient and cost-effective and can be used for irradiations of various small field sizes. The same approach can also be adopted for clinical proton dosimetry for small-field irradiations.

physics.med-ph↗

arXiv2022

Analyzing Robustness of End-to-End Neural Models for Automatic Speech Recognition

We investigate robustness properties of pre-trained neural models for automatic speech recognition. Real life data in machine learning is usually very noisy and almost never clean, which can be attributed to various factors depending on the domain, e.g. outliers, random noise and adversarial noise. Therefore, the models we develop for various tasks should be robust to such kinds of noisy data, which led to the thriving field of robust machine learning. We consider this important issue in the setting of automatic speech recognition. With the increasing popularity of pre-trained models, it's an important question to analyze and understand the robustness of such models to noise. In this work, we perform a robustness analysis of the pre-trained neural models wav2vec2, HuBERT and DistilHuBERT on the LibriSpeech and TIMIT datasets. We use different kinds of noising mechanisms and measure the model performances as quantified by the inference time and the standard Word Error Rate metric. We also do an in-depth layer-wise analysis of the wav2vec2 model when injecting noise in between layers, enabling us to predict at a high level what each layer learns. Finally for this model, we visualize the propagation of errors across the layers and compare how it behaves on clean versus noisy data. Our experiments conform the predictions of Pasad et al. [2021] and also raise interesting directions for future work.

cs.CL↗

arXiv2022

Audio-Visual Wake Word Spotting System For MISP Challenge 2021

This paper presents the details of our system designed for the Task 1 of Multimodal Information Based Speech Processing (MISP) Challenge 2021. The purpose of Task 1 is to leverage both audio and video information to improve the environmental robustness of far-field wake word spotting. In the proposed system, firstly, we take advantage of speech enhancement algorithms such as beamforming and weighted prediction error (WPE) to address the multi-microphone conversational audio. Secondly, several data augmentation techniques are applied to simulate a more realistic far-field scenario. For the video information, the provided region of interest (ROI) is used to obtain visual representation. Then the multi-layer CNN is proposed to learn audio and visual representations, and these representations are fed into our two-branch attention-based network which can be employed for fusion, such as transformer and conformed. The focal loss is used to fine-tune the model and improve the performance significantly. Finally, multiple trained models are integrated by casting vote to achieve our final 0.091 score.

cs.SD↗

arXiv2022

Time Domain Adversarial Voice Conversion for ADD 2022

In this paper, we describe our speech generation system for the first Audio Deep Synthesis Detection Challenge (ADD 2022). Firstly, we build an any-to-many voice conversion (VC) system to convert source speech with arbitrary language content into the target speaker%u2019s fake speech. Then the converted speech generated from VC is post-processed in the time domain to improve the deception ability. The experimental results show that our system has adversarial ability against anti-spoofing detectors with a little compromise in audio quality and speaker similarity. This system ranks top in Track 3.1 in the ADD 2022, showing that our method could also gain good generalization ability against different detectors.

eess.AS↗

arXiv2022

Audio Deep Fake Detection System with Neural Stitching for ADD 2022

This paper describes our best system and methodology for ADD 2022: The First Audio Deep Synthesis Detection Challenge\cite{Yi2022ADD}. The very same system was used for both two rounds of evaluation in Track 3.2 with a similar training methodology. The first round of Track 3.2 data is generated from Text-to-Speech(TTS) or voice conversion (VC) algorithms, while the second round of data consists of generated fake audio from other participants in Track 3.1, aiming to spoof our systems. Our systems use a standard 34-layer ResNet, with multi-head attention pooling \cite{india2019self} to learn the discriminative embedding for fake audio and spoof detection. We further utilize neural stitching to boost the model's generalization capability in order to perform equally well in different tasks, and more details will be explained in the following sessions. The experiments show that our proposed method outperforms all other systems with a 10.1% equal error rate(EER) in Track 3.2.

eess.AS↗

arXiv2022

Role of limiting dispersal on metacommunity stability and persistence

The role of dispersal on the stability and synchrony of a metacommunity is a topic of considerable interest in theoretical ecology. Dispersal is known to promote both synchrony, which enhances the likelihood of extinction, and spatial heterogeneity, which favors the persistence of the population. Several efforts have been made to understand the effect of diverse variants of dispersal in the spatially distributed ecological community. Despite the environmental change strongly affect the dispersal, the effects of controlled dispersal on the metacommunity stability and their persistence remain unknown. We study the influence of limiting the immigration using two patch prey-predator metacommunity at both local and spatial scales. We find that the spread of the inhomogeneous stable steady states (asynchronous states) decreases monotonically upon limiting the predator dispersal. Nevertheless, at the local scale, the spread of the inhomogeneous steady states increases up to a critical value of the limiting factor, favoring the metacommunity persistence, and then starts decreasing for further decrease in the limiting factor with varying local interaction. Interestingly, limiting the prey dispersal promotes inhomogeneous steady states in a large region of the parameter space, thereby increasing the metacommunity persistence both at spatial and local scales. Further, we show similar qualitative dynamics in an entire class of complex networks consisting of a large number of patches. We also deduce various bifurcation curves and stability condition for the inhomogeneous steady states, which we find to agree well with the simulation results. Thus, our findings on the effect of the limiting dispersal can help to develop conservation measures for ecological community.

q-bio.PE↗

arXiv2021

Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning

Self-supervised visual pretraining has shown significant progress recently. Among those methods, SimCLR greatly advanced the state of the art in self-supervised and semi-supervised learning on ImageNet. The input feature representations for speech and visual tasks are both continuous, so it is natural to consider applying similar objective on speech representation learning. In this paper, we propose Speech SimCLR, a new self-supervised objective for speech representation learning. During training, Speech SimCLR applies augmentation on raw speech and its spectrogram. Its objective is the combination of contrastive loss that maximizes agreement between differently augmented samples in the latent space and reconstruction loss of input representation. The proposed method achieved competitive results on speech emotion recognition and speech recognition.

cs.CL↗

arXiv2021

GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio

This paper introduces GigaSpeech, an evolving, multi-domain English speech recognition corpus with 10,000 hours of high quality labeled audio suitable for supervised training, and 40,000 hours of total audio suitable for semi-supervised and unsupervised training. Around 40,000 hours of transcribed audio is first collected from audiobooks, podcasts and YouTube, covering both read and spontaneous speaking styles, and a variety of topics, such as arts, science, sports, etc. A new forced alignment and segmentation pipeline is proposed to create sentence segments suitable for speech recognition training, and to filter out segments with low-quality transcription. For system training, GigaSpeech provides five subsets of different sizes, 10h, 250h, 1000h, 2500h, and 10000h. For our 10,000-hour XL training subset, we cap the word error rate at 4% during the filtering/validation stage, and for all our other smaller training subsets, we cap it at 0%. The DEV and TEST evaluation sets, on the other hand, are re-processed by professional human transcribers to ensure high transcription quality. Baseline systems are provided for popular speech recognition toolkits, namely Athena, ESPnet, Kaldi and Pika.

cs.SD↗

arXiv2021

Semantic Data Augmentation for End-to-End Mandarin Speech Recognition

End-to-end models have gradually become the preferred option for automatic speech recognition (ASR) applications. During the training of end-to-end ASR, data augmentation is a quite effective technique for regularizing the neural networks. This paper proposes a novel data augmentation technique based on semantic transposition of the transcriptions via syntax rules for end-to-end Mandarin ASR. Specifically, we first segment the transcriptions based on part-of-speech tags. Then transposition strategies, such as placing the object in front of the subject or swapping the subject and the object, are applied on the segmented sentences. Finally, the acoustic features corresponding to the transposed transcription are reassembled based on the audio-to-text forced-alignment produced by a pre-trained ASR system. The combination of original data and augmented one is used for training a new ASR system. The experiments are conducted on the Transformer[2] and Conformer[3] based ASR. The results show that the proposed method can give consistent performance gain to the system. Augmentation related issues, such as comparison of different strategies and ratios for data combination are also investigated.

eess.AS↗

arXiv2021

SEPAL: Towards a Large-scale Analysis of SEAndroid Policy Customization

To investigate the status quo of SEAndroid policy customization, we propose SEPAL, a universal tool to automatically retrieve and examine the customized policy rules. SEPAL applies the NLP technique and employs and trains a wide&deep model to quickly and precisely predict whether one rule is unregulated or not.Our evaluation shows SEPAL is effective, practical and scalable. We verify SEPAL outperforms the state of the art approach (i.e., EASEAndroid) by 15% accuracy rate on average. In our experiments, SEPAL successfully identifies 7,111 unregulated policy rules with a low false positive rate from 595,236 customized rules (extracted from 774 Android firmware images of 72 manufacturers). We further discover the policy customization problem is getting worse in newer Android versions (e.g., around 8% for Android 7 and nearly 20% for Android 9), even though more and more efforts are made. Then, we conduct a deep study and discuss why the unregulated rules are introduced and how they can compromise user devices. Last, we report some unregulated rules to seven vendors and so far four of them confirm our findings.

cs.CR↗