arXiv subjects

Yanzhe Zhang

Publications and source records attributed to Yanzhe Zhang.

34 records · Page 2Linked to original sources

arXiv2025

Attacking Voice Anonymization Systems with Augmented Feature and Speaker Identity Difference

This study focuses on the First VoicePrivacy Attacker Challenge within the ICASSP 2025 Signal Processing Grand Challenge, which aims to develop speaker verification systems capable of determining whether two anonymized speech signals are from the same speaker. However, differences between feature distributions of original and anonymized speech complicate this task. To address this challenge, we propose an attacker system that combines Data Augmentation enhanced feature representation and Speaker Identity Difference enhanced classifier to improve verification performance, termed DA-SID. Specifically, data augmentation strategies (i.e., data fusion and SpecAugment) are utilized to mitigate feature distribution gaps, while probabilistic linear discriminant analysis (PLDA) is employed to further enhance speaker identity difference. Our system significantly outperforms the baseline, demonstrating exceptional effectiveness and robustness against various voice anonymization systems, ultimately securing a top-5 ranking in the challenge.

eess.AS↗

arXiv2024

A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration

Recent studies show that collaborating multiple large language model (LLM) powered agents is a promising way for task solving. However, current approaches are constrained by using a fixed number of agents and static communication structures. In this work, we propose automatically selecting a team of agents from candidates to collaborate in a dynamic communication structure toward different tasks and domains. Specifically, we build a framework named Dynamic LLM-Powered Agent Network ($\textbf{DyLAN}$) for LLM-powered agent collaboration, operating a two-stage paradigm: (1) Team Optimization and (2) Task Solving. During the first stage, we utilize an $\textit{agent selection}$ algorithm, based on an unsupervised metric called $\textit{Agent Importance Score}$, enabling the selection of best agents according to their contributions in a preliminary trial, oriented to the given task. Then, in the second stage, the selected agents collaborate dynamically according to the query. Empirically, we demonstrate that DyLAN outperforms strong baselines in code generation, decision-making, general reasoning, and arithmetic reasoning tasks with moderate computational cost. On specific subjects in MMLU, selecting a team of agents in the team optimization stage improves accuracy by up to 25.0% in DyLAN.

cs.CL↗

arXiv2024

Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping

Sketches are a natural and accessible medium for UI designers to conceptualize early-stage ideas. However, existing research on UI/UX automation often requires high-fidelity inputs like Figma designs or detailed screenshots, limiting accessibility and impeding efficient design iteration. To bridge this gap, we introduce Sketch2Code, a benchmark that evaluates state-of-the-art Vision Language Models (VLMs) on automating the conversion of rudimentary sketches into webpage prototypes. Beyond end-to-end benchmarking, Sketch2Code supports interactive agent evaluation that mimics real-world design workflows, where a VLM-based agent iteratively refines its generations by communicating with a simulated user, either passively receiving feedback instructions or proactively asking clarification questions. We comprehensively analyze ten commercial and open-source models, showing that Sketch2Code is challenging for existing VLMs; even the most capable models struggle to accurately interpret sketches and formulate effective questions that lead to steady improvement. Nevertheless, a user study with UI/UX experts reveals a significant preference for proactive question-asking over passive feedback reception, highlighting the need to develop more effective paradigms for multi-turn conversational agents.

cs.CL↗

arXiv2024

Distilling an End-to-End Voice Assistant Without Instruction Training Data

Voice assistants, such as Siri and Google Assistant, typically model audio and text separately, resulting in lost speech information and increased complexity. Recent efforts to address this with end-to-end Speech Large Language Models (LLMs) trained with supervised finetuning (SFT) have led to models ``forgetting" capabilities from text-only LLMs. Our work proposes an alternative paradigm for training Speech LLMs without instruction data, using the response of a text-only LLM to transcripts as self-supervision. Importantly, this process can be performed without annotated responses. We show that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question Answering, Classification, and Translation. Furthermore, we show that DiVA better meets user preferences, achieving a 72\% win rate compared with state-of-the-art models like Qwen 2 Audio, despite using $>$100x less training compute.

cs.CL↗

arXiv2024

Probing the CP Structure of the Top Quark Yukawa at the Future Muon Collider

We study the top-Higgs coupling with a CP violating phase $ξ$ at a future multi-TeV muon collider. We focus on processes that are directly sensitive to the top quark Yukawa coupling: $t\bar{t}h$, $tbhμν$, and $t\bar{t}hν\barν$ with $h\rightarrow b\bar{b}$ and semileptonic top decays. At different energies, different processes dominate the cross section, providing complementary information. At and above an energy of $\mathcal{O}(10)$ TeV, vector boson fusion processes dominate. As we show, in the Standard Model there is destructive interference in the vector boson fusion processes $t\bar{t}hν\barν$ and $tbhμν$ between the top quark Yukawa and Higgs-gauge boson couplings. A CP-violating phase changes this interference, and the cross section measurement is very sensitive to the size of the CP-violating angle. Although we find that the cross sections are measured to $\mathcal{O}(50\%)$ statistical uncertainty at $1σ$, a 10 and 30 TeV muon collider can bound the CP-violating angle $|ξ|\lesssim9.0^\circ$ and $|ξ|\lesssim5.4^\circ$, respectively. However, cross section measurements are insensitive to the sign of the CP-violating angle. To determine that the coupling is truly CP violating, observables sensitive to CP-violation must be measured. We find in the $t\bar{t}h$ process the azimuthal angle between the $t+\bar{t}$ plane and the initial state muon+Higgs plane shows good discrimination for $ξ=\pm0.1π$. For the $tbhμν$ and $t\bar{t}hν\barν$ processes, the operator proportional to $\left(\vec{p}_μ\times\vec{p}_h\right)\cdot \vec{p}_t$ is sensitive to the sign of CP phase $ξ$. From these observables, we construct asymmetry parameters that show good distinction between different values and signs of the CP violating angle.

hep-ph↗

arXiv2024

Best Practices and Lessons Learned on Synthetic Data

The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and high costs. Synthetic data has emerged as a promising solution by generating artificial data that mimics real-world patterns. This paper provides an overview of synthetic data research, discussing its applications, challenges, and future directions. We present empirical evidence from prior art to demonstrate its effectiveness and highlight the importance of ensuring its factuality, fidelity, and unbiasedness. We emphasize the need for responsible use of synthetic data to build more powerful, inclusive, and trustworthy language models.

cs.CL↗

arXiv2024

TRINS: Towards Multimodal Language Models that Can Read

Large multimodal language models have shown remarkable proficiency in understanding and editing images. However, a majority of these visually-tuned models struggle to comprehend the textual content embedded in images, primarily due to the limitation of training data. In this work, we introduce TRINS: a Text-Rich image INStruction dataset, with the objective of enhancing the reading ability of the multimodal large language model. TRINS is built upon LAION using hybrid data annotation strategies that include machine-assisted and human-assisted annotation processes. It contains 39,153 text-rich images, captions, and 102,437 questions. Specifically, we show that the number of words per annotation in TRINS is significantly longer than that of related datasets, providing new challenges. Furthermore, we introduce a simple and effective architecture, called a Language-vision Reading Assistant (LaRA), which is good at understanding textual content within images. LaRA outperforms existing state-of-the-art multimodal large language models on the TRINS dataset, as well as other classical benchmarks. Lastly, we conducted a comprehensive evaluation with TRINS on various text-rich image understanding and generation tasks, demonstrating its effectiveness.

cs.CV↗

arXiv2024

LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

Instruction tuning unlocks the superior capability of Large Language Models (LLM) to interact with humans. Furthermore, recent instruction-following datasets include images as visual inputs, collecting responses for image-based instructions. However, visual instruction-tuned models cannot comprehend textual details within images well. This work enhances the current visual instruction tuning pipeline with text-rich images (e.g., movie posters, book covers, etc.). Specifically, we first use publicly available OCR tools to collect results on 422K text-rich images from the LAION dataset. Moreover, we prompt text-only GPT-4 with recognized texts and image captions to generate 16K conversations, each containing question-answer pairs for text-rich images. By combining our collected data with previous multi-modal instruction-following data, our model, LLaVAR, substantially improves the LLaVA model's capability on text-based VQA datasets (up to 20% accuracy improvement) while achieving an accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following evaluation also demonstrates the improvement of our model on both natural images and text-rich images. Through qualitative analysis, LLaVAR shows promising interaction (e.g., reasoning, writing, and elaboration) skills with humans based on the latest real-world online content that combines text and images. We make our code/data/models publicly available at https://llavar.github.io/.

cs.CV↗

arXiv2023

Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints

The limits of open-ended generative models are unclear, yet increasingly important. What causes them to succeed and what causes them to fail? In this paper, we take a prompt-centric approach to analyzing and bounding the abilities of open-ended generative models. We present a generic methodology of analysis with two challenging prompt constraint types: structural and stylistic. These constraint types are categorized into a set of well-defined constraints that are analyzable by a single prompt. We then systematically create a diverse set of simple, natural, and useful prompts to robustly analyze each individual constraint. Using the GPT-3 text-davinci-002 model as a case study, we generate outputs from our collection of prompts and analyze the model's generative failures. We also show the generalizability of our proposed method on other large models like BLOOM and OPT. Our results and our in-context mitigation strategies reveal open challenges for future research. We have publicly released our code at https://github.com/SALT-NLP/Bound-Cap-LLM.

cs.CL↗

arXiv2023

Auditing Gender Presentation Differences in Text-to-Image Models

Text-to-image models, which can generate high-quality images based on textual input, have recently enabled various content-creation tools. Despite significantly affecting a wide range of downstream applications, the distributions of these generated images are still not fully understood, especially when it comes to the potential stereotypical attributes of different genders. In this work, we propose a paradigm (Gender Presentation Differences) that utilizes fine-grained self-presentation attributes to study how gender is presented differently in text-to-image models. By probing gender indicators in the input text (e.g., "a woman" or "a man"), we quantify the frequency differences of presentation-centric attributes (e.g., "a shirt" and "a dress") through human annotation and introduce a novel metric: GEP. Furthermore, we propose an automatic method to estimate such differences. The automatic GEP metric based on our approach yields a higher correlation with human annotations than that based on existing CLIP scores, consistently across three state-of-the-art text-to-image models. Finally, we demonstrate the generalization ability of our metrics in the context of gender stereotypes related to occupations.

cs.CV↗

arXiv2022

Robustness of Demonstration-based Learning Under Limited Data Scenario

Demonstration-based learning has shown great potential in stimulating pretrained language models' ability under limited data scenario. Simply augmenting the input with some demonstrations can significantly improve performance on few-shot NER. However, why such demonstrations are beneficial for the learning process remains unclear since there is no explicit alignment between the demonstrations and the predictions. In this paper, we design pathological demonstrations by gradually removing intuitively useful information from the standard ones to take a deep dive of the robustness of demonstration-based sequence labeling and show that (1) demonstrations composed of random tokens still make the model a better few-shot learner; (2) the length of random demonstrations and the relevance of random tokens are the main factors affecting the performance; (3) demonstrations increase the confidence of model predictions on captured superficial patterns. We have publicly released our code at https://github.com/SALT-NLP/RobustDemo.

cs.CL↗

arXiv2022

GJ 1252b: A Hot Terrestrial Super-Earth With No Atmosphere

The increasing numbers of rocky, terrestrial exoplanets known to orbit nearby stars (especially M dwarfs) has drawn increased attention to the possibility of studying these planets' surface properties, and atmospheric compositions & escape histories. Here we report the detection of the secondary eclipse of the terrestrial exoplanet GJ1252b using the Spitzer Space Telescope's IRAC2 4.5 micron channel. We measure an eclipse depth of 149(+25/-32) ppm, corresponding to a day-side brightness temperature of 1410(+91/-125) K and consistent with the prediction for no atmosphere. Comparing our measurement to atmospheric models indicates that GJ1252b has a surface pressure of <10 bar, substantially less than Venus. Assuming energy-limited escape, even a 100 bar atmosphere would be lost in <1 Myr, far shorter than estimated age of 3.9+/-0.4 Gyr. The expected mass loss could be overcome by mantle outgassing, but only if the mantle's carbon content were >7% by mass - over two orders of magnitude greater than that found in Earth. We therefore conclude that GJ1252b has no significant atmosphere. Model spectra with granitoid or feldspathic surface composition, but with no atmosphere, are disfavored at >2 sigma. The eclipse occurs just +1.4(+2.8/-1.0) min after orbital phase 0.5, indicating e cos omega=+0.0025(+0.0049/-0.0018), consistent with a circular orbit. Tidal heating is therefore likely to be negligible to GJ1252b's global energy budget. Finally, we also analyze additional, unpublished TESS transit photometry of GJ1252b which improves the precision of the transit ephemeris by a factor of ten, provides a more precise planetary radius of 1.180+/-0.078 R_E, and rules out any transit timing variations with amplitudes <1 min.

astro-ph.EP↗

arXiv2022

Continual Sequence Generation with Adaptive Compositional Modules

Continual learning is essential for real-world deployment when there is a need to quickly adapt the model to new tasks without forgetting knowledge of old tasks. Existing work on continual sequence generation either always reuses existing parameters to learn new tasks, which is vulnerable to catastrophic forgetting on dissimilar tasks, or blindly adds new parameters for every new task, which could prevent knowledge sharing between similar tasks. To get the best of both worlds, in this work, we propose continual sequence generation with adaptive compositional modules to adaptively add modules in transformer architectures and compose both old and new modules for new tasks. We also incorporate pseudo experience replay to facilitate knowledge transfer in those shared modules. Experiment results on various sequences of generation tasks show that our framework can adaptively add modules or reuse modules based on task similarity, outperforming state-of-the-art baselines in terms of both performance and parameter efficiency. We make our code public at https://github.com/GT-SALT/Adaptive-Compositional-Modules.

cs.CL↗

arXiv2022

Leveraging Expert Guided Adversarial Augmentation For Improving Generalization in Named Entity Recognition

Named Entity Recognition (NER) systems often demonstrate great performance on in-distribution data, but perform poorly on examples drawn from a shifted distribution. One way to evaluate the generalization ability of NER models is to use adversarial examples, on which the specific variations associated with named entities are rarely considered. To this end, we propose leveraging expert-guided heuristics to change the entity tokens and their surrounding contexts thereby altering their entity types as adversarial attacks. Using expert-guided heuristics, we augmented the CoNLL 2003 test set and manually annotated it to construct a high-quality challenging set. We found that state-of-the-art NER systems trained on CoNLL 2003 training data drop performance dramatically on our challenging set. By training on adversarial augmented training examples and using mixup for regularization, we were able to significantly improve the performance on the challenging set as well as improve out-of-domain generalization which we evaluated by using OntoNotes data. We have publicly released our dataset and code at https://github.com/GT-SALT/Guided-Adversarial-Augmentation.

cs.CL↗

arXiv2022

Directly Probing the CP-structure of the Higgs-Top Yukawa at HL-LHC and Future Colliders

Constraining the Higgs boson properties is a cornerstone of the LHC program and future colliders. In this Snowmass contribution, we study the potential to directly probe the Higgs-top CP-structure via the $t\bar{t}h$ production at the HL-LHC, 100 TeV FCC and muon colliders. We find the limits on the CP phase ($α$) at 95% CL are $|α| \lesssim 36^\circ$ with dileptonic $t\bar t (h\to b\bar b) $ and $|α| \lesssim 25^\circ$ with combined $t\bar t (h\to γγ) $ at the HL-LHC. The 100 TeV FCC brings a significant improvement in sensitivity with $|α| \lesssim 3^\circ$ for the dileptonic $t\bar t (h\to b\bar b) $, due to the remarkable gain in the signal cross-section and the increased luminosity. At future muon colliders, we find that the bounds with semileptonic $t\bar t (h\to b\bar b) ν\barν$ are $|α| \lesssim 9^\circ$ for 10 TeV and $|α| \lesssim 3^\circ$ for 30 TeV, respectively.

hep-ph↗

arXiv2021

Continual Learning for Text Classification with Information Disentanglement Based Regularization

Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an information disentanglement based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at https://github.com/GT-SALT/IDBR.

cs.CL↗