arXiv subjects

Yu Pan

Publications and source records attributed to Yu Pan.

At least 73 records · Page 4Linked to original sources

arXiv2025

Wearable intelligent throat enables natural speech in stroke patients with dysarthria

Wearable silent speech systems hold significant potential for restoring communication in patients with speech impairments. However, seamless, coherent speech remains elusive, and clinical efficacy is still unproven. Here, we present an AI-driven intelligent throat (IT) system that integrates throat muscle vibrations and carotid pulse signal sensors with large language model (LLM) processing to enable fluent, emotionally expressive communication. The system utilizes ultrasensitive textile strain sensors to capture high-quality signals from the neck area and supports token-level processing for real-time, continuous speech decoding, enabling seamless, delay-free communication. In tests with five stroke patients with dysarthria, IT's LLM agents intelligently corrected token errors and enriched sentence-level emotional and logical coherence, achieving low error rates (4.2% word error rate, 2.9% sentence error rate) and a 55% increase in user satisfaction. This work establishes a portable, intuitive communication platform for patients with dysarthria with the potential to be applied broadly across different neurological conditions and in multi-language support systems.

eess.AS↗

arXiv2025

IDInit: A Universal and Stable Initialization Method for Neural Network Training

Deep neural networks have achieved remarkable accomplishments in practice. The success of these networks hinges on effective initialization methods, which are vital for ensuring stable and rapid convergence during training. Recently, initialization methods that maintain identity transition within layers have shown good efficiency in network training. These techniques (e.g., Fixup) set specific weights to zero to achieve identity control. However, settings of remaining weight (e.g., Fixup uses random values to initialize non-zero weights) will affect the inductive bias that is achieved only by a zero weight, which may be harmful to training. Addressing this concern, we introduce fully identical initialization (IDInit), a novel method that preserves identity in both the main and sub-stem layers of residual networks. IDInit employs a padded identity-like matrix to overcome rank constraints in non-square weight matrices. Furthermore, we show the convergence problem of an identity matrix can be solved by stochastic gradient descent. Additionally, we enhance the universality of IDInit by processing higher-order weights and addressing dead neuron problems. IDInit is a straightforward yet effective initialization method, with improved convergence, stability, and performance across various settings, including large-scale datasets and deep models.

cs.LG↗

arXiv2025

Valley resolved dynamics of phonon bottleneck in semiconductor molybdenum ditelluride

Semiconductor molybdenum ditelluride (2H-MoTe2) possess multiple valleys in the band structure, enriching its physical properties and potentials in applications. The understanding of its multivalley nature of fundamental processes involving population and relaxation of carriers and phonons is still evolving; particularly, the possible phonon bottleneck has not yet been addressed. Here, we investigate the carrier intra- and intervalley scattering and the phonon dynamics in different valleys in photoexcited few-layer 2H-MoTe2, by using the time resolved measurements of optical absorption and electron diffraction, together with the density functional theory calculation and molecular dynamics simulation. The pathways and timescales of carrier relaxation, accompanied with the emissions of optical phonons at the Brillouin zone center and acoustic phonons at the zone border are revealed. We present a couple of approaches to estimate the population of different phonon modes based on the results of optical and electron diffraction measurements, hence quantitatively identify the occurrences of phonon bottleneck located in different valleys. Our findings make possible to construct a comprehensive picture of the complex interactions between carriers and phonons in 2H-MoTe2 with the valley degree of freedom resolved.

cond-mat.mtrl-sci↗

arXiv2025

Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling

Expressive zero-shot voice conversion (VC) is a critical and challenging task that aims to transform the source timbre into an arbitrary unseen speaker while preserving the original content and expressive qualities. Despite recent progress in zero-shot VC, there remains considerable potential for improvements in speaker similarity and speech naturalness. Moreover, existing zero-shot VC systems struggle to fully reproduce paralinguistic information in highly expressive speech, such as breathing, crying, and emotional nuances, limiting their practical applicability. To address these issues, we propose Takin-VC, a novel expressive zero-shot VC framework via adaptive hybrid content encoding and memory-augmented context-aware timbre modeling. Specifically, we introduce an innovative hybrid content encoder that incorporates an adaptive fusion module, capable of effectively integrating quantized features of the pre-trained WavLM and HybridFormer in an implicit manner, so as to extract precise linguistic features while enriching paralinguistic elements. For timbre modeling, we propose advanced memory-augmented and context-aware modules to generate high-quality target timbre features and fused representations that seamlessly align source content with target timbre. To enhance real-time performance, we advocate a conditional flow matching model to reconstruct the Mel-spectrogram of the source speech. Experimental results show that our Takin-VC consistently surpasses state-of-the-art VC systems, achieving notable improvements in terms of speech naturalness, speech expressiveness, and speaker similarity, while offering enhanced inference speed.

cs.SD↗

arXiv2024

Fully and partially distributed Quantum Generalized Benders Decomposition for Unit Commitment Problems

A series of hybrid quantum-classical generalized Benders decomposition (GBD) algorithms are proposed to address unit commitment (UC) problems under centralized, distributed, and partially distributed frameworks. In the centralized approach, the quantum GBD transforms the master problem (MP) into a quadratic unconstrained binary optimization form suitable for quantum computing. For distributed systems, the distributed consensus quantum GBD employs an average consensus strategy to reformulate subproblems into local subproblems. By leveraging the dual information, local cutting planes are constructed to decompose the MP into local master problems (LMPs). This approach reduces the qubit overhead and addresses the partitioning requirements. The consensus-inspired quantum GBD (CIQGBD) and its partially distributed variant, D-CIQGBD are proposed based on optimizing the allocation of relaxation variables directly, the algorithms construct more rational cutting planes, thereby enhancing the minimum eigenenergy gap of the system Hamiltonian during quantum annealing and improving the computational efficiency. Extensive experiments under various UC scenarios validate the performance of the above-mentioned hybrid algorithms. Compared to the classical solver Gurobi, D-CIQGBD demonstrates a speed advantage in solving the security-constrained UC problem on the IEEE-RTS 24-bus system. These results provide new perspectives on leveraging quantum computing for the distributed optimization of power systems.

quant-ph↗

arXiv2024

On the Power of Adaptive Weighted Aggregation in Heterogeneous Federated Learning and Beyond

Federated averaging (FedAvg) is the most fundamental algorithm in Federated learning (FL). Previous theoretical results assert that FedAvg convergence and generalization degenerate under heterogeneous clients. However, recent empirical results show that FedAvg can perform well in many real-world heterogeneous tasks. These results reveal an inconsistency between FL theory and practice that is not fully explained. In this paper, we show that common heterogeneity measures contribute to this inconsistency based on rigorous convergence analysis. Furthermore, we introduce a new measure \textit{client consensus dynamics} and prove that \textit{FedAvg can effectively handle client heterogeneity when an appropriate aggregation strategy is used}. Building on this theoretical insight, we present a simple and effective FedAvg variant termed FedAWARE. Extensive experiments on three datasets and two modern neural network architectures demonstrate that FedAWARE ensures faster convergence and better generalization in heterogeneous client settings. Moreover, our results show that FedAWARE can significantly enhance the generalization performance of advanced FL algorithms when used as a plug-in module.

cs.LG↗

arXiv2024

Simultaneous Multiband Photometry of the Early Optical Afterglow of GRB 240825A with Mephisto

Gamma-ray bursts (GRBs) are the most luminous transients in the universe. The interaction of the relativistic jet with the circumburst medium produces an afterglow and generates multiwavelength emission. In this work, we present simultaneous multiband photometry of GRB~240825A with the Multi-channel Photometric Survey Telescope (Mephisto) and analyze its temporal and spectral properties. The measurement began 128 seconds after the GRB trigger and continued until the fourth day when the afterglow essentially diminished and the measured brightness was close to that of the host galaxy. Based on the multiband light curves in the $uvgriz$ bands, we find that the optical flux density satisfies $F_{ν,{\rm obs}}\propto t^{-1.34}ν^{-2.48}$ with a spectral index of $2.48$ much larger than those of most other GRBs. To reconcile the measured much softer spectral energy distribution (SED) with that predicted by the standard afterglow model, an extra host-galaxy extinction of $E_{B-V}\sim(0.37-0.57)$ mag is required. We interpreted this excess as arising from a dense circumburst medium. We further find that the SED of the optical afterglow hardened as the afterglow decayed and the color excess $E_{B-V}$ decreased $\sim0.26$ mag from 100 seconds to 3000 seconds after the GRB trigger. Finally, we analyze the properties of the host galaxy of GRB~240825A based on data from the SDSS, PanSTARRS and HSC-SSP surveys. For a host redshift of $z=0.659$, the stellar mass and star formation rate of the host galaxy are estimated to be $\log(M_*/M_\odot)=10.0^{+0.3}_{-0.3}$ and $\log({\rm SFR}/M_{\odot}{\rm yr}^{-1})= 0.6^{+0.8}_{-3.3}$, respectively, pointing to a gas-rich, star-forming, medium-size galaxy.

astro-ph.HE↗

arXiv2024

StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching

Zero-shot voice conversion (VC) aims to transfer the timbre from the source speaker to an arbitrary unseen speaker while preserving the original linguistic content. Despite recent advancements in zero-shot VC using language model-based or diffusion-based approaches, several challenges remain: 1) current approaches primarily focus on adapting timbre from unseen speakers and are unable to transfer style and timbre to different unseen speakers independently; 2) these approaches often suffer from slower inference speeds due to the autoregressive modeling methods or the need for numerous sampling steps; 3) the quality and similarity of the converted samples are still not fully satisfactory. To address these challenges, we propose a style controllable zero-shot VC approach named StableVC, which aims to transfer timbre and style from source speech to different unseen target speakers. Specifically, we decompose speech into linguistic content, timbre, and style, and then employ a conditional flow matching module to reconstruct the high-quality mel-spectrogram based on these decomposed features. To effectively capture timbre and style in a zero-shot manner, we introduce a novel dual attention mechanism with an adaptive gate, rather than using conventional feature concatenation. With this non-autoregressive design, StableVC can efficiently capture the intricate timbre and style from different unseen speakers and generate high-quality speech significantly faster than real-time. Experiments demonstrate that our proposed StableVC outperforms state-of-the-art baseline systems in zero-shot VC and achieves flexible control over timbre and style from different unseen speakers. Moreover, StableVC offers approximately 25x and 1.65x faster sampling compared to autoregressive and diffusion-based baselines.

eess.AS↗

arXiv2024

Quantum Discrete Adiabatic Linear Solver based on Block Encoding and Eigenvalue Separator

Linear system solvers are widely used in scientific computing, with the primary goal of solving linear system problems. Classical iterative algorithms typically rely on the conjugate gradient method. The rise of quantum computing has spurred interest in quantum linear system problems (QLSP), particularly following the introduction of the HHL algorithm by Harrow et al. in 2009, which demonstrated the potential for exponential speedup compared to classical algorithms. However, the performance of the HHL algorithm is constrained by its dependence on the square of the condition number. To address this limitation, alternative approaches based on adiabatic quantum computing (AQC) have been proposed, which exhibit complexity scaling linearly with the condition number. AQC solves QLSP by smoothly varying the parameters of the Hamiltonian. However, this method suffers from high Hamiltonian simulation complexity. In response, this work designs new Hamiltonians and proposes a quantum discrete adiabatic linear solver based on block encoding and eigenvalue separation techniques (BEES-QDALS). This approach bypasses Hamiltonian simulation through a first-order approximation and leverages block encoding to achieve equivalent non-unitary operations on qubits. By comparing the fidelity of the original algorithm and BEES-QDALS when solving QLSP with a fixed number of steps, and the number of steps required to reach a target fidelity, it is found that BEES-QDALS significantly outperforms the original algorithm. Specifically, BEES-QDALS achieves higher fidelity with the same number of steps and requires fewer steps to reach the same target fidelity.

quant-ph↗

arXiv2024

PSCodec: A Series of High-Fidelity Low-bitrate Neural Speech Codecs Leveraging Prompt Encoders

Neural speech codecs have recently emerged as a focal point in the fields of speech compression and generation. Despite this progress, achieving high-quality speech reconstruction under low-bitrate scenarios remains a significant challenge. In this paper, we propose PSCodec, a series of neural speech codecs based on prompt encoders, comprising PSCodec-Base, PSCodec-DRL-ICT, and PSCodec-CasAN, which are capable of delivering high-performance speech reconstruction with low bandwidths. Specifically, we first introduce PSCodec-Base, which leverages a pretrained speaker verification model-based prompt encoder (VPP-Enc) and a learnable Mel-spectrogram-based prompt encoder (MelP-Enc) to effectively disentangle and integrate voiceprint and Mel-related features in utterances. To further enhance feature utilization efficiency, we propose PSCodec-DRL-ICT, incorporating a structural similarity (SSIM) based disentangled representation loss (DRL) and an incremental continuous training (ICT) strategy. While PSCodec-DRL-ICT demonstrates impressive performance, its reliance on extensive hyperparameter tuning and multi-stage training makes it somewhat labor-intensive. To circumvent these limitations, we propose PSCodec-CasAN, utilizing an advanced cascaded attention network (CasAN) to enhance representational capacity of the entire system. Extensive experiments show that our proposed PSCodec-Base, PSCodec-DRL-ICT, and PSCodec-CasAN all significantly outperform several state-of-the-art neural codecs, exhibiting substantial improvements in both speech reconstruction quality and speaker similarity under low-bitrate conditions.

cs.SD↗

arXiv2024

Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

Learning a reward model (RM) from human preferences has been an important component in aligning large language models (LLMs). The canonical setup of learning RMs from pairwise preference data is rooted in the classic Bradley-Terry (BT) model that accepts binary feedback, i.e., the label being either Response 1 is better than Response 2, or the opposite. Such a setup inevitably discards potentially useful samples (such as "tied" between the two responses) and loses more fine-grained information (such as "slightly better"). In this paper, we propose a framework for learning RMs under ordinal feedback which generalizes the case of binary preference feedback to any arbitrary granularity. Specifically, we first identify a marginal unbiasedness condition, which generalizes the assumption of the BT model in the existing binary feedback setting. The condition validates itself via the sociological concept of the wisdom of the crowd. Under the condition, we develop a natural probability model for pairwise preference data under ordinal feedback and analyze its properties. We prove the statistical benefits of ordinal feedback in terms of reducing the Rademacher complexity compared to the case of binary feedback. The proposed learning objective and the theory also extend to hinge loss and direct policy optimization (DPO). In particular, the theoretical analysis may be of independent interest when applying to a seemingly unrelated problem of knowledge distillation to interpret the bias-variance trade-off therein. The framework also sheds light on writing guidance for human annotators. Our numerical experiments validate that fine-grained feedback leads to better reward learning for both in-distribution and out-of-distribution settings. Further experiments show that incorporating a certain proportion of samples with tied preference boosts RM learning.

cs.LG↗

arXiv2024

Can Language Models Enable In-Context Database?

Large language models (LLMs) are emerging as few-shot learners capable of handling a variety of tasks, including comprehension, planning, reasoning, question answering, arithmetic calculations, and more. At the core of these capabilities is LLMs' proficiency in representing and understanding structural or semi-structural data, such as tables and graphs. Numerous studies have demonstrated that reasoning on tabular data or graphs is not only feasible for LLMs but also gives a promising research direction which treats these data as in-context data. The lightweight and human readable characteristics of in-context database can potentially make it an alternative for the traditional database in typical RAG (Retrieval Augmented Generation) settings. However, almost all current work focuses on static in-context data, which does not allow dynamic update. In this paper, to enable dynamic database update, delta encoding of database is proposed. We explore how data stored in traditional RDBMS can be encoded as in-context text and evaluate LLMs' proficiency for CRUD (Create, Read, Update and Delete) operations on in-context databases. A benchmark named InConDB is presented and extensive experiments are conducted to show the performance of different language models in enabling in-context database by varying the database encoding method, prompting method, operation type and input data distribution, revealing both the proficiency and limitations.

cs.DB↗

arXiv2024

Building Multi-Agent Copilot towards Autonomous Agricultural Data Management and Analysis

Current agricultural data management and analysis paradigms are to large extent traditional, in which data collecting, curating, integration, loading, storing, sharing and analyzing still involve too much human effort and know-how. The experts, researchers and the farm operators need to understand the data and the whole process of data management pipeline to make fully use of the data. The essential problem of the traditional paradigm is the lack of a layer of orchestrational intelligence which can understand, organize and coordinate the data processing utilities to maximize data management and analysis outcome. The emerging reasoning and tool mastering abilities of large language models (LLM) make it a potentially good fit to this position, which helps a shift from the traditional user-driven paradigm to AI-driven paradigm. In this paper, we propose and explore the idea of a LLM based copilot for autonomous agricultural data management and analysis. Based on our previously developed platform of Agricultural Data Management and Analytics (ADMA), we build a proof-of-concept multi-agent system called ADMA Copilot, which can understand user's intent, makes plans for data processing pipeline and accomplishes tasks automatically, in which three agents: a LLM based controller, an input formatter and an output formatter collaborate together. Different from existing LLM based solutions, by defining a meta-program graph, our work decouples control flow and data flow to enhance the predictability of the behaviour of the agents. Experiments demonstrates the intelligence, autonomy, efficacy, efficiency, extensibility, flexibility and privacy of our system. Comparison is also made between ours and existing systems to show the superiority and potential of our system.

cs.AI↗

arXiv2024

On the Convergence of Federated Averaging under Partial Participation for Over-parameterized Neural Networks

Federated learning (FL) is a widely employed distributed paradigm for collaboratively training machine learning models from multiple clients without sharing local data. In practice, FL encounters challenges in dealing with partial client participation due to the limited bandwidth, intermittent connection and strict synchronized delay. Simultaneously, there exist few theoretical convergence guarantees in this practical setting, especially when associated with the non-convex optimization of neural networks. To bridge this gap, we focus on the training problem of federated averaging (FedAvg) method for two canonical models: a deep linear network and a two-layer ReLU network. Under the over-parameterized assumption, we provably show that FedAvg converges to a global minimum at a linear rate $\mathcal{O}\left((1-\frac{min_{i \in [t]}|S_i|}{N^2})^t\right)$ after $t$ iterations, where $N$ is the number of clients and $|S_i|$ is the number of the participated clients in the $i$-th iteration. Experimental evaluations confirm our theoretical results.

cs.LG↗

arXiv2024

Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach

In this paper, we study the problem of uncertainty estimation and calibration for LLMs. We begin by formulating the uncertainty estimation problem, a relevant yet underexplored area in existing literature. We then propose a supervised approach that leverages labeled datasets to estimate the uncertainty in LLMs' responses. Based on the formulation, we illustrate the difference between the uncertainty estimation for LLMs and that for standard ML models and explain why the hidden neurons of the LLMs may contain uncertainty information. Our designed approach demonstrates the benefits of utilizing hidden activations to enhance uncertainty estimation across various tasks and shows robust transferability in out-of-distribution settings. We distinguish the uncertainty estimation task from the uncertainty calibration task and show that better uncertainty estimation leads to better calibration performance. Furthermore, our method is easy to implement and adaptable to different levels of model accessibility including black box, grey box, and white box.

cs.LG↗

arXiv2024

Monocular Visual Place Recognition in LiDAR Maps via Cross-Modal State Space Model and Multi-View Matching

Achieving monocular camera localization within pre-built LiDAR maps can bypass the simultaneous mapping process of visual SLAM systems, potentially reducing the computational overhead of autonomous localization. To this end, one of the key challenges is cross-modal place recognition, which involves retrieving 3D scenes (point clouds) from a LiDAR map according to online RGB images. In this paper, we introduce an efficient framework to learn descriptors for both RGB images and point clouds. It takes visual state space model (VMamba) as the backbone and employs a pixel-view-scene joint training strategy for cross-modal contrastive learning. To address the field-of-view differences, independent descriptors are generated from multiple evenly distributed viewpoints for point clouds. A visible 3D points overlap strategy is then designed to quantify the similarity between point cloud views and RGB images for multi-view supervision. Additionally, when generating descriptors from pixel-level features using NetVLAD, we compensate for the loss of geometric information, and introduce an efficient scheme for multi-view generation. Experimental results on the KITTI and KITTI-360 datasets demonstrate the effectiveness and generalization of our method. The code will be released upon acceptance.

cs.CV↗

arXiv2024

Understanding the Training and Generalization of Pretrained Transformer for Sequential Decision Making

In this paper, we consider the supervised pre-trained transformer for a class of sequential decision-making problems. The class of considered problems is a subset of the general formulation of reinforcement learning in that there is no transition probability matrix; though seemingly restrictive, the subset class of problems covers bandits, dynamic pricing, and newsvendor problems as special cases. Such a structure enables the use of optimal actions/decisions in the pre-training phase, and the usage also provides new insights for the training and generalization of the pre-trained transformer. We first note the training of the transformer model can be viewed as a performative prediction problem, and the existing methods and theories largely ignore or cannot resolve an out-of-distribution issue. We propose a natural solution that includes the transformer-generated action sequences in the training procedure, and it enjoys better properties both numerically and theoretically. The availability of the optimal actions in the considered tasks also allows us to analyze the properties of the pre-trained transformer as an algorithm and explains why it may lack exploration and how this can be automatically resolved. Numerically, we categorize the advantages of pre-trained transformers over the structured algorithms such as UCB and Thompson sampling into three cases: (i) it better utilizes the prior knowledge in the pre-training data; (ii) it can elegantly handle the misspecification issue suffered by the structured algorithms; (iii) for short time horizon such as $T\le50$, it behaves more greedy and enjoys much better regret than the structured algorithms designed for asymptotic optimality.

cs.LG↗

arXiv2024

Takin: A Cohort of Superior Quality Zero-shot Speech Generation Models

With the advent of the big data and large language model era, zero-shot personalized rapid customization has emerged as a significant trend. In this report, we introduce Takin AudioLLM, a series of techniques and models, mainly including Takin TTS, Takin VC, and Takin Morphing, specifically designed for audiobook production. These models are capable of zero-shot speech production, generating high-quality speech that is nearly indistinguishable from real human speech and facilitating individuals to customize the speech content according to their own needs. Specifically, we first introduce Takin TTS, a neural codec language model that builds upon an enhanced neural speech codec and a multi-task training framework, capable of generating high-fidelity natural speech in a zero-shot way. For Takin VC, we advocate an effective content and timbre joint modeling approach to improve the speaker similarity, while advocating for a conditional flow matching based decoder to further enhance its naturalness and expressiveness. Last, we propose the Takin Morphing system with highly decoupled and advanced timbre and prosody modeling approaches, which enables individuals to customize speech production with their preferred timbre and prosody in a precise and controllable manner. Extensive experiments validate the effectiveness and robustness of our Takin AudioLLM series models. For detailed demos, please refer to https://everest-ai.github.io/takinaudiollm/.

cs.SD↗