arXiv subjects

Jianyu Wang

Publications and source records attributed to Jianyu Wang.

At least 55 records · Page 3Linked to original sources

arXiv2025

Towards High-fidelity 3D Talking Avatar with Personalized Dynamic Texture

Significant progress has been made for speech-driven 3D face animation, but most works focus on learning the motion of mesh/geometry, ignoring the impact of dynamic texture. In this work, we reveal that dynamic texture plays a key role in rendering high-fidelity talking avatars, and introduce a high-resolution 4D dataset \textbf{TexTalk4D}, consisting of 100 minutes of audio-synced scan-level meshes with detailed 8K dynamic textures from 100 subjects. Based on the dataset, we explore the inherent correlation between motion and texture, and propose a diffusion-based framework \textbf{TexTalker} to simultaneously generate facial motions and dynamic textures from speech. Furthermore, we propose a novel pivot-based style injection strategy to capture the complicity of different texture and motion styles, which allows disentangled control. TexTalker, as the first method to generate audio-synced facial motion with dynamic texture, not only outperforms the prior arts in synthesising facial motions, but also produces realistic textures that are consistent with the underlying facial movements. Project page: https://xuanchenli.github.io/TexTalk/.

cs.CV↗

arXiv2025

Determined Blind Source Separation with Sinkhorn Divergence-based Optimal Allocation of the Source Power

Blind source separation (BSS) refers to the process of recovering multiple source signals from observations recorded by an array of sensors. Common approaches to BSS, including independent vector analysis (IVA), and independent low-rank matrix analysis (ILRMA), typically rely on second-order models to capture the statistical independence of source signals for separation. However, these methods generally do not account for the implicit structural information across frequency bands, which may lead to model mismatches between the assumed source distributions and the distributions of the separated source signals estimated from the observed mixtures. To tackle these limitations, this paper shows that conventional approaches such as IVA and ILRMA can easily be leveraged by the Sinkhorn divergence, incorporating an optimal transport (OT) framework to adaptively correct source variance estimates. This allows for the recovery of the source distribution while modeling the inter-band signal dependence and reallocating source power across bands. As a result, enhanced versions of these algorithms are developed, integrating a Sinkhorn iterative scheme into their standard implementations. Extensive simulations demonstrate that the proposed methods consistently enhance BSS performance.

cs.SD↗

arXiv2025

Statistical QoS Provisioning for Underwater Magnetic Induction Communication

Magnetic induction (MI) communication, with stable channel conditions and small antenna size, is considered as a promising solution for underwater communication network. However, the narrowband nature of the MI link can cause significant delays in the network. To comprehensively ensure the timeliness and effectiveness of the MI network, in this paper we introduce a statistical quality of service (QoS) framework for MI communication, aiming to maximize the achievable rate while provisioning delay and queue-length requirements. Specifically, we employ effective capacity theory to model underwater MI communication. Based on convex optimization theory, we propose a current control strategy that maximizes the effective capacity under the constraints of limited channel capacity and limited power. Simulations demonstrate that the current control strategy proposed for MI communication differs significantly from that in the conventional statistical QoS provisioning framework. In addition, compared to other current control strategies, the proposed strategy substantially improves the achievable rate under various delay QoS requirements.

cs.ET↗

arXiv2025

Singularity and universality from von Neumann to Rényi entanglement entropy and disorder operator in Motzkin chains

The Rényi entanglement entropy is widely used in studying quantum entanglement properties in strongly correlated systems, whose analytic continuation as the Rényi index $n \to 1$ is often believed to yield the von Neumann entanglement entropy. However, earlier findings indicate that this process exhibits a singularity for the colored Motzkin spin chain problem, leading to different scaling behaviors of $\sim \sqrt{l}$ and $\sim \log{l}$ for the von Neumann and Rényi entropies, respectively. Our analytical and numerical calculations confirm this transition, which can be explained by the exponentially increasing density of states in the entanglement spectrum that we extract numerically. Disorder operators are further employed under various symmetries to study such a system. Both analytical and numerical results demonstrate that the scaling of the disorder operators also follows $\log{l}$ as the leading behavior, matching that of the Rényi entropy. We propose that the coefficient of the term $\log{l}$ is a universal constant shared by both the Rényi entropies and disorder operators. This universal constant could potentially help capture the underlying constraint physics of Motzkin walks.

cond-mat.str-el↗

arXiv2024

Momentum Approximation in Asynchronous Private Federated Learning

Asynchronous protocols have been shown to improve the scalability of federated learning (FL) with a massive number of clients. Meanwhile, momentum-based methods can achieve the best model quality in synchronous FL. However, naively applying momentum in asynchronous FL algorithms leads to slower convergence and degraded model performance. It is still unclear how to effective combinie these two techniques together to achieve a win-win. In this paper, we find that asynchrony introduces implicit bias to momentum updates. In order to address this problem, we propose momentum approximation that minimizes the bias by finding an optimal weighted average of all historical model updates. Momentum approximation is compatible with secure aggregation as well as differential privacy, and can be easily integrated in production FL systems with a minor communication and storage cost. We empirically demonstrate that on benchmark FL datasets, momentum approximation can achieve $1.15 \textrm{--}4\times$ speed up in convergence compared to naively combining asynchronous FL with momentum.

cs.LG↗

arXiv2024

Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding

Large Language Models (LLMs) exhibit impressive capabilities across various applications but encounter substantial challenges such as high inference latency, considerable training costs, and the generation of hallucinations. Collaborative decoding between large and small language models (SLMs) presents a promising strategy to mitigate these issues through methods including speculative decoding, contrastive decoding, and emulator or proxy fine-tuning. However, the specifics of such collaborations, particularly from a unified perspective, remain largely unexplored. Inspired by dual-process cognitive theory, we propose a unified framework in this paper, termed Fast and Slow Generating (FS-GEN). Within this framework, LLMs (sometimes along with SLMs) are categorized as System 2 (slow and deliberate), while independent SLMs are designated as System 1 (fast and intuitive). We provide a comprehensive analysis of these collaborative methodologies, elucidating their common properties and shedding light on the differential knowledge capabilities of System 2 versus System 1 through the FS-GEN framework. Our findings indicate that only a small proportion of collaborative interactions (approximately less than 20\% in most instances) are necessary across various methods. These interactions between System 1 and System 2 conform to a scaling law related to the parameter ratios, enabling predictable collaboration. Furthermore, we explore the specific conditions under which collaboration proves most effective, particularly from an uncertainty perspective, offering novel insights that may guide future optimization efforts. Our research underscores that the fundamental distinction between System 1 and System 2 lies in the uncertainty of next token predictions, where interventions by System 2 are crucial to support System 1. Code for Reproduction: https://github.com/TsinghuaC3I/FS-GEN

cs.CL↗

arXiv2024

Room temperature synthesis of gallium oxide film with a fluidic exfoliation method

Two-dimensional metal oxides play an important role in electronics and optoelectronics, and it is still a challenge to obtain thin oxides film. Here, a fluidic exfoliation method is applied to synthesis the metal oxides film by using galinstan as the reactant, and Ga2O3 film with ~1 cm size is obtained. Optical microscope and scanning electron microscope images show that the Ga2O3 film is exfoliated from the galinstan without any droplets left. Energy Dispersive X-Ray measurements confirm the existence of the Ga2O3 film. Transmission electron microscope and selected area electron diffraction patterns indicate the oxidation process do not have a prior direction. The alloy liquid based fluidic exfoliation method in room temperature provide a promising route for the synthesis of two-dimensional mental oxides, which shows significant applications in electronic and photoelectronic devices.

cond-mat.mtrl-sci↗

arXiv2024

SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages

Large Language Models (LLMs) have shown remarkable abilities across various tasks, yet their development has predominantly centered on high-resource languages like English and Chinese, leaving low-resource languages underserved. To address this disparity, we present SeaLLMs 3, the latest iteration of the SeaLLMs model family, tailored for Southeast Asian languages. This region, characterized by its rich linguistic diversity, has lacked adequate language technology support. SeaLLMs 3 aims to bridge this gap by covering a comprehensive range of languages spoken in this region, including English, Chinese, Indonesian, Vietnamese, Thai, Tagalog, Malay, Burmese, Khmer, Lao, Tamil, and Javanese. Leveraging efficient language enhancement techniques and a specially constructed instruction tuning dataset, SeaLLMs 3 significantly reduces training costs while maintaining high performance and versatility. Our model excels in tasks such as world knowledge, mathematical reasoning, translation, and instruction following, achieving state-of-the-art performance among similarly sized models. Additionally, we prioritized safety and reliability by addressing both general and culture-specific considerations and incorporated mechanisms to reduce hallucinations. This work underscores the importance of inclusive AI, showing that advanced LLM capabilities can benefit underserved linguistic and cultural communities.

cs.CL↗

arXiv2024

Virtual Full-Duplex Wireless Communications with Zero-Interval Modulation and Sampling

In this paper, we propose a virtual full-duplex (VFD) technique with zero-interval modulation and sampling (ZIMS), where two half-duplex (HD) transceivers can simultaneously transmit signals and each transceiver can effectively receive the desired information. In ZIMS-VFD, the transceiver inserts a zero-interval for each symbol in the transmit signal and provides self-interference (SI)-free intervals for itself. Meanwhile, it samples the receive signal in the provided SI-free intervals and restores the desired symbols. Based on orthogonal frequency division multiplexing (OFDM), we formulate the system model and show the transmit signal structure. Then, we give the transceiver design for single input single output (SISO) ZIMS-VFD and extend it to multiple input multiple output (MIMO) communications. Numerical results verify our theoretical analyses and show that ZIMS-VFD can effectively increase the capacity and approach the FD without SI.

eess.SP↗

arXiv2024

Multi-Frequency Resonant Circuit Based Multi-User Emergency Through-the-Earth Communication with Magnetic Induction

Magnetic induction (MI) is an effective technique in emergency through-the-earth communications due to the higher penetration efficiency and lower propagation loss as compared with electromagnetic wave communication. How to cancel the interference between different users and enhance the effectiveness of multi-user transmissions is imperative for the practical application of MI communication. In this paper, we use multi-frequency resonant circuit to establish multiple resonant frequencies for MI communication. The transmissions corresponding to different users operate at different resonant frequencies and multi-user interferences can be naturally mitigated. Numerical results verify our theoretical analyses and show that the proposed system can significantly enhance the performance.

eess.SP↗

arXiv2024

RIS-Based Self-Interference Cancellation for Full-Duplex Broadband Transmission

Full-duplex (FD) is an attractive technology that can significantly boost the throughput of wireless communications. However, it is limited by the severe self-interference (SI) from the transmitter to the local receiver. In this paper, we propose a new SI cancellation (SIC) scheme based on reconfigurable intelligent surface (RIS), where small RISs are deployed inside FD devices to enhance SIC capability and system capacity under frequencyselective fading channels. The novel scheme can not only address the challenges associated with SIC but also improve the overall performance. We first analyze the near-field behavior of the RIS and then formulate an optimization problem to maximize the SIC capability by controlling the reflection coefficients (RCs) of the RIS and allocating the transmit power of the device. The problem is solved with alternate optimization (AO) algorithm in three cases: ideal case, where both the amplitude and phase of each RIS unit cell can be controlled independently and continuously, continuous phases, where the phase of each RIS unit cell can be controlled independently, while the amplitude is fixed to one, and discrete phases, where the RC of each RIS unit cell can only take discrete values and these discrete values are equally spaced on the unit circle. For the ideal case, the closed-form solution to RC is derived with Karush-Kuhn-Tucker (KKT) conditions. Based on Riemannian conjugate gradient (RCG) algorithm, we optimize the RC for the case of continuous phases and then extend the solution to the case of discrete phases by the nearest point projection (NPP) method. Simulation results are given to validate the performance of our proposed SIC scheme.

eess.SP↗

arXiv2024

SeaLLMs -- Large Language Models for Southeast Asia

Despite the remarkable achievements of large language models (LLMs) in various tasks, there remains a linguistic bias that favors high-resource languages, such as English, often at the expense of low-resource and regional languages. To address this imbalance, we introduce SeaLLMs, an innovative series of language models that specifically focuses on Southeast Asian (SEA) languages. SeaLLMs are built upon the Llama-2 model and further advanced through continued pre-training with an extended vocabulary, specialized instruction and alignment tuning to better capture the intricacies of regional languages. This allows them to respect and reflect local cultural norms, customs, stylistic preferences, and legal considerations. Our comprehensive evaluation demonstrates that SeaLLM-13b models exhibit superior performance across a wide spectrum of linguistic tasks and assistant-style instruction-following capabilities relative to comparable open-source models. Moreover, they outperform ChatGPT-3.5 in non-Latin languages, such as Thai, Khmer, Lao, and Burmese, by large margins while remaining lightweight and cost-effective to operate.

cs.CL↗

arXiv2024

CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following

With the advancement of language models (LMs), their exposure to private data is increasingly inevitable, and their deployment (especially for smaller ones) on personal devices, such as PCs and smartphones, has become a prevailing trend. In contexts laden with user information, enabling models to both safeguard user privacy and execute commands efficiently emerges as an essential research imperative. In this paper, we propose CoGenesis, a collaborative generation framework integrating large (hosted on cloud infrastructure) and small models (deployed on local devices) to address privacy concerns logically. Initially, we design a pipeline to create personalized writing instruction datasets enriched with extensive context details as the testbed of this research issue. Subsequently, we introduce two variants of CoGenesis based on sketch and logits respectively. Our experimental findings, based on our synthesized dataset and two additional open-source datasets, indicate that: 1) Large-scale models perform well when provided with user context but struggle in the absence of such context. 2) While specialized smaller models fine-tuned on the synthetic dataset show promise, they still lag behind their larger counterparts. 3) Our CoGenesis framework, utilizing mixed-scale models, showcases competitive performance, providing a feasible solution to privacy issues.

cs.CL↗

arXiv2024

Operators' cognitive performance under extreme hot-humid exposure and its physiological-psychological mechanism based on ECG, fNIRS, and Eye Tracking

Operators' cognitive functions are impaired significantly under extreme heat stress, potentially resulting in more severe secondary disasters. This research investigated the impact of elevated temperature and humidity (25 60%RH, 30 70%RH, 35 80%RH, 40 90%RH) on the cognitive functions and performance of operators. Meanwhile, we explored the psychological-physiological mechanism underlying the change in performance by electrocardiogram (ECG), functional near-infrared spectroscopy (fNIRS), and eye tracking physiologically. Psychological aspects such as situation awareness, workload, and working memory were assessed. Eventually, we verified and extended the maximal adaptability model to the extreme condition. Unexpectedly, a temporary improvement in simple reaction tasks but rapid impairment in advanced cognitive functions (i.e. situation awareness, communication, working memory) was obtained above 35 WBGT. The best performance in a suitable environment was due to more effective activation in the prefrontal cortex (PFC). With temperature increasing, more mistakes occurred and comprehension was impaired due to drowsiness and lower arousal levels, according to evidence of compensatory effect in fNIRS. In the extreme environment, the enhanced PFC cooperation with higher functional connectivity resulted in a temporary improvement, while depressed activation in PFC, heavy physical load, and poor regulation of the cardiovascular system restricted it. Our results provide a detailed study of the process of operators' performance and cognitive functions when encountering increasing heat stress, as well as its underlying mechanisms from a neuroergonomics perspective. This can contribute to a better understanding of the interaction between operators' performance and workplace conditions, and help to achieve a more reliable human-centered production system in the promising era of Industry 5.0.

q-bio.NC↗

arXiv2024

Determined Multichannel Blind Source Separation with Clustered Source Model

The independent low-rank matrix analysis (ILRMA) method stands out as a prominent technique for multichannel blind audio source separation. It leverages nonnegative matrix factorization (NMF) and nonnegative canonical polyadic decomposition (NCPD) to model source parameters. While it effectively captures the low-rank structure of sources, the NMF model overlooks inter-channel dependencies. On the other hand, NCPD preserves intrinsic structure but lacks interpretable latent factors, making it challenging to incorporate prior information as constraints. To address these limitations, we introduce a clustered source model based on nonnegative block-term decomposition (NBTD). This model defines blocks as outer products of vectors (clusters) and matrices (for spectral structure modeling), offering interpretable latent vectors. Moreover, it enables straightforward integration of orthogonality constraints to ensure independence among source images. Experimental results demonstrate that our proposed method outperforms ILRMA and its extensions in anechoic conditions and surpasses the original ILRMA in simulated reverberant environments.

cs.SD↗

arXiv2024

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models up to 30B parameters, including both dense models and mixture-of-experts (MoE) variants, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.

cs.CV↗

arXiv2024

Independent low-rank matrix analysis based on the Sinkhorn divergence source model for blind source separation

The so-called independent low-rank matrix analysis (ILRMA) has demonstrated a great potential for dealing with the problem of determined blind source separation (BSS) for audio and speech signals. This method assumes that the spectra from different frequency bands are independent and the spectral coefficients in any frequency band are Gaussian distributed. The Itakura-Saito divergence is then employed to estimate the source model related parameters. In reality, however, the spectral coefficients from different frequency bands may be dependent, which is not considered in the existing ILRMA algorithm. This paper presents an improved version of ILRMA, which considers the dependency between the spectral coefficients from different frequency bands. The Sinkhorn divergence is then exploited to optimize the source model parameters. As a result of using the cross-band information, the BSS performance is improved. But the number of parameters to be estimated also increases significantly, and so is the computational complexity. To reduce the algorithm complexity, we apply the Kronecker product to decompose the modeling matrix into the product of a number of matrices of much smaller dimensionality. An efficient algorithm is then developed to implement the Sinkhorn divergence based BSS algorithm and the complexity is reduced by an order of magnitude.

cs.SD↗

arXiv2024

Multichannel blind speech source separation with a disjoint constraint source model

Multichannel convolutive blind speech source separation refers to the problem of separating different speech sources from the observed multichannel mixtures without much a priori information about the mixing system. Multichannel nonnegative matrix factorization (MNMF) has been proven to be one of the most powerful separation frameworks and the representative algorithms such as MNMF and the independent low-rank matrix analysis (ILRMA) have demonstrated great performance. However, the sparseness properties of speech source signals are not fully taken into account in such a framework. It is well known that speech signals are sparse in nature, which is considered in this work to improve the separation performance. Specifically, we utilize the Bingham and Laplace distributions to formulate a disjoint constraint regularizer, which is subsequently incorporated into both MNMF and ILRMA. We then derive majorization-minimization rules for updating parameters related to the source model, resulting in the development of two enhanced algorithms: s-MNMF and s-ILRMA. Comprehensive simulations are conducted, and the results unequivocally demonstrate the efficacy of our proposed methodologies.

cs.SD↗